Streaming voice information processing method

Abstract

The invention discloses a streaming voice information processing method. The method comprises the following steps: creating a long connection; a step of starting an instruction by streaming voice; a step of reporting streaming audio packet data; a step of forwarding the streaming audio packet data; a step of forwarding delay event adding authority selective machine circulation; a streaming voice end signaling step; and returning an identification result. By means of the scheme, efficient parallel processing of stream audio receiving and forwarding and identification result stream return is achieved, and the situation that due to the fact that a large amount of data inrush in within a short time, stream identification resources are insufficient, processing is not timely, and the output effect is affected is prevented. The audio stream data units are continuously spliced to form new audio stream data units, response feedback of a user side is kept in a continuous refreshing state, voice signals continuously sent by a user are matched, and the interaction rhythm is controlled in the voice output state of the user.

Description

Technical Field

[0001] The present application relates to the field of information processing, and in particular to a streaming voice information processing method, a streaming voice information processing device, an electronic device, a storage medium and a data platform. Background Art

[0002] In the vehicle environment, voice interaction has become an important means of human-computer interaction. Existing voice interaction technologies include streaming recognition strategies and non-streaming recognition strategies.

[0003] Non-streaming recognition usually inputs an independent audio at one time and outputs the recognized text at one time. Since the data size of non-streaming recognition input is unstable and the rhythm is uncontrollable, there is a noticeable sense of lag in the processing effect, and the user experience is not good. Streaming recognition usually means cutting the entire audio into many audio segments, which can be input one by one for recognition. When outputting, it supports the output of the entire text at one time and the output of text segment by segment. Streaming recognition is more flexible than non-streaming recognition and improves the recognition speed.

[0004] In driving scenarios, concise sentences are needed for faster query and command execution, which places high demands on the response time of voice recognition. For example, in real-time subtitles, the recognition speed needs to be kept as synchronized as possible with the person speaking, and the recognized content is displayed on the screen in real time and can be automatically corrected. Let users always feel that the system is updating feedback without lag, which reduces user anxiety.

[0005] Because a streaming voice information processing solution is needed, a duplex communication connection channel is maintained between the data acquisition end and the data processing end, and limited CPU, memory and network bandwidth resources are efficiently utilized to achieve high concurrency and orderly forwarding of the user-side audio packet data to the automatic speech recognition engine, while at the same time transmitting the streaming recognition results back to the client. Summary of the invention

[0006] The purpose of the present invention is to provide a streaming voice information processing method, a streaming voice information processing device, an electronic device, a storage medium and a data platform, which at least solve the problem of streaming recognition caused by unstable network quality, solve the problem of improving concurrency and throughput under limited computing resources, and solve one of the technical problems in the problem of coordinated work of the three tasks of streaming audio reception, forwarding and response.

[0007] The present invention provides the following scheme:

[0008] According to one aspect of the present invention, a method for processing streaming voice information is provided, the streaming voice information processing method comprising:

[0009] Steps to create a long connection;

[0010] Steps for starting streaming voice instructions;

[0011] The step of receiving streaming audio packet data;

[0012] The step of forwarding streaming audio packet data;

[0013] Forwarding delayed events adds steps for transferring permissions at an appropriate time;

[0014] Steps for signaling the end of streaming voice;

[0015] Identify the steps where the results are returned.

[0016] Furthermore, the step of creating a long connection includes:

[0017] Create a user context;

[0018] Apply for audio forwarding thread resources based on the created user context;

[0019] Evenly distribute thread resources based on the audio forwarding thread resources requested;

[0020] Wherein, the balanced allocation of thread resources includes allocating tasks to the thread with the least number of users among the multiple threads;

[0021] The user context includes an audio receiving context and an audio forwarding context.

[0022] Furthermore, the step of starting the streaming voice instruction includes:

[0023] Audio request resource preparation;

[0024] The audio request resource preparation includes checking whether the forwarding channel is available;

[0025] If,unavailable, then create a forwarding channel;

[0026] Wherein, corresponding to the creation of the forwarding channel, the state of the forwarding channel includes that the state flag is set to a receiving state.

[0027] Furthermore, the step of receiving streaming audio packet data includes:

[0028] Determine whether the audio packet data needs to be put into the data queue according to the audio forwarding channel status;

[0029] If,needed, the audio packet data is queued and it is determined whether a forwarding event needs to be added according to the shared variable state of the forwarding event adding weight;

[0030] If necessary, add the audio forwarding delay event into the event queue.

[0031] Furthermore, the step of forwarding the streaming audio packet data includes:

[0032] Based on the forwarding thread waiting for the forwarding event to arrive, when the forwarding delay event arrives, the user context information contained in the event is obtained, and the user's data queue is checked to see whether there is audio packet data to be sent;

[0033] If yes, check whether it is the first group of audio packet data in the queue;

[0034] If,yes, then update the forwarding status to forwarding;

[0035] If,not, then check the status in the context to update the current forwarding status, and take out the audio packet data in the data queue and forward it;

[0036] It also includes the step of adding permissions to forward the delayed event and transferring it at an appropriate time;

[0037] The step of transferring the forwarding delay event adding authority at an appropriate time comprises checking again whether there is still data in the user's audio packet data queue;

[0038] If there is still one, add the audio forwarding delay event into the event queue;

[0039] If not, then mark the right to add the forwarding event and transfer the right to the receiving side of the audio packet data.

[0040] Further, the step of the streaming voice end signaling includes:

[0041] Query the forwarding status of the current audio;

[0042] The querying of the forwarding status of the current audio includes querying the status of the audio forwarding context;

[0043] Determine whether the current audio forwarding status requires the end audio packet data to be put into the data queue;

[0044] If,necessary, the audio packet data is queued and it is determined whether the next round of forwarding events,needs to be added;

[0045] The step of queuing the audio packet data includes queuing the data of the data queue of the audio receiving context;

[0046] If necessary, add the audio forwarding delay event into the event queue;

[0047] The step of adding an audio forwarding delay event into a queue includes adding an event forwarding event into a queue of an audio forwarding context.

[0048] Furthermore, the step of returning the recognition result includes:

[0049] Get the information of the streaming recognition response;

[0050] Determining whether a stream identification result is normal according to the information of the stream identification response;

[0051] If,normal, then determine whether the streaming recognition result is a successful connection creation result, an intermediate recognition result, or a final recognition result;

[0052] If the result is a successful connection creation, the forwarding channel is placed in an available state according to the preset successful connection creation result processing strategy;

[0053] If it is an intermediate recognition result, the streaming recognition result is processed according to the preset intermediate recognition result processing strategy, and the channel state is set to a two-way communication state;

[0054] If it is the final recognition result, the streaming recognition result is processed according to the preset final recognition result processing strategy, and the channel status flag is set to the recognition end state, idle state, or recognition completed but sending incomplete state corresponding to the reception and forwarding status of the audio data.

[0055] Furthermore, it also includes:

[0056] Initiate the creation of a persistent connection based on the creation user context;

[0057] Register the ASR result processor based on the successful creation of a persistent connection;

[0058] After the persistent connection is successfully created and the ASR result processor is registered, the callback is used to update the user context and set the channel status to the available status.

[0059] Furthermore, it also includes:

[0060] Get the information about the arrival time of the audio forwarding delay event;

[0061] The information of the arrival time of the audio forwarding delay event includes that the event queue of the audio forwarding context expires and the event is dequeued;

[0062] Check whether there is corresponding audio packet data in the user data queue according to the information of the arrival time of the audio forwarding delay event;

[0063] If,it exists, then check whether the forwarding channel is available;

[0064] If,available, then forward the audio data and update the information of the audio forwarding context,state;

[0065] Determine whether to add the next round of forwarding events based on the information of forwarding audio data and updating the audio forwarding context status;

[0066] If, added, the audio data is forwarded and the audio forwarding context state is updated.

[0067] Furthermore, it also includes:

[0068] According to the information of the arrival time of the audio forwarding delay event, it is checked that there is no corresponding audio packet data, and the right to create the forwarding event is transferred;

[0069] According to the information of forwarding the audio data and updating the audio forwarding context state, it is determined not to add the next round of forwarding events, and the right to create the forwarding event is transferred to the receiving side of the audio packet data;

[0070] According to the inspection, the forwarding channel is unavailable and the connection is reestablished.

[0071] According to two aspects of the present invention, there is provided a streaming voice information processing device, the streaming voice information processing device comprising:

[0072] Long connection module, used to create long connection steps;

[0073] A start instruction module is used for the step of starting the streaming voice instruction;

[0074] A data reporting module, used for the step of receiving streaming audio packet data;

[0075] A data forwarding module, used for the step of forwarding streaming audio packet data;

[0076] The permission adding module is used to forward the delayed event and add the permission to transfer at the right time;

[0077] End instruction module, used for the step of streaming voice end signaling;

[0078] The result return module is used to identify the step of result return.

[0079] According to three aspects of the present invention, there is provided an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0080] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the streaming voice information processing method.

[0081] According to four aspects of the present invention, a computer-readable storage medium is provided, comprising: a computer program executable by an electronic device is stored therein, and when the computer program runs on the electronic device, the electronic device executes the steps of the streaming voice information processing method.

[0082] According to five aspects of the present invention, a data platform is provided, including:

[0083] An electronic device for implementing the steps of a streaming voice information processing method;

[0084] A processor, the processor runs a program, and when the program runs, the steps of the streaming voice information processing method are executed from the data output by the electronic device;

[0085] The storage medium is used to store a program, and when the program is running, the program executes the steps of the streaming voice information processing method for the data output from the electronic device.

[0086] Through the above scheme, the following beneficial technical effects are obtained:

[0087] The present application divides the audio stream data into data segments of relatively fixed data size, adapts the resources and capabilities of streaming recognition, and enables streaming recognition to proceed smoothly with less lag.

[0088] The present application controls the rate of the audio stream data input into the streaming recognition process to prevent a large amount of data from pouring in within a short period of time, causing insufficient streaming recognition resources and untimely processing, thereby affecting the output effect.

[0089] This application registers the input end of the audio stream data and adjusts the uneven state of data transmission through the forwarding thread to maintain the working saturation of the streaming recognition.

[0090] The present application marks an ID for an audio stream data unit (audio packet data), so that the data processing process is carried out in an orderly data queue.

[0091] The present application forms new audio stream data units by continuously splicing audio stream data units, keeps the response feedback on the user side in a constantly refreshing state, matches the voice signals continuously emitted by the user, and controls the interaction rhythm under the user's voice output state. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 It is a flow chart of a method for processing streaming voice information provided by one or more embodiments of the present invention.

[0093] Figure 2 It is a structural diagram of a streaming voice information processing device provided by one or more embodiments of the present invention.

[0094] Figure 3 A schematic diagram of a streaming speech recognition process according to a specific embodiment of the present invention.

[0095] Figure 4 A schematic diagram of a forwarding channel state transfer mechanism according to a specific embodiment of the present invention.

[0096] Figure 5 A schematic diagram of a forwarding state transfer mechanism according to a specific embodiment of the present invention.

[0097] Figure 6 A structural block diagram of an electronic device of a streaming voice information processing method provided by one or more embodiments of the present invention. DETAILED DESCRIPTION

[0098] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0099] Figure 1 It is a flow chart of a method for processing streaming voice information provided by one or more embodiments of the present invention.

[0100] like Figure 1 The streaming voice information processing method comprises:

[0101] S1, steps to create a long connection;

[0102] S2, step of starting streaming voice instruction;

[0103] S3, step of receiving streaming audio packet data;

[0104] S4, step of forwarding streaming audio packet data;

[0105] S5, the step of adding permissions to forward the delayed event and transferring them at an appropriate time;

[0106] S6, step of signaling the end of streaming voice;

[0107] S7: Step of returning the recognition result.

[0108] Specifically, the user terminal requests to establish a long connection 1. At this time, the cloud voice service will create the corresponding user context (audio receiving context and audio forwarding context), and asynchronously request the ASR processor to establish a long connection 2 and bind the corresponding ASR result processor. At the same time, it also registers and applies to be bound to the thread in the corresponding forwarding thread pool; after receiving the streaming voice start signaling requested by the user terminal, check the forwarding condition status and update the forwarding status. When the audio stream packet arrives, the data enqueue and event enqueue are processed according to the forwarding status. When a forwarding event arrives, the forwarding action and the addition of the next forwarding event will be triggered. As the forwarding progresses, the ASR result processor begins to return the recognized streaming results one after another. At this time, the forwarding state will also reach a new state, and the streaming recognition results will be given to the user terminal to form recognition feedback of the user's voice data.

[0109] In this embodiment, the steps of creating a long connection include:

[0110] Create a user context;

[0111] Apply for audio forwarding thread resources based on the created user context;

[0112] Evenly distribute thread resources based on the audio forwarding thread resources requested;

[0113] The balanced allocation of thread resources includes allocating tasks to a thread with the least number of users among the multiple threads;

[0114] The user context includes an audio receiving context and an audio forwarding context.

[0115] Specifically, when the user establishes a long connection, the forwarding threads will be evenly distributed. Before sending voice packet data to the voice cloud, the user will synchronize the status information and page information of the user after the connection is established. The cloud will integrate and refine the context information and cache it in the user's corresponding context in the memory. After the user starts to send audio packet data, under normal circumstances, the voice cloud will temporarily store the audio packet data in the data queue after receiving it, adjust the sending rhythm and send it to the ASR processor to prevent the audio packet data sequence from being too concentrated or too dispersed due to network problems, affecting the recognition effect and recognition performance of the ASR processor.

[0116] The audio receiving context includes the data queue, hot words on the current user end page, the current user end status, etc.; the audio forwarding context includes the forwarding delay event queue, adding forwarding event right shared variables, audio forwarding channel status, etc.

[0117] In this embodiment, the steps of starting the streaming voice instruction include:

[0118] Audio request resource preparation;

[0119] Audio request resource preparation includes checking whether the forwarding channel is available;

[0120] If,unavailable, then create a forwarding channel;

[0121] Wherein, corresponding to the creation of the forwarding channel, the state of the forwarding channel includes that the state flag is set to a receiving state.

[0122] Specifically, the forwarding channels include established and newly created ones. If the established forwarding channel is available, it is used directly. If the established forwarding channel is not available, a forwarding channel is created. Corresponding to the creation of the forwarding channel, the system's working flag is set, and the state flag is set to the idle state, indicating that the forwarding channel is available.

[0123] In this embodiment, the step of receiving streaming audio packet data includes:

[0124] Determine whether the audio packet data needs to be put into the data queue according to the audio forwarding channel status;

[0125] If,needed, the audio packet data is queued and it is determined whether a forwarding event needs to be added according to the shared variable state of the forwarding event adding weight;

[0126] If necessary, add the audio forwarding delay event into the event queue.

[0127] Specifically, during the reporting process of streaming audio packet data, first determine whether the audio packet data needs to be queued according to the current forwarding channel status. If necessary, it will be sent to the queue. If the forwarding channel is unavailable at this time, the audio packet data does not need to be queued and is directly discarded. Check whether the right to add the forwarding event trigger belongs to the receiving thread. If it does, it is necessary to add the audio forwarding delay event to the queue. Otherwise, the audio forwarding delay event is not added. The right to add the audio forwarding delay event flows between the receiving thread and the forwarding thread, which is used to solve the problem of the fixed-length interval event running idle due to the long interval between the two audio packet data before and after the network problem.

[0128] In this embodiment, it is determined whether the audio packet data needs to be queued according to the state of the audio forwarding channel; if yes, the audio packet data is queued; if no, all the audio packet data of the current round is discarded. At the same time, it is determined whether a delayed forwarding event needs to be added according to the state of the forwarding event adding weight shared variable; if yes, the audio forwarding delay time is added to the event queue; if no, no operation is performed.

[0129] In this embodiment, the steps of forwarding streaming audio packet data include:

[0130] Based on the forwarding thread waiting for the forwarding event to arrive, when the forwarding delay event arrives, the user context information contained in the event is obtained, and the user's data queue is checked to see whether there is audio packet data to be sent;

[0131] If yes, check whether it is the first group of audio packet data in the queue;

[0132] If,yes, then update the forwarding status to forwarding;

[0133] If,not, then check the status in the context to update the current forwarding status, and take out the audio packet data in the data queue and forward it;

[0134] It also includes the step of adding permissions to forward the delayed event and transferring it at an appropriate time;

[0135] The step of adding the permission to forward the delayed event to flow selectively includes checking again whether there is still data in the audio packet data queue of the user;

[0136] If there is still one, add the audio forwarding delay event into the event queue;

[0137] If not, then mark the right to add the forwarding event and transfer the right to the receiving side of the audio packet data.

[0138] Specifically, in one embodiment, a forwarding thread is bound to a delay event queue, and multiple users are registered on a forwarding thread. For each user registered to a forwarding thread, there is at most one delayed forwarding event in the delay queue bound to the forwarding thread, which is used to send an audio packet data in the data queue of the user. After forwarding the current audio packet data of the current user, it will be checked whether there is the next audio packet data in the data queue of the user. If it exists, the next forwarding delay event will be created. If it does not exist, the right to add a delayed forwarding event will be handed over to the receiving thread side, and the receiving thread will add a delayed forwarding event after receiving the new audio packet data. At this time, the forwarding thread can be triggered by the delay event added by other users to continue working.

[0139] In this embodiment, the steps of signaling the end of streaming voice include:

[0140] Query the forwarding status of the current audio;

[0141] Querying the current audio forwarding status includes querying the status of the audio forwarding context;

[0142] Determine whether the current audio forwarding status requires the end audio packet data to be put into the data queue;

[0143] If,necessary, the audio packet data is queued and it is determined whether the next round of forwarding events,needs to be added;

[0144] The audio packet data enqueuing includes enqueuing data of a data queue of an audio receiving context;

[0145] If necessary, add the audio forwarding delay event into the event queue;

[0146] The step of adding the audio forwarding delay event into a queue includes adding the forwarding event into a queue of the event queue of the audio forwarding context.

[0147] Specifically, the streaming semantic end signaling is generated by the user-side voice signal processing module as the end of a complete voice command request. When the cloud voice service receives the end signaling, it will check the forwarding status to determine whether it needs to add specific packet data and whether it needs to add audio forwarding delay events to the event queue.

[0148] In this embodiment, the step of returning the recognition result includes:

[0149] Get the information of the streaming recognition response;

[0150] According to the information of the stream identification response, determine whether the stream identification result is normal;

[0151] If,normal, then determine whether the streaming recognition result is a successful connection creation result, an intermediate recognition result, or a final recognition result;

[0152] If the result is a successful connection creation, the forwarding channel is placed in an available state according to the preset successful connection creation result processing strategy;

[0153] If it is an intermediate recognition result, the streaming recognition result is processed according to the preset intermediate recognition result processing strategy, and the channel state is set to a two-way communication state;

[0154] If it is the final recognition result, the streaming recognition result is processed according to the preset final recognition result processing strategy, and the channel status flag is set to the recognition end state, idle state, or recognition completed but sending incomplete state corresponding to the reception and forwarding status of the audio data.

[0155] Specifically, after being processed by the ASR result processor, it is determined whether the streaming recognition result is normal. If it is normal, it is determined whether the streaming recognition result is an intermediate recognition result or a final recognition result. During this period, due to the streaming recognition method, the voice data is not processed all at once, but processed in segments, so there is a difference between the intermediate recognition result and the final recognition result. For the intermediate recognition result, on the one hand, it is forwarded to the user end, and on the other hand, it is cached and waits for the subsequent intermediate results to arrive before continuing to process; if it is the final recognition result, it corresponds to the end signaling of this round of streaming voice.

[0156] In this embodiment, it also includes:

[0157] Initiate the creation of a persistent connection based on the creation user context;

[0158] Register the ASR result processor based on the successful creation of a persistent connection;

[0159] After the persistent connection is successfully created and the ASR result processor is registered, the callback is used to update the user context and set the channel status to the available status.

[0160] Specifically, there is a sequence for creating a persistent connection and registering an ASR result processor. First, create a user context, initiate the creation of a persistent connection, and then register the ASR result processor based on the successful creation of the persistent connection. At this time, the callback updates the user context and sets the channel status to the available state, which is feasible.

[0161] In this embodiment, the long connection means that the channel state remains in a stable connected state and the data flow state is maintained.

[0162] A long connection is also called a persistent connection. After the TCP layer handshake is successful, the connection is not disconnected immediately, and multiple messages (including heartbeats) are interacted on the basis of this connection until either party of the connection (client or server) actively disconnects. This process is called a complete long connection.

[0163] In this embodiment, it also includes:

[0164] Get the information about the arrival time of the audio forwarding delay event;

[0165] The information of the audio forwarding delay event time arrival includes the expiration of the event queue of the audio forwarding context and the event dequeuing;

[0166] Check whether there is corresponding audio packet data in the user data queue according to the information of the arrival time of the audio forwarding delay event;

[0167] If,it exists, then check whether the forwarding channel is available;

[0168] If,available, then forward the audio data and update the information of the audio forwarding context,state;

[0169] Determine whether to add the next round of forwarding events based on the information of forwarding audio data and updating the audio forwarding context status;

[0170] If, added, the audio data is forwarded and the audio forwarding context state is updated.

[0171] In this embodiment, it also includes:

[0172] According to the information of the arrival time of the audio forwarding delay event, it is checked that there is no corresponding audio packet data, and the right to create the forwarding event is transferred;

[0173] In the forwarding thread, after the audio forwarding delay event time arrives, the audio packet data in the user data queue is forwarded, and then the user data queue is checked to see if there is still audio packet data to be forwarded. If there is no audio packet data to be forwarded, the right to create a forwarding event is transferred to the receiving thread.

[0174] According to the information of forwarding the audio data and updating the audio forwarding context state, it is determined not to add the next round of forwarding events, and the right to create the forwarding event is transferred to the receiving side of the audio packet data;

[0175] According to the inspection, the forwarding channel is unavailable and the connection is reestablished.

[0176] Specifically, in one embodiment, the functional module of the vehicle computer running VAD generates vad start and vad end, and vad start and vad end can be used as streaming voice start signaling and streaming voice end signaling.

[0177] The ASR service module is used to convert the vocabulary content in human speech into computer-readable input, binary code or character sequence, etc.

[0178] In the process of streaming recognition, the service of the ASR service module is called, that is, the audio data stream is sent to the ASR service, the text corresponding to the audio stream is recognized word by word, and the results are accumulated and output. The results can be continuously sent back to the display screen on the vehicle side to form an instant interactive feedback effect.

[0179] In one embodiment, it further includes:

[0180] Setting a segmentation rule for the audio stream data, and segmenting the audio stream data corresponding to the voice signal according to the rule before sending;

[0181] Generating a plurality of audio stream data units according to the audio stream data corresponding to the segmented speech signal;

[0182] A plurality of audio stream data units constitute an audio stream data unit queue;

[0183] Each audio stream data unit in the queue is marked with an ID.

[0184] In this embodiment, it also includes:

[0185] Setting a splicing rule for audio stream data, splicing the current audio stream data unit of the received audio stream data unit queue according to the rule and then inputting it into the stream recognition process;

[0186] Wherein, the order of splicing audio stream data units is controlled according to the ID of each audio stream data unit in the queue;

[0187] According to the control order of splicing the audio stream data units, splicing the received audio stream data units to generate the current audio stream data unit and refreshing it;

[0188] According to the current audio stream data unit, the streaming recognition is converted into an interactive feedback result, and according to the refresh of the current audio stream data unit, the interactive feedback result of the streaming recognition conversion is refreshed and sent to the vehicle end.

[0189] In one embodiment, each audio stream data unit can be sent to the ASR service separately, and based on the audio stream data unit, the result of streaming recognition is output and continuously transmitted back to refresh the display screen on the vehicle side.

[0190] For example, if the user's voice is a request for "how will the weather be tomorrow?", based on the user's voice, it is divided into audio stream data units of "tomorrow", "weather" and "how". The ASR service returns the results of "tomorrow", "tomorrow's weather" and "how will the weather be tomorrow". The user decides whether to end the voice output at the user end, and the ASR service continues to follow up the process of the user's voice output until the user ends the voice output, and refreshes the obtained results corresponding to the voice output that the user wants to end.

[0191] In this embodiment, it also includes:

[0192] Setting a rate control rule for audio stream data, and controlling the time interval of audio stream data units input into the stream recognition process according to the rule;

[0193] Among them, acquiring speech rate feature information;

[0194] According to the speech rate feature information, a lower limit threshold of the time interval of the audio stream data unit for controlling the input stream recognition process is set;

[0195] caching the audio stream data unit queue according to the time interval for receiving the audio stream data unit being less than the time interval lower limit threshold;

[0196] According to the time interval of the audio stream data unit of the input streaming recognition process being greater than or equal to the lower time interval threshold, the cached audio stream data unit queue is read in the order of the audio stream data unit tag ID to obtain the audio stream data unit.

[0197] In one embodiment, the speech rate of a person has the characteristics of being predictable and measurable. According to the speech rate feature information, the lower limit threshold of the time interval of the audio stream data unit input into the streaming recognition process is set, that is, matching the speech rate of the person, and designing the streaming recognition process, such as the size of the audio stream data unit,

[0198] For example, the time interval for generating audio stream data units, the time interval for transmitting audio data units, the rate for splicing audio data units, etc.

[0199] When the vehicle-side user requests to establish a connection, a thread in the forwarding thread pool will be assigned to serve the audio forwarding task of the vehicle-side user according to the minimum number of connections. This thread is bound to a delay queue to adjust the sending interval of the user's audio packet data. Multiple users share this thread to complete the forwarding of audio packet data.

[0200] Figure 2 1 is a structural diagram of a streaming voice information processing device provided by one or more embodiments of the present invention. Figure 2The streaming voice information processing device shown includes: a long connection module, a start instruction module, a data reporting module, a data forwarding module, a permission adding module, an end instruction module, and a result returning module;

[0201] Long connection module, used to create long connection steps;

[0202] A start instruction module is used for the step of starting the streaming voice instruction;

[0203] A data reporting module, used for the step of receiving streaming audio packet data;

[0204] A data forwarding module, used for the step of forwarding streaming audio packet data;

[0205] The permission adding module is used to forward the delayed event and add the permission to transfer at the right time;

[0206] End instruction module, used for the step of streaming voice end signaling;

[0207] The result return module is used to identify the step of result return.

[0208] It is worth noting that although the system only discloses a long connection module, a start instruction module, a data reporting module, a data forwarding module, a permission addition module, an end instruction module, and a result return module, it does not mean that the device is limited to the above-mentioned basic functional modules. On the contrary, what the present invention wants to express is that, based on the above-mentioned basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with the prior art to form an infinite number of embodiments or technical solutions. In other words, the system is open rather than closed. Just because the present embodiment only discloses individual basic functional modules, it cannot be considered that the scope of protection of the claims of the present invention is limited to the above-mentioned basic functional modules.

[0209] Through the above scheme, the following beneficial technical effects are obtained:

[0210] The present application divides the audio stream data into data segments of relatively fixed data size, adapts the resources and capabilities of streaming recognition, and enables streaming recognition to proceed smoothly with less lag.

[0211] The present application controls the rate of the audio stream data input into the streaming recognition process to prevent a large amount of data from pouring in within a short period of time, causing insufficient streaming recognition resources and untimely processing, thereby affecting the output effect.

[0212] This application registers the input end of the audio stream data and adjusts the uneven state of data transmission through the forwarding thread to maintain the working saturation of the streaming recognition.

[0213] The present application marks an ID for an audio stream data unit (audio packet data), so that the data processing process is carried out in an orderly data queue.

[0214] The present application forms new audio stream data units by continuously splicing audio stream data units, keeps the response feedback on the user side in a constantly refreshing state, matches the voice signals continuously emitted by the user, and controls the interaction rhythm under the user's voice output state.

[0215] Figure 3 A schematic diagram of a streaming speech recognition process according to a specific embodiment of the present invention.

[0216] Figure 4 A schematic diagram of a forwarding channel state transfer mechanism according to a specific embodiment of the present invention.

[0217] Figure 5 A schematic diagram of a forwarding state transfer mechanism according to a specific embodiment of the present invention.

[0218] In a specific embodiment, Figure 3 The streaming speech recognition process shown in the figure decouples the streaming speech recognition process into three functional parts: audio reception management, audio forwarding management, and streaming recognition result processing. In addition, based on the scene information of these three parts, two contexts are separated, including: audio reception context and audio forwarding context. In order to ensure the orderly operation of each function, multiple forwarding states and multiple ASR channel states are abstracted.

[0219] In another specific embodiment, in order to improve the ability to cope with the problem of failure in forwarding event creation caused by uncertainty in the arrival time of audio packet data, a thread shared variable is set to transfer the creation right of the audio forwarding event.

[0220] like Figure 4The forwarding channel state flow mechanism shown includes: the initial state is no forwarding long connection (NO_CONN), the forwarding long connection is successfully created and enters the idle state (IDLE), at which time the forwarding long connection channel is available, when the first audio packet is forwarded to the ASR service via the channel, the channel jumps to the audio forwarding state (SENDING), the audio packets are continuously sent, and the state remains in the audio forwarding state (SENDING). If the channel is abnormally disconnected at this time, it jumps to the no connection state (NO_CONN), the current audio forwarding will be terminated, and the audio packet data of the current request will be cleared. When the first ASR recognition result is returned, the forwarding channel state switches to the two-way communication state (DUPLEX), at which time the audio sending or intermediate recognition result return events keep the channel in its current state. If the channel is abnormally disconnected at this time, it jumps to the no connection state (NO_CONN), the current audio forwarding will be terminated, and the audio packet data of the current request will be cleared. Under normal circumstances, when the channel is in a two-way communication state, the audio may be sent first, but the final recognition result has not been returned. In this case, it will jump to the audio sending completion state (SENDING_COMPLETE). It is also possible that the final recognition result is returned, but the audio is still not sent. At this time, the state switches to the final recognition result response but not sent complete state (FINAL_ASR_RESP_NOT_SENT_COMPLETELY). During the session, if the forwarding channel state is in the non-forwarding long connection (NO_CONN), it will trigger the re-establishment of the connection.

[0221] In this embodiment, if Figure 5The forwarding state flow mechanism shown includes: the forwarding state mechanism combined with the queue mechanism to be forwarded is mainly used to deal with the accumulation of voice requests. For example, if the previous round or several rounds of voice requests have not yet completed the request or response, this mechanism is relied on to solve the accumulation problem. The accumulation is limited by the maximum accumulation round set by the queue size. The initial state is that the forwarding channel is not connected (CHANNEL_DISCONNECT). After the forwarding connection is successfully established, if the number of pending forwarding is less than or equal to 1, the forwarding state switches to the waiting state (READY). If the number of pending forwarding is greater than 1, the state switches to the waiting state (WAITING). In the waiting state (READY), when the first audio packet is received, a forwarding event will be created, and the forwarding state switches to the running state (RUNNING). In the running state (RUNNING), there is a round of unfinished voice requests in the forwarding queue. At this time, if the next round of audio requests arrives, the state switches to the waiting state (WAITING). In this state, the voice request round that arrives only stores the audio packet data in the user data queue, and does not create a forwarding event. This is the accumulation phenomenon mentioned above. When the voice request at the head of the queue to be forwarded is sent in one round, it will be removed from the queue to be forwarded. At the same time, the forwarding state jump and the creation of a forwarding event are determined based on the forwarding channel status, the user audio packet data queue status, and the queue status of the queue to be forwarded; if the queue to be forwarded has only one voice request and a forwarding channel, the state switches to (READY). If the size of the queue to be forwarded is greater than one, it remains in the queue waiting state (WAITING), and another forwarding event is created for the next round of voice request audio packet data from the user.

[0222] In another specific embodiment, a method for efficiently processing streaming audio reception, forwarding, and streaming return of recognition results in parallel is provided based on an event-driven non-blocking thread model. Improvements are made based on the problems existing in the streaming speech recognition process:

[0223] 1. Network congestion, user audio packet data arrives at the cloud in batches, and the sending time cannot be determined by the arrival interval of the data sequence. A data forwarding event mechanism is designed to establish a delay event for the next data to be sent to determine its sending time.

[0224] 2. Aiming at the high resource consumption problem of the traditional one-user-one-forwarding-thread system, a group of thread groups adopts a registration mechanism to enable multiple users to share one forwarding thread in the thread group, thereby improving thread resource utilization efficiency and reducing resource consumption.

[0225] 3. When there is a "data vacuum" (when the previous set of data is sent, the next set of data has not yet arrived), set up an event to trigger a reversal mechanism to dynamically respond and reduce event idling.

[0226] 4. Integrate the entire process of audio grouping reception, forwarding and streaming recognition result reception, and establish a state machine mechanism to solve request congestion.

[0227] Figure 6 A structural block diagram of an electronic device of a streaming voice information processing method provided by one or more embodiments of the present invention.

[0228] like Figure 6 As shown, the present application provides an electronic device, including: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0229] A computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of a streaming voice information processing method.

[0230] The present application also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of a streaming voice information processing method.

[0231] The present application also provides a data platform, including:

[0232] An electronic device for implementing the steps of a streaming voice information processing method;

[0233] A processor, the processor runs a program, and when the program runs, the steps of the streaming voice information processing method are executed from the data output by the electronic device;

[0234] The storage medium is used to store a program, and when the program is running, the program executes the steps of the streaming voice information processing method for the data output from the electronic device.

[0235] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0236] The electronic device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory. The operating system can be any one or more computer operating systems that implement electronic device control through a process, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a Windows operating system. In addition, in an embodiment of the present invention, the electronic device can be a handheld device such as a smart phone or a tablet computer, or can be an electronic device such as a desktop computer or a portable computer, which is not particularly limited in the embodiment of the present invention.

[0237] The execution subject of the electronic device control in the embodiment of the present invention may be an electronic device, or a functional module in the electronic device that can call and execute a program. The electronic device may obtain the firmware corresponding to the storage medium. The firmware corresponding to the storage medium is provided by the supplier. The firmware corresponding to different storage media may be the same or different, which is not limited here. After the electronic device obtains the firmware corresponding to the storage medium, the firmware corresponding to the storage medium may be written into the storage medium, specifically, the firmware corresponding to the storage medium may be burned into the storage medium. The process of burning the firmware into the storage medium may be implemented using existing technology, which will not be described in detail in the embodiment of the present invention.

[0238] The electronic device may also obtain a reset command corresponding to the storage medium. The reset command corresponding to the storage medium is provided by the supplier. The reset commands corresponding to different storage media may be the same or different, and are not limited here.

[0239] At this time, the storage medium of the electronic device is a storage medium in which the corresponding firmware is written, and the electronic device can respond to the reset command corresponding to the storage medium in the storage medium in which the corresponding firmware is written, so that the electronic device resets the storage medium in which the corresponding firmware is written according to the reset command corresponding to the storage medium. The process of resetting the storage medium according to the reset command can be implemented by the existing technology and will not be described in detail in the embodiments of the present invention.

[0240] For the convenience of description, the above devices are described in terms of functions and are divided into various units and modules. Of course, when implementing the present application, the functions of each unit and module can be implemented in the same or multiple software and / or hardware.

[0241] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined.

[0242] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0243] It can be known from the description of the above implementation modes that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute the methods described in the various implementation modes of the present application or certain parts of the implementation modes.

[0244] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing streaming voice information, characterized in that: The streaming voice information processing method comprises: Steps to create a long connection; Steps for starting streaming voice instructions; The step of receiving streaming audio packet data; The step of forwarding streaming audio packet data; Forwarding delayed events adds steps for transferring permissions at an appropriate time; Steps for signaling the end of streaming voice; Identify the steps where the results are returned.

2. The method for processing streaming voice information according to claim 1, characterized in that: The steps of creating a long connection include: Create a user context; Apply for audio forwarding thread resources based on the created user context; Evenly distribute thread resources based on the audio forwarding thread resources requested; Wherein, the balanced allocation of thread resources includes allocating tasks to the thread with the least number of users among the multiple threads; The user context includes an audio receiving context and an audio forwarding context.

3. The method for processing streaming voice information according to claim 1, characterized in that: The step of starting the streaming voice instruction comprises: Audio request resource preparation; The audio request resource preparation includes checking whether the forwarding channel is available; If,unavailable, then create a forwarding channel; Wherein, corresponding to the creation of the forwarding channel, the state of the forwarding channel includes that the state flag is set to a receiving state.

4. The method for processing streaming voice information according to claim 1, characterized in that: The step of receiving the streaming audio packet data comprises: Determine whether the audio packet data needs to be put into the data queue according to the audio forwarding channel status; If,needed, the audio packet data is queued and it is determined whether a forwarding event needs to be added according to the shared variable state of the forwarding event adding weight; If necessary, add the audio forwarding delay event to the event queue.

5. The method for processing streaming voice information according to claim 1, characterized in that: The step of forwarding the streaming audio packet data comprises: Based on the forwarding thread waiting for the forwarding event to arrive, when the forwarding delay event arrives, the user context information contained in the event is obtained, and the user's data queue is checked to see whether there is audio packet data to be sent; If yes, check whether it is the first group of audio packet data in the queue; If,yes, then update the forwarding status to forwarding; If,not, then check the status in the context to update the current forwarding status, and take out the audio packet data in the data queue and forward it; It also includes the step of adding permissions to forward the delayed event and transferring it at an appropriate time; The step of transferring the forwarding delay event adding authority at an appropriate time comprises checking again whether there is still data in the user's audio packet data queue; If there is still one, add the audio forwarding delay event into the event queue; If not, then mark the right to add the forwarding event and transfer the right to the receiving side of the audio packet data.

6. The method for processing streaming voice information according to claim 1, characterized in that: The step of the streaming voice end signaling comprises: Query the forwarding status of the current audio; The querying of the forwarding status of the current audio includes querying the status of the audio forwarding context; Determine whether the current audio forwarding status requires the end audio packet data to be put into the data queue; If,necessary, the audio packet data is queued and it is determined whether the next round of forwarding events,needs to be added; The step of queuing the audio packet data includes queuing the data of the data queue of the audio receiving context; If necessary, add the audio forwarding delay event into the event queue; The step of adding an audio forwarding delay event into a queue includes adding an event forwarding event into a queue of an audio forwarding context.

7. The method for processing streaming voice information according to claim 1, characterized in that: The step of returning the recognition result comprises: Get the information of the streaming recognition response; Determining whether a stream identification result is normal according to the information of the stream identification response; If,normal, then determine whether the streaming recognition result is a successful connection creation result, an intermediate recognition result, or a final recognition result; If the connection is successfully created, the forwarding channel is placed in an available state according to the preset connection creation success result processing strategy; If it is an intermediate recognition result, the streaming recognition result is processed according to the preset intermediate recognition result processing strategy, and the channel state is set to a two-way communication state; If it is the final recognition result, the streaming recognition result is processed according to the preset final recognition result processing strategy, and the channel status flag is set to the recognition end state, idle state, or recognition completed but sending incomplete state corresponding to the reception and forwarding status of the audio data.

8. The method for processing streaming voice information according to claim 2, characterized in that: Also includes: Initiate the creation of a persistent connection based on the creation user context; Register the ASR result processor based on the successful creation of a persistent connection; After the persistent connection is successfully created and the ASR result processor is registered, the callback is used to update the user context and set the channel status to the available status.

9. The method for processing streaming voice information according to claim 5, characterized in that: Also includes: Get the information about the arrival time of the audio forwarding delay event; The information of the arrival time of the audio forwarding delay event includes that the event queue of the audio forwarding context expires and the event is dequeued; Check whether there is corresponding audio packet data in the user data queue according to the information of the arrival time of the audio forwarding delay event; If,it exists, then check whether the forwarding channel is available; If,available, then forward the audio data and update the information of the audio forwarding context,state; Determine whether to add the next round of forwarding events based on the information of forwarding audio data and updating the audio forwarding context status; If, added, the audio data is forwarded and the audio forwarding context state is updated.

10. The method for processing streaming voice information according to claim 9, characterized in that: Also includes: According to the information of the arrival time of the audio forwarding delay event, it is checked that there is no corresponding audio packet data, and the right to create the forwarding event is transferred; According to the information of forwarding audio data and updating the audio forwarding context state, it is determined not to add the next round of forwarding events, and the right to create forwarding events is transferred to the receiving side of the audio packet data; according to the check that the forwarding channel is unavailable, the connection is rebuilt.

Images