Streaming voice information processing method

Abstract

The application discloses a streaming voice information processing method, which comprises the following steps: creating a long connection; starting a streaming voice instruction; reporting streaming audio packet data; forwarding the streaming audio packet data; adding a delay event to forward the streaming voice; ending the streaming voice signaling; and returning the recognition result. The method realizes efficient processing of streaming audio reception, forwarding and recognition result streaming return in parallel, prevents a large amount of data from flowing into a short time, causes insufficient streaming recognition resources, causes processing to be not timely, and influences output effect. The audio stream data units are continuously spliced to form new audio stream data units, the response feedback of the user end is kept in a continuous refreshing state, the voice signals continuously sent by the user are matched, and the interaction rhythm is controlled in the voice output state of the user.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information processing, in particular to a streaming voice information processing method, a streaming voice information processing device, an electronic device, a storage medium and a data platform. BACKGROUND

[0002] In a vehicle-mounted environment, voice interaction has become an important means of human-computer interaction. Existing voice interaction technologies include streaming recognition strategies and non-streaming recognition strategies.

[0003] Non-streaming recognition usually inputs an independent audio at a time and outputs the recognized text at a time. Since the data size input by non-streaming recognition is unstable and the rhythm is uncontrollable, the processing effect has obvious lag, and the user experience is not good. Streaming recognition usually refers to cutting an entire audio into many segments of audio, which can be input and recognized segment by segment. When outputting, it supports one-time entire text output and segment-by-segment text output. Streaming recognition is more flexible than non-streaming recognition, and improves the recognition speed.

[0004] In a driving scene, concise sentences need to be used to query and instruct more quickly, and there is a high requirement for the response time of voice recognition. For example, in real-time subtitles, the recognition speed needs to be kept as synchronized as possible with the person speaking, the recognized content is real-time on screen and can be automatically corrected. The user always feels that the system has feedback and is updating, and there is no lag, so the user's anxiety is reduced.

[0005] Because a streaming voice information processing scheme is needed, a duplex communication connection channel is maintained between the data acquisition end and the data processing end, limited CPU, memory and network bandwidth resources are efficiently utilized, and high-concurrency audio grouping data of the user end is forwarded to an automatic speech recognition engine in an orderly and rhythmic manner, and the streaming recognition result is returned to the client. SUMMARY

[0006] The purpose of the present application is to provide a streaming voice information processing method, a streaming voice information processing device, an electronic device, a storage medium and a data platform, which at least solve the problem of streaming recognition caused by unstable network quality, solve the problem of improving the concurrency and throughput under limited computing resources, and solve the problem of collaborative work of streaming audio reception, forwarding and response.

[0007] The present application provides the following scheme:

[0008] According to one aspect of the present application, a streaming voice information processing method is provided, which comprises:

[0009] a step of creating a long connection;

[0010] a step of streaming voice start instruction;

[0011] streaming audio packet data receiving step;

[0012] streaming audio packet data forwarding step;

[0013] forwarding delay event adding permission transfer step;

[0014] streaming voice end signaling step;

[0015] recognition result returning step.

[0016] Further, the step of creating a long connection comprises:

[0017] creating a user context;

[0018] applying an audio forwarding thread resource according to the created user context;

[0019] equally allocating thread resources according to the applied audio forwarding thread resource;

[0020] The equally allocating thread resources comprises allocating a task to a thread with the least number of users among multiple threads.

[0021] The user context comprises an audio receiving context and an audio forwarding context.

[0022] Further, the step of the streaming voice start instruction comprises:

[0023] audio request resource preparation;

[0024] The audio request resource preparation comprises checking whether a forwarding channel is available.

[0025] If not, a forwarding channel is created.

[0026] The state of the forwarding channel comprises a state flag position bit set to a receiving state corresponding to the creation of the forwarding channel.

[0027] Further, the step of the streaming audio packet data receiving comprises:

[0028] judging whether audio packet data needs to enter a data queue according to an audio forwarding channel state;

[0029] If yes, the audio packet data is enqueued and whether a forwarding event needs to be added is judged according to a forwarding event adding permission shared variable state.

[0030] If yes, an audio forwarding delay event is added into an event queue.

[0031] Further, the step of the streaming audio packet data forwarding comprises:

[0032] Based on the forwarding thread waiting for the forwarding event to come, when the forwarding delay event arrives, the user context information contained in the event is obtained, and it is checked whether the user's data queue has audio packet data to be sent;

[0033] If yes, it is checked whether it is the first set of audio packet data of the queue;

[0034] If yes, the forwarding state is updated to forwarding in progress;

[0035] If no, the state in the context is checked to update the current forwarding state, and the audio packet data of the data queue is taken out and forwarded;

[0036] Further, the step of adding the forwarding delay event includes the steps of:

[0037] The step of adding the forwarding delay event includes the steps of:

[0038] If yes, the audio forwarding delay event is added to the event queue;

[0039] If no, the right to add the forwarding event is marked, and the right is transferred to the receiving side of the audio packet data.

[0040] Further, the step of adding the forwarding delay event includes the steps of:

[0041] Inquiring the forwarding state of the current audio;

[0042] The step of inquiring the forwarding state of the current audio includes the step of inquiring the state of the audio forwarding context;

[0043] Judging whether the forwarding state of the current audio needs to add the end audio packet data to the data queue;

[0044] If yes, the audio packet data is enqueued and it is judged whether the next round of forwarding event needs to be added;

[0045] The step of enqueuing the audio packet data includes the step of enqueuing the data queue of the audio receiving context;

[0046] If yes, the audio forwarding delay event is added to the event queue;

[0047] The step of adding the audio forwarding delay event to the event queue includes the step of adding the forwarding event to the event queue of the audio forwarding context.

[0048] Further, the step of returning the recognition result includes the steps of:

[0049] Obtaining the information of the streaming recognition response;

[0050] According to the information of the flow identification response, it is judged whether the flow identification result is normal or not;

[0051] If yes, it is judged whether the flow identification result is a connection creation success result, an intermediate identification result or a final identification result;

[0052] If yes, it is a connection creation success result, according to a preset connection creation success result processing strategy, the forwarding channel is placed in an available state;

[0053] If yes, it is an intermediate identification result, according to a preset intermediate identification result processing strategy, the flow identification result is processed, and the channel state is set to a bidirectional communication state;

[0054] If yes, it is a final identification result, according to a preset final identification result processing strategy, the flow identification result is processed, and corresponding to the receiving and forwarding state of the audio data, the channel state flag is set to an identification end state, an idle state or an identification completion but sending incomplete state.

[0055] Further, it further comprises:

[0056] According to the creation of the user context, a long connection is initiated;

[0057] According to the success of initiating the creation of the long connection, an ASR result processor is registered;

[0058] According to the success of initiating the creation of the long connection and the registration of the ASR result processor, the user context is updated by callback, and the channel state is set to an available state.

[0059] Further, it further comprises:

[0060] The information of the arrival of the audio forwarding delay event time is obtained;

[0061] The information of the arrival of the audio forwarding delay event time comprises that the event queue of the audio forwarding context expires and the event is dequeued;

[0062] According to the information of the arrival of the audio forwarding delay event time, it is checked whether there is corresponding audio packet data in the user data queue;

[0063] If yes, it is checked whether the forwarding channel is available;

[0064] If yes, the audio data is forwarded and the information of the audio forwarding context state is updated;

[0065] According to the information of forwarding the audio data and updating the audio forwarding context state, it is judged whether to add a next round of forwarding event;

[0066] If yes, the audio data is forwarded and the audio forwarding context state is updated.

[0067] Further, it also includes:

[0068] According to the information of the audio forwarding delay event time arrival, the corresponding audio packet data is checked for absence, and the forwarding event right is transferred for creation;

[0069] According to the information of the audio forwarding data and the audio forwarding context state update, it is judged that the next round of forwarding event is not added, and the forwarding event right is transferred to the receiving side of the audio packet data for creation;

[0070] According to the check of the forwarding channel being unavailable, the connection is reestablished.

[0071] According to the two aspects of the present application, a streaming voice information processing device is provided, and the streaming voice information processing device comprises:

[0072] A long connection module is configured to create a long connection;

[0073] A start instruction module is configured to start a streaming voice start instruction;

[0074] A data reporting module is configured to receive streaming audio packet data;

[0075] A data forwarding module is configured to forward streaming audio packet data;

[0076] A right adding module is configured to transfer a forwarding delay event adding right;

[0077] An end instruction module is configured to end a streaming voice end signaling;

[0078] A result returning module is configured to return a recognition result.

[0079] According to the three aspects of the present application, an electronic device is provided, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;

[0080] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the streaming voice information processing method.

[0081] According to the four aspects of the present application, a computer readable storage medium is provided, which comprises a computer program executable by an electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the streaming voice information processing method.

[0082] According to the five aspects of the present application, a data platform is provided, which comprises:

[0083] An electronic device for implementing the steps of the streaming voice information processing method.

[0084] A processor, the processor running a program, the program performing the steps of the streaming voice information processing method on data output from the electronic device when the program is running.

[0085] A storage medium for storing a program, the program performing the steps of the streaming voice information processing method on data output from the electronic device when the program is running.

[0086] Through the above scheme, the following beneficial technical effects are obtained:

[0087] The present application forms a data segment with a relatively fixed data size by segmenting audio stream data, adapts the resources and capabilities of streaming recognition, and enables smooth streaming recognition with less lag.

[0088] The present application controls the rate of audio stream data input into the streaming recognition process, prevents a large amount of data from flowing in within a short period of time, and causes insufficient streaming recognition resources to cause delayed processing and affect output effect.

[0089] The present application registers the input end of the audio stream data, adjusts the uneven state of data transmission through the forwarding thread, and maintains the working saturation of streaming recognition.

[0090] The present application marks the ID for the audio stream data unit (audio packet data), and the data processing process is performed under an ordered data queue.

[0091] The present application continuously splices audio stream data units to form new audio stream data units, keeps the user's response feedback in a constantly refreshed state, matches the user's continuously issued voice signals, and controls the interaction rhythm under the user's voice output state. BRIEF DESCRIPTION OF DRAWINGS

[0092] Figure 1 is a flowchart of the streaming voice information processing method provided by one or more embodiments of the present application.

[0093] Figure 2 is a structural diagram of the streaming voice information processing device provided by one or more embodiments of the present application.

[0094] Figure 3 is a schematic diagram of the streaming voice recognition process of one specific embodiment of the present application.

[0095] Figure 4 is a schematic diagram of the forwarding channel state flow mechanism of one specific embodiment of the present application.

[0096] Figure 5 is a schematic diagram of the forwarding state flow mechanism of one specific embodiment of the present application.

[0097] Figure 6 An electronic device structure block diagram of the streaming voice information processing method provided by one or more embodiments of the present application. DETAILED DESCRIPTION

[0098] The technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0099] Figure 1 A flowchart of the streaming voice information processing method provided by one or more embodiments of the present application.

[0100] As Figure 1 The streaming voice information processing method comprises:

[0101] S1, a step of creating a long connection;

[0102] S2, a step of a streaming voice start instruction;

[0103] S3, a step of streaming audio packet data reception;

[0104] S4, a step of streaming audio packet data forwarding;

[0105] S5, a step of forwarding delay event addition permission opportunistic flow transfer;

[0106] S6, a step of streaming voice end signaling;

[0107] S7, a step of recognition result return.

[0108] Specifically, the user end requests to build a long connection 1, at this time, the cloud voice service creates corresponding user context (audio reception context and audio forwarding context), and asynchronously requests the ASR processor to establish a long connection 2 and binds the corresponding ASR result processor. At the same time, it also registers and applies to bind to the thread in the forwarding thread pool; after receiving the streaming voice start signaling of the user end request, the forwarding condition state is checked and the forwarding state is updated. When the audio stream packet arrives, the data enqueuing and event enqueuing are processed according to the forwarding state. When the forwarding event arrives, the forwarding action and the addition of the next forwarding event are triggered. With the forwarding, the ASR result processor starts to return the recognized streaming results, at this time, the forwarding state also reaches a new state, and the streaming recognition result is given to the user end, forming the recognition feedback of the user voice data.

[0109] In the present embodiment, the step of creating a long connection comprises:

[0110] creating a user context;

[0111] applying for an audio forwarding thread resource according to the created user context;

[0112] equally allocating the thread resource according to the application for the audio forwarding thread resource;

[0113] The equally allocating the thread resource comprises assigning the task to a thread with the least number of users in the plurality of threads.

[0114] The user context comprises an audio receiving context and an audio forwarding context.

[0115] Specifically, when the user end establishes a long connection, the forwarding thread is equally allocated. Before the user end sends the voice packet data to the voice cloud end, the state information and the page information of the user end are synchronized after the connection is established, and the cloud end integrates and extracts the context information and caches it into the memory corresponding to the context of the user. After the user starts sending the audio packet data, the voice cloud end generally stores the data in a data queue, adjusts the sending rhythm, and sends it to the ASR processor to prevent the audio packet data sequence from being too concentrated or too dispersed due to network problems, which affects the recognition effect and recognition performance of the ASR processor.

[0116] The audio receiving context comprises a data queue, a current user end page hot word, a current user end state, etc.; and the audio forwarding context comprises a forwarding delay event queue, an added forwarding event right sharing variable, and an audio forwarding channel state, etc.

[0117] In the embodiment, the step of starting the streaming voice instruction comprises:

[0118] audio request resource preparation;

[0119] The audio request resource preparation comprises checking whether the forwarding channel is available.

[0120] If not, a forwarding channel is created.

[0121] Corresponding to the creation of the forwarding channel, the state of the forwarding channel comprises a state flag position bit set to a receiving state.

[0122] Specifically, the forwarding channel comprises an established one and a newly created one. If the established forwarding channel is available, it is directly used, and if the established forwarding channel is not available, a forwarding channel is created. Corresponding to the creation of the forwarding channel, the working flag bit of the system is set, and the state flag bit is set to an idle state, indicating that the forwarding channel is available.

[0123] In the embodiment, the step of receiving the streaming audio packet data comprises:

[0124] determining whether the audio packet data needs to enter the data queue according to the audio forwarding channel state;

[0125] If yes, the audio packet data is enqueued and it is determined whether the forwarding event needs to be added according to the state of the right sharing variable added by the forwarding event;

[0126] If yes, the audio forwarding delay event is added into the event queue.

[0127] Specifically, in the reporting process of the streaming audio packet data, it is first determined whether the audio packet data needs to enter the data queue according to the current forwarding channel state, and if yes, the audio packet data is sent into the queue. If the forwarding channel is unavailable at this time, the audio packet data does not need to enter the queue and is discarded directly. It is checked whether the right to add the forwarding event trigger belongs to the receiving thread, and if yes, the audio forwarding delay event is added into the queue. Otherwise, the audio forwarding delay event is not added. The right to add the audio forwarding delay event is circulated between the receiving thread and the forwarding thread, and is used to solve the problem of the fixed interval event idling due to the long interval between two audio packet data caused by network problems.

[0128] In the embodiment, it is determined whether the audio packet data needs to enter the data queue according to the audio forwarding channel state; if yes, the audio packet data is enqueued, and if no, all audio packet data of the current round is discarded. Meanwhile, it is determined whether the delay forwarding event needs to be added according to the state of the right sharing variable added by the forwarding event; if yes, the audio forwarding delay event is added into the event queue, and if no, no operation is performed.

[0129] In the embodiment, the steps of forwarding the streaming audio packet data include:

[0130] Based on the forwarding thread waiting for the forwarding event, when the forwarding delay event arrives, the user context information contained in the event is acquired, and it is checked whether the data queue of the user has audio packet data to be sent;

[0131] If yes, it is checked whether the audio packet data is the first group of audio packet data of the queue;

[0132] If yes, the forwarding state is updated to forwarding in progress;

[0133] If no, the state in the context is checked to update the current forwarding state, and the audio packet data of the data queue is taken out and forwarded;

[0134] Further, the step of circulating the right to add the forwarding delay event at an opportunity is included.

[0135] The step of circulating the right to add the forwarding delay event at an opportunity includes checking again whether there is still data in the audio packet data queue of the user;

[0136] If yes, add an audio forwarding delay event into the event queue.

[0137] If no, mark the right to add a forwarding event and transfer the right to the receiving side of the audio packet data.

[0138] Specifically, in one embodiment, one forwarding thread is bound to a delay event queue, and multiple users are registered to the forwarding thread. Each user registered to a forwarding thread has at most one delay forwarding event in the delay queue bound to the forwarding thread, which is used to send an audio packet data in the data queue of the user. After forwarding the current audio packet data of the current user, it is checked whether there is a next audio packet data in the data queue of the user. If yes, a next forwarding delay event is created. If no, the right to add a delay forwarding event is transferred to the receiving thread side, and the receiving thread adds a delay forwarding event after receiving a new audio packet data. At this time, the forwarding thread can be triggered to continue working by a delay event added by another user.

[0139] In the embodiment, the step of streaming voice end signaling includes:

[0140] Inquiring the forwarding state of the current audio;

[0141] Inquiring the forwarding state of the current audio includes inquiring the state of the audio forwarding context;

[0142] Determining whether the forwarding state of the current audio needs to add the end audio packet data into the data queue;

[0143] If yes, the audio packet data is enqueued, and it is determined whether a next forwarding event needs to be added;

[0144] The enqueuing of the audio packet data includes the enqueuing of the data queue of the audio receiving context.

[0145] If yes, an audio forwarding delay event is added into the event queue.

[0146] The adding of the audio forwarding delay event into the event queue includes the enqueuing of the forwarding event of the event queue of the audio forwarding context.

[0147] Specifically, the streaming semantic end signaling is generated by a user end voice signal processing module. As the end of a complete voice instruction request. After the cloud end voice service receives the end signaling, it checks the forwarding state to determine whether a specially designated packet data needs to be added and whether an audio forwarding delay event needs to be added to the event queue.

[0148] In the embodiment, the step of returning the recognition result includes:

[0149] Obtaining information of a streaming recognition response.

[0150] According to the information of the stream recognition response, it is judged whether the stream recognition result is normal or not;

[0151] If yes, it is judged whether the stream recognition result is a connection creation success result, an intermediate recognition result or a final recognition result;

[0152] If yes, it is a connection creation success result, according to a preset connection creation success result processing strategy, the forwarding channel is placed in an available state;

[0153] If yes, it is an intermediate recognition result, according to a preset intermediate recognition result processing strategy, the stream recognition result is processed, and the channel state is set to a bidirectional communication state;

[0154] If yes, it is a final recognition result, according to a preset final recognition result processing strategy, the stream recognition result is processed, and the channel state flag is set to an identification end state, an idle state or an identification completion but sending incomplete state corresponding to the receiving and forwarding state of the audio data.

[0155] Specifically, after the processing of the ASR result processor, it is judged whether the stream recognition result is normal or not, and if yes, it is judged whether the stream recognition result is an intermediate recognition result or a final recognition result. During this period, due to the stream recognition mode, the voice data is not processed at one time, but is processed in segments, and there is a difference between the intermediate recognition result and the final recognition result. For the intermediate recognition result, on the one hand, it is forwarded to the user end, and on the other hand, it is cached and waits for the arrival of the subsequent intermediate result for continuous processing; if it is a final recognition result, it corresponds to the end signaling of this round of stream voice.

[0156] In this embodiment, it also includes:

[0157] According to the creation of the user context, a long connection is initiated;

[0158] According to the success of the initiation of the long connection, the ASR result processor is registered;

[0159] According to the success of the initiation of the long connection and the registration of the ASR result processor, the user context is updated by callback, and the channel state is set to an available state.

[0160] Specifically, the creation of the long connection and the registration of the ASR result processor have a sequence. First, the user context is created, the long connection is initiated, and then according to the success of the initiation of the long connection, the ASR result processor is registered. At this time, it is feasible to update the user context by callback and set the channel state to an available state.

[0161] In this embodiment, the long connection is represented as a channel state that keeps stable and connected, maintaining the state of the data stream.

[0162] Long connection is also called persistent connection, after the TCP layer handshake is successful, the connection is not immediately disconnected, and multiple message (including heartbeat) interactions are carried out on the basis of this connection, until either party (client or server) initiates the disconnection of the connection, this process is called a complete long connection.

[0163] In this embodiment, it also includes:

[0164] Obtaining information of audio forwarding delay event time arrival;

[0165] The information of audio forwarding delay event time arrival includes that the event queue of the audio forwarding context expires and the event is dequeued.

[0166] According to the information of audio forwarding delay event time arrival, it is checked whether there is corresponding audio packet data in the user data queue;

[0167] If there is, it is checked whether the forwarding channel is available;

[0168] If it is available, the audio data is forwarded and the information of updating the audio forwarding context state is updated;

[0169] According to the information of forwarding audio data and updating the audio forwarding context state, it is judged whether to add the next round of forwarding event;

[0170] If yes, the audio data is forwarded and the audio forwarding context state is updated.

[0171] In this embodiment, it also includes:

[0172] According to the information of audio forwarding delay event time arrival, it is checked that there is no corresponding audio packet data, and the right to create a forwarding event is transferred;

[0173] In the forwarding thread, after the audio forwarding delay event time arrives, the audio packet data in the user data queue is forwarded, and then it is checked whether there is audio packet data to be forwarded in the user data queue, if there is no audio packet data to be forwarded, the right to create a forwarding event is transferred to the receiving thread.

[0174] According to the information of forwarding audio data and updating the audio forwarding context state, it is judged that the next round of forwarding event is not added, and the right to create a forwarding event is transferred to the receiving side of the audio packet data;

[0175] According to the check that the forwarding channel is not available, the connection is reestablished.

[0176] Specifically, in one embodiment, the function module of the car machine running VAD generates vad start and vad end, which can be used as streaming voice start signaling and streaming voice end signaling.

[0177] An ASR service module is configured to convert the lexical content in human speech into computer-readable input, binary code or character sequence, etc.

[0178] In the streaming recognition process, the service of the ASR service module is invoked, i.e., the audio stream is sent into the ASR service, the corresponding text of the audio stream is recognized word by word or by word, and the result is accumulated and output. The result can be constantly fed back to the display screen of the vehicle end to form the effect of instant interactive feedback.

[0179] In one embodiment, it further comprises:

[0180] A segmentation rule of the audio stream data is set, and the audio stream data corresponding to the voice signal is segmented according to the rule and then sent;

[0181] According to the segmented audio stream data corresponding to the voice signal, a plurality of audio stream data units are generated;

[0182] The plurality of audio stream data units form an audio stream data unit queue;

[0183] Each audio stream data unit in the queue is marked with an ID.

[0184] In this embodiment, it further comprises:

[0185] A splicing rule of the audio stream data is set, and the current audio stream data unit of the received audio stream data unit queue is spliced according to the rule and then input into the process of streaming recognition;

[0186] According to the ID marked on each audio stream data unit in the queue, the order of splicing the audio stream data units is controlled;

[0187] According to the order of splicing the audio stream data units, the received audio stream data units are spliced to generate the current audio stream data unit and refreshed;

[0188] According to the current audio stream data unit, the streaming recognition is converted into the result of interactive feedback, and according to the refresh of the current audio stream data unit, the interactive feedback result of the streaming recognition is refreshed and sent to the vehicle end.

[0189] In one embodiment, each audio stream data unit can be sent into the ASR service individually, and the result of streaming recognition is output based on the audio stream data unit, which is constantly fed back and refreshed to the display screen of the vehicle end.

[0190] For example, the user's voice is a request of "what is the weather tomorrow", based on the user's voice, the audio stream data unit is divided into "tomorrow", "weather" and "how". The ASR service returns the results of "tomorrow", "tomorrow's weather" and "tomorrow's weather". The user decides the appropriate end of the voice output, and the ASR service continuously follows the user's voice output process until the user ends the voice output, and refreshes the obtained results corresponding to the voice output that the user wants to end.

[0191] In the embodiment, further comprising:

[0192] The speed control rule of the audio stream data is set, and the time interval of the audio stream data unit of the input stream recognition process is controlled according to the rule;

[0193] The speed feature information is obtained.

[0194] The lower limit threshold of the time interval of the audio stream data unit of the input stream recognition process is set according to the speed feature information.

[0195] The audio stream data unit queue is buffered according to that the time interval of the received audio stream data unit is less than the time interval lower limit threshold.

[0196] The audio stream data unit is obtained by reading the buffered audio stream data unit queue according to the time interval of the audio stream data unit of the input stream recognition process being greater than or equal to the time interval lower limit threshold, and the audio stream data unit is marked ID sequence.

[0197] In one embodiment, the speed of the person has predictable and measurable characteristics. According to the speed feature information, the lower limit threshold of the time interval of the audio stream data unit of the input stream recognition process is set, that is, the speed of the person is matched, and the stream recognition process is designed, such as the size of the audio stream data unit,

[0198] For example, the time interval of the generated audio stream data unit, for example, the time interval of the transmitted audio data unit, for example, the rate of splicing the audio data unit, etc.

[0199] When the vehicle end user requests to connect, a thread in the forwarding thread pool is allocated according to the minimum connection number to serve the audio forwarding task of the vehicle end user, and the thread binds a delay queue for adjusting the sending interval of the user audio packet data. Multiple users share the thread to complete the forwarding of the audio packet data.

[0200] Figure 2 is the structural diagram of the stream voice information processing device provided by one or more embodiments of the application. For example Figure 2The shown stream voice information processing device comprises a long connection module, a start instruction module, a data reporting module, a data forwarding module, a permission adding module, an end instruction module, and a result returning module.

[0201] The long connection module is used for creating a long connection.

[0202] The start instruction module is used for a stream voice start instruction step.

[0203] The data reporting module is used for a stream audio packet data receiving step.

[0204] The data forwarding module is used for a stream audio packet data forwarding step.

[0205] The permission adding module is used for a forwarding delay event adding permission opportunity flow conversion step.

[0206] The end instruction module is used for a stream voice end signaling step.

[0207] The result returning module is used for a result returning step.

[0208] It is worth noting that although the system only discloses the long connection module, the start instruction module, the data reporting module, the data forwarding module, the permission adding module, the end instruction module, and the result returning module, it does not mean that the device is limited to the above basic function modules. On the contrary, the meaning expressed by the present application is that on the basis of the above basic function modules, a person skilled in the art can add one or more function modules to form an infinite number of embodiments or technical solutions in combination with the prior art. That is to say, the system is open rather than closed, and the protection scope of the present application claimed in the present embodiment cannot be limited to the above disclosed basic function modules.

[0209] Through the above scheme, the following beneficial technical effects are obtained:

[0210] The present application forms a data segment with a relatively fixed data size by segmenting audio stream data, adapts the resources and capabilities of stream recognition, and makes the stream recognition smooth and less likely to cause a lagging feeling.

[0211] The present application controls the rate of audio stream data input into the stream recognition process, prevents a large amount of data from flowing in within a short time, causes insufficient stream recognition resources, and causes processing to be not timely, and affects the output effect.

[0212] The present application registers the input end of the audio stream data, adjusts the uneven state of data transmission through a forwarding thread, and maintains the working saturation of stream recognition.

[0213] This application enables data processing to be performed in an ordered data queue by marking the audio stream data units (audio packet data) with IDs.

[0214] This application continuously splices audio stream data units to form new audio stream data units, keeping the user's response feedback in a constantly refreshed state, matching the voice signals continuously emitted by the user, and controlling the interaction rhythm under the user's voice output state.

[0215] Figure 3 A schematic diagram of a streaming speech recognition process according to a specific embodiment of the present invention.

[0216] Figure 4 A schematic diagram of the forwarding channel state transition mechanism according to a specific embodiment of the present invention.

[0217] Figure 5 A schematic diagram of the forwarding state transition mechanism of a specific embodiment of the present invention.

[0218] In one specific embodiment, such as Figure 3 The streaming speech recognition process shown decouples the process into three functional parts: audio reception management, audio forwarding management, and streaming recognition result processing. Furthermore, combining the scenario information of these three parts, two contexts are extracted: an audio reception context and an audio forwarding context. To ensure the orderly operation of each function, multiple forwarding states and multiple ASR channel states are abstracted.

[0219] In another specific embodiment, to improve the handling of the problem of audio packet data arrival time uncertainty causing the failure of forwarding event creation, a thread-shared variable is set to transfer the creation right of audio forwarding events.

[0220] like Figure 4The forwarding channel state transition mechanism shown includes: the initial state is no forwarding long connection (NO_CONN), the forwarding long connection is created successfully and enters the idle state (IDLE), at this time the forwarding long connection channel is available, when the first audio packet is forwarded to the ASR service via the channel, the channel jumps to the audio forwarding state (SENDING), the audio packet continues to be sent, and the state remains in the audio forwarding state (SENDING). If the channel is abnormally disconnected at this time, it jumps to the no connection state (NO_CONN), the current audio forwarding is terminated, and the current requested audio packet data is cleared. When the first ASR recognition result is returned, the forwarding channel state switches to the duplex communication state (DUPLEX), at this time, the audio sending or intermediate recognition result return event keeps the channel in the present state. If the channel is abnormally disconnected at this time, it jumps to the no connection state (NO_CONN), the current audio forwarding is terminated, and the current requested audio packet data is cleared. In general, when the channel is in the duplex communication state, it is possible that the audio is sent first, but the final recognition result is not returned, which will jump to the audio sending complete state (SENDING_COMPLETE). It is also possible that the final recognition result is returned, but the audio is not sent completely, at this time the state switches to the final recognition result response but not sent completely state (FINAL_ASR_RESP_NOT_SENT_COMPLETELY). During the session, if the forwarding channel state is in the no forwarding long connection (NO_CONN), the connection is triggered to be rebuilt.

[0221] In the embodiment, as Figure 5The forwarding state transition mechanism shown includes: the forwarding state mechanism combined with the pending forwarding queue mechanism mainly deals with the accumulation of voice requests, such as the request or response of the previous round or several rounds of voice requests not being completed, relying on this mechanism to solve the accumulation problem, and the accumulation is limited by the queue size to set the maximum accumulation round. The initial state is that the forwarding channel is not connected (CHANNEL_DISCONNECT), after the forwarding connection is successfully connected, if the number of pending forwarding is less than or equal to 1, the forwarding state is switched to the ready state (READY), and if the number of pending forwarding is greater than 1, the state is switched to the waiting state (WAITING). In the ready state (READY), when the first audio packet is received, a forwarding event is created, and at this time the forwarding state is switched to the running state (RUNNING). In the running state (RUNNING), there is a round of unfinished voice request in the forwarding queue, at this time if the next round of audio request arrives, the state is switched to the waiting state (WAITING), and in this state the voice request round that arrives will only store the audio packet data into the user data queue, and will not create a forwarding event, which is the accumulation phenomenon mentioned above. When the voice request round at the head of the pending forwarding queue completes the sending, it will be dequeued from the pending forwarding queue, and the forwarding state transition and the creation of the forwarding event are determined by combining the forwarding channel state, the user audio packet data queue, and the pending forwarding queue; if there is only one voice request in the pending forwarding queue and the forwarding channel, the state is switched to (READY), and if the size of the pending forwarding queue is greater than one, it still stays in the waiting state (WAITING), and a forwarding event is created again, and the audio packet data of the next round of voice request of the user is sent.

[0222] In another specific embodiment, an event-driven non-blocking thread model is provided, which realizes an efficient processing method for parallel processing of streaming audio reception, forwarding, and streaming return of recognition results. Improvements are made based on the problems existing in the streaming voice recognition process:

[0223] 1. Network congestion, user audio packet data arrives in the cloud, and cannot rely on the arrival interval of the data sequence to determine the sending time. The data forwarding event mechanism is designed to establish a delay event for the next data to be sent to determine its sending time.

[0224] 2. For the high resource consumption problem of traditional one user one forwarding thread, a group of thread groups is used to realize that multiple users share a forwarding thread in a thread group through a registration mechanism, improve the thread resource utilization efficiency, and reduce resource consumption.

[0225] 3. When there is "data vacuum" (when the previous group of data sending is completed and the next data has not arrived), an event is set to establish a trigger reverse mechanism to dynamically respond and reduce event idling.

[0226] 4. Fusion audio packet receiving, forwarding and stream identification result receiving whole process, set up state machine mechanism to solve request congestion.

[0227] Figure 6 The electronic device structure block diagram of the stream voice information processing method provided by one or more embodiments of the application.

[0228] As shown in Figure 6 The application provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus.

[0229] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the stream voice information processing method.

[0230] The application further provides a computer readable storage medium, which stores a computer program executable by the electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the stream voice information processing method.

[0231] The application further provides a data platform, comprising:

[0232] The electronic device is used to implement the steps of the stream voice information processing method.

[0233] The processor runs the program, and when the program runs, the data output from the electronic device executes the steps of the stream voice information processing method.

[0234] The storage medium is used to store the program, and when the program runs, the data output from the electronic device executes the steps of the stream voice information processing method.

[0235] The communication bus mentioned in the above electronic device can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0236] The electronic device includes a hardware layer, an operating system layer running above the hardware layer, and an application layer running on the operating system. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory. The operating system can be any one or more computer operating systems that implement control of the electronic device through a process, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a windows operating system. In embodiments of the present application, the electronic device can be a handheld device such as a smartphone or a tablet computer, or an electronic device such as a desktop computer or a portable computer, and is not particularly limited in embodiments of the present application.

[0237] The execution subject of the electronic device control in embodiments of the present application can be an electronic device or a functional module in the electronic device that can call and execute a program. The electronic device can obtain firmware corresponding to the storage medium, which is provided by a vendor. The firmware corresponding to different storage media can be the same or different, and is not limited herein. After the electronic device obtains the firmware corresponding to the storage medium, the electronic device can write the firmware corresponding to the storage medium into the storage medium, specifically by burning the firmware corresponding to the storage medium into the storage medium. The process of burning the firmware into the storage medium can be implemented using existing technology, and is not described in detail in embodiments of the present application.

[0238] The electronic device can also obtain a reset command corresponding to the storage medium, which is provided by a vendor. The reset command corresponding to different storage media can be the same or different, and is not limited herein.

[0239] At this time, the storage medium of the electronic device is a storage medium into which the corresponding firmware is written, and the electronic device can respond to the reset command corresponding to the storage medium in the storage medium into which the corresponding firmware is written, so that the electronic device resets the storage medium into which the corresponding firmware is written according to the reset command corresponding to the storage medium. The process of resetting the storage medium according to the reset command can be implemented using existing technology, and is not described in detail in embodiments of the present application.

[0240] For the convenience of description, the above-described apparatus is described as various units and modules in terms of functions. Of course, the functions of the units and modules can be implemented in one or more software and / or hardware in the implementation of the present application.

[0241] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art in the field of the present application. It should also be understood that terms such as those defined in a generally used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and should not be interpreted in an idealized or overly formal sense unless specifically so defined.

[0242] For the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited by the action sequence described, because according to the embodiments of the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential for the embodiments of the present application.

[0243] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware platform. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments of the present application.

[0244] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some or all of the technical features; and such modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for processing streaming speech information, characterized in that, The streaming voice information processing method includes: Steps to create a persistent connection; Steps to start streaming voice; The steps of receiving streaming audio packet data further include marking each audio packet data with an ID, and performing data processing in order of ID under an ordered data queue; The steps of streaming audio packet data forwarding; The step of adding permissions for forwarding delay events and transferring them at an opportune time further includes: when the audio packet data queue is empty, marking the permission to add forwarding events and transferring the permission to add forwarding events to the receiving side of the audio packet data; when the audio packet data queue still has data, continuing to add audio forwarding delay events into the event queue. The steps for ending a streaming voice call; The step of returning the recognition result further includes forming new audio stream data units by continuously splicing audio stream data units, keeping the user's response feedback in a constantly refreshed state, and matching the voice signals continuously emitted by the user.

2. The streaming speech information processing method according to claim 1, characterized in that, The steps for creating a long-lived connection include: Create a user context; Request audio forwarding thread resources based on the user creation context; Based on the requested audio forwarding thread resources, allocate thread resources evenly. The balanced allocation of thread resources includes assigning tasks to the thread with the fewest number of users among multiple threads; The user context includes an audio receiving context and an audio forwarding context. The audio receiving context includes a data queue, current user-side page hot words, and current user-side status. The audio forwarding context includes a forwarding delay event queue, added forwarding event right-sharing variables, and audio forwarding channel status.

3. The streaming speech information processing method according to claim 1, characterized in that, The steps for initiating the streaming voice command include: Audio request resource preparation; The audio request resource preparation includes checking whether the forwarding channel is available; If unavailable, create a forwarding channel; The creation of the corresponding forwarding channel includes the following states: the status flag is set to receive state, the initial state of the channel is a long connection without forwarding, and it enters an idle state after successful creation.

4. The streaming speech information processing method according to claim 1, characterized in that, The steps for receiving streaming audio packet data include: Determine whether audio packet data needs to be added to the data queue based on the status of the audio forwarding channel; If necessary, the audio group data is enqueued and the status of the shared variable for adding forwarding events is used to determine whether a forwarding event needs to be added. If necessary, add the audio forwarding delay event to the event queue; If not needed, discard all audio group data for the current round.

5. The streaming speech information processing method according to claim 1, characterized in that, The steps of forwarding streaming audio packet data include: Based on the forwarding thread waiting for the forwarding event to arrive, when the forwarding delay event arrives, the user context information contained in the event is obtained, and the user's data queue is checked to see if there is audio packet data to be sent; If so, check if it is the first audio group data in the queue; If yes, then update the forwarding status to forwarding; If not, check the state in the context to update the current forwarding state, and retrieve the audio packet data from the data queue for forwarding; It also includes the step of adding permissions to forwarding delayed events and allowing them to be transferred at an opportune time; The steps for adding permissions to the forwarding delay event for opportune transfer include checking again whether there is still data in the user's audio group data queue; If it still exists, add the audio forwarding delay event to the event queue; If not, mark the permission to add forwarding events and transfer that permission to the receiving side of the audio packet data.

6. The streaming speech information processing method according to claim 1, characterized in that, The steps of the streaming voice termination signaling include: Query the current audio forwarding status; The query of the current audio forwarding status includes querying the status of the audio forwarding context; Determine whether the current audio forwarding status requires adding the end audio packet data to the data queue; If necessary, the audio group data is enqueued and it is determined whether the next round of forwarding events needs to be added; The audio packet data enqueueing includes enqueuing the audio receiving context data queue data; If necessary, add the audio forwarding delay event to the queue; The addition of audio forwarding delay events to the queue includes enqueuing forwarding events in the event queue of the audio forwarding context.

7. The streaming speech information processing method according to claim 1, characterized in that, The steps for returning the recognition result include: Obtain information from the streaming recognition response; Based on the information from the streaming recognition response, determine whether the streaming recognition result is normal; If normal, then determine whether the streaming recognition result is a successful connection creation result, an intermediate recognition result, or a final recognition result; If the result is a successful connection creation, then the forwarding channel will be put into an available state according to the preset successful connection creation result handling strategy. If it is an intermediate identification result, then process the streaming identification result according to the preset intermediate identification result processing strategy and set the channel status to bidirectional communication status. If it is the final recognition result, then the streaming recognition result is processed according to the preset final recognition result processing strategy, and the channel status flag is set to the recognition end state, idle state, or recognition completed but transmission not completed state according to the corresponding audio data reception and forwarding status.

8. The streaming speech information processing method according to claim 2, characterized in that, Also includes: Initiate the creation of a long-lived connection based on the user creation context; Upon successful creation of the long connection, register the ASR result processor. Upon successful creation of a long connection and registration of an ASR result handler, the callback updates the user context and sets the channel status to available.

9. The streaming speech information processing method according to claim 5, characterized in that, Also includes: Obtain information about the arrival time of the audio forwarding delay event; The information regarding the arrival time of the audio forwarding delay event includes the expiration of the event queue in the audio forwarding context and the event being dequeued. Check if the corresponding audio packet data exists in the user data queue based on the information of the audio forwarding delay event arrival time; If it exists, check if the forwarding channel is available; If available, forward the audio data and update the audio forwarding context state information; Determine whether to add a next round of forwarding events based on the forwarded audio data and the updated audio forwarding context status. If added, the audio data will be forwarded and the audio forwarding context state will be updated.

10. The streaming speech information processing method according to claim 9, characterized in that, Also includes: Based on the information about the arrival time of the audio forwarding delay event, check if there is any corresponding audio group data, and transfer the permission to create a forwarding event; Based on the information of forwarding audio data and updating the audio forwarding context state, it is determined not to add a next round of forwarding events, and the permission to create forwarding events is transferred to the receiving side of the audio group data; If the forwarding channel is found to be unavailable, rebuild the connection.

Images