Method, device, medium and equipment for streaming recognition of voice instructions

Abstract

Embodiments of the present application provide a speech instruction streaming recognition method and device, a computer readable storage medium and an electronic device, and relate to the technical field of artificial intelligence. The method comprises: performing speech activity detection on the to-be-processed audio to determine a speech activity sub-paragraph in the to-be-processed audio, wherein the speech activity sub-paragraph is audio between a start state and a quasi-end state obtained in sequence by detection; in the process of performing speech recognition on the current ith speech activity sub-paragraph, in response to detecting the quasi-end state, an ith streaming speech recognition result corresponding to the ith speech activity sub-paragraph is obtained, wherein i is a positive integer not greater than the total number of speech activity sub-paragraphs in the to-be-processed audio; and the ith streaming speech recognition result is input into a natural language processing (NLP) model, wherein the NLP model outputs an ith instruction corresponding to the ith speech activity sub-paragraph. The embodiments of the present application can improve the instruction recognition accuracy while saving computing resources.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of artificial intelligence, and in particular, to a method and device for streaming recognition of voice instructions, a computer readable storage medium, and an electronic device. BACKGROUND

[0002] Speech recognition (SR) refers to a technology of converting speech into text. One application of speech recognition includes a streaming recognition system for multiple instructions. Specifically, the input information of the system can be an audio containing multiple instructions, and the system outputs the multiple instructions after recognizing the audio, rather than recognizing the next instruction after outputting a single instruction. For example, a driver can say, “turn on seat massage, turn on air conditioner, and turn up volume”, and a one-sentence multi-instruction recognition system can recognize the three instructions and execute them in sequence.

[0003] In the related art, for a user's voice input, a speech activity detection is first performed to obtain a speech activity paragraph, and further, speech recognition is performed on each speech activity paragraph. Specifically, in order to improve the response speed, the SR model timely issues a streaming speech recognition result to a natural language processing (NLP) model as soon as it recognizes part of the speech of a speech activity paragraph, so that the NLP model extracts instructions from each received speech recognition result. For example, the user's voice is “turn off sunroof and turn on air conditioner”, and the speech recognition result issued to the natural language processing model each time is “off”, “turn off”, “turn off sun”, “turn off sunroof”, “turn off sunroof and turn on”, “turn off sunroof and turn on air”, and “turn off sunroof and turn on air conditioner” in sequence.

[0004] However, the related art has the problem of consuming a large amount of computing resources. SUMMARY

[0005] Embodiments of the present application provide a method and device for streaming recognition of voice instructions, a computer readable storage medium, and an electronic device, which can improve the accuracy of instruction recognition while saving computing resources.

[0006] In a first aspect, an embodiment of the present application provides a streaming recognition method of a voice instruction, the method comprising: performing voice activity detection on a to-be-processed audio to determine a voice activity sub-paragraph in the to-be-processed audio, wherein the voice activity sub-paragraph is audio between a start state and a quasi-end state obtained in sequence through detection; in a process of performing voice recognition on a current i-th voice activity sub-paragraph, in response to detecting the quasi-end state, obtaining an i-th streaming voice recognition result corresponding to the i-th voice activity sub-paragraph, wherein i is a positive integer not greater than a total number of voice activity sub-paragraphs in the to-be-processed audio; and inputting the i-th streaming voice recognition result into a natural language processing (NLP) model, wherein the NLP model outputs an i-th instruction corresponding to the i-th voice activity sub-paragraph.

[0007] In a second aspect, an embodiment of the present application provides a streaming recognition device of a voice instruction, the device comprising: a voice detection module, a streaming result acquisition module, and an instruction extraction module.

[0008] The voice detection module is configured to perform voice activity detection on a to-be-processed audio to determine a voice activity sub-paragraph in the to-be-processed audio, wherein the voice activity sub-paragraph is audio between a start state and a quasi-end state obtained in sequence through detection. The streaming result acquisition module is configured to, in a process of performing voice recognition on a current i-th voice activity sub-paragraph, in response to detecting the quasi-end state, obtain an i-th streaming voice recognition result corresponding to the i-th voice activity sub-paragraph, wherein i is a positive integer not greater than a total number of voice activity sub-paragraphs in the to-be-processed audio. The instruction extraction module is configured to input the i-th streaming voice recognition result into a natural language processing (NLP) model, wherein the NLP model outputs an i-th instruction corresponding to the i-th voice activity sub-paragraph.

[0009] In an exemplary embodiment, based on the above scheme, the voice detection module is specifically configured to: input the to-be-processed audio into a voice activity detection (VAD) processing module; in a case where a feature of an audio signal detected by the VAD processing module satisfies a first preset condition, output a start state; and in a case where the feature of the audio signal detected by the VAD processing module does not satisfy the first preset condition and lasts for a first preset time length, output a quasi-end state.

[0010] The first preset time length is less than a first threshold.

[0011] The feature of the audio signal comprises at least one of the following: an energy level of the audio signal, a spectral feature, and a change characteristic in a time domain.

[0012] In the example embodiment, based on the above scheme, the streaming result acquisition module includes a comparison unit and a first acquisition unit. The comparison unit is configured to, in the process of performing speech recognition on the current i-th speech activity sub-paragraph, continue to compare the subset of speech recognition results that have been generated and have not been input into the NLP model with a second preset condition before a quasi-end state is detected. The first acquisition unit is configured to, if the comparison result is that the subset of speech recognition results does not satisfy the second preset condition, acquire the i-th streaming speech recognition result corresponding to the i-th speech activity sub-paragraph in response to detecting the quasi-end state.

[0013] In the example embodiment, based on the above scheme, the first acquisition unit is specifically configured to acquire the speech recognition result of the i-th speech activity sub-paragraph to obtain current sub-paragraph speech information, acquire all speech recognition results of the audio to be processed that have been input into the NLP model to obtain context speech information of the current sub-paragraph, and combine the current sub-paragraph speech information and the context speech information of the current sub-paragraph to obtain the i-th streaming speech recognition result.

[0014] In the example embodiment, based on the above scheme, the streaming result acquisition module further includes a second acquisition unit.

[0015] The second acquisition unit is configured to, if the comparison result is that the subset of speech recognition results satisfies the second preset condition, acquire the i-th streaming speech recognition result corresponding to the subset of speech recognition results. The i-th streaming speech recognition result is input into the natural language processing (NLP) model, and the NLP model outputs one instruction in the instruction set corresponding to the i-th speech activity sub-paragraph.

[0016] In the example embodiment, based on the above scheme, the second acquisition unit is specifically configured to acquire the current speech recognition result of the i-th speech activity sub-paragraph to obtain current subset speech information in the i-th speech activity sub-paragraph, acquire all speech recognition results of the audio to be processed that have been input into the NLP model to obtain context speech information of the current subset, and combine the current subset speech information and the context speech information of the current subset to obtain the i-th streaming speech recognition result.

[0017] In the example embodiment, based on the above scheme, the type of the speech recognition result is a text string, and the second preset condition includes that the length of the text string is greater than a second threshold.

[0018] In the example embodiment, based on the above scheme, the device further includes a semantic completeness detection module.

[0019] The semantic integrity detection module is configured to perform semantic integrity detection on the i-th stream speech recognition result before the instruction extraction module inputs the i-th stream speech recognition result into the NLP model, to determine whether the semantic integrity of the i-th stream speech recognition result is greater than a third threshold value; and in a case where the semantic integrity is greater than the third threshold value, input the i-th stream speech recognition result into the NLP model.

[0020] In an example embodiment, based on the above scheme, the semantic integrity detection module is further configured to, in a case where the semantic integrity is less than or equal to the third threshold value, pause inputting information into the NLP model, and wait for performing semantic integrity detection on the i+1-th stream speech recognition result; and in a case where the semantic integrity does not meet the requirement continuously for a fourth threshold value of times, adjust the third threshold value according to a preset strategy, to perform semantic integrity detection again based on the adjusted third threshold value; and the adjusted third threshold value is greater than a fifth threshold value.

[0021] In an example embodiment, based on the above scheme, the semantic integrity detection module is specifically configured to input the i-th stream speech recognition result into a large language model trained, to perform semantic integrity analysis on the i-th stream speech recognition result by the large language model.

[0022] In an example embodiment, based on the above scheme, the apparatus further includes an execution module, and the execution module is configured to execute the i-th instruction in a process in which the instruction extraction module identifies a subsequent instruction of the i-th instruction, and / or generate response information corresponding to the to-be-processed audio according to the i-th instruction in the process in which the instruction extraction module identifies the subsequent instruction of the i-th instruction, and output the response information.

[0023] In a third aspect, an embodiment of the present application provides an electronic device including a processor and a memory. The memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory, to execute the speech instruction stream recognition method provided in the first aspect.

[0024] In a sixth aspect, an embodiment of the present application provides a chip for implementing the speech instruction stream recognition method provided in the first aspect or the second aspect. Specifically, the chip includes a processor configured to invoke and run a computer program from a memory, so that a device installed with the chip executes the speech instruction stream recognition method provided in the first aspect.

[0025] In a seventh aspect, an embodiment of the present application provides a computer readable storage medium for storing a computer program, and the computer program causes a computer to perform the speech instruction stream recognition method provided in the first aspect.

[0026] In an eighth aspect, an embodiment of the present application provides a computer program product comprising computer program instructions, and the computer program instructions cause a computer to perform the speech instruction stream recognition method provided in the first aspect.

[0027] In a ninth aspect, an embodiment of the present application provides a computer program which, when running on a computer, causes the computer to perform the speech instruction stream recognition method provided in the first aspect.

[0028] To sum up, in the speech instruction stream recognition scheme provided in the embodiments of the present application, speech activity detection is performed on the audio to be processed to determine a speech activity sub-paragraph in the audio to be processed, wherein the speech activity sub-paragraph is the audio between the start state and the quasi-end state obtained in sequence through detection. Further, in the process of performing speech recognition on the current i-th speech activity sub-paragraph, in response to the detection of the quasi-end state, the i-th stream speech recognition result corresponding to the i-th speech activity sub-paragraph is obtained. The i-th stream speech recognition result is input into a natural language processing (NLP) model, wherein the NLP model outputs the i-th instruction corresponding to the i-th speech activity sub-paragraph. It can be seen that in the embodiments of the present application, the SR model takes the speech activity sub-paragraph as a unit, and the speech recognition result is input into the NLP model, so that the NLP model identifies the semantic recognition result corresponding to the speech activity sub-paragraph in each identification process, thereby identifying multiple instructions contained in the audio to be processed. Compared with the related art, in the embodiments of the present application, the NLP model extracts instructions in units of speech activity sub-paragraphs, which can reduce the computational amount of the NLP model and is conducive to reducing resource occupation. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0030] Figure 1 The system architecture diagram of the application environment of the speech instruction stream recognition scheme provided in an embodiment of the present application;

[0031] Figure 2 The flowchart of the speech instruction stream recognition method provided in an embodiment of the present application;

[0032] Figure 3 A flowchart of a speech instruction stream recognition method provided by an embodiment of the present application;

[0033] Figure 4A A VAD detection diagram for a sentence audio containing multiple instructions provided by the related art;

[0034] Figure 4B An improved VAD detection diagram for a sentence audio containing multiple instructions provided by an embodiment of the present application;

[0035] Figure 5A A diagram for determining a stream speech recognition result provided by an embodiment of the present application;

[0036] Figure 5B A diagram for determining a stream speech recognition result provided by the related art;

[0037] Figure 6 A flowchart of a method for determining an i-th stream speech recognition result provided by an embodiment of the present application;

[0038] Figure 7 A flowchart of a method for detecting semantic completeness of a stream speech recognition result provided by an embodiment of the present application;

[0039] Figure 8 A flowchart of a method for processing an identified instruction provided by an embodiment of the present application;

[0040] Figure 9 A flowchart of a speech instruction stream recognition method provided by an embodiment of the present application;

[0041] Figure 10 A schematic block diagram of a speech instruction stream recognition apparatus provided by an embodiment of the present application;

[0042] Figure 11 A schematic block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0044] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In the embodiments of the present application, "B corresponding to A" means that B is associated with A. In an implementation, B can be determined according to A. However, it should also be understood that determining B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices. In the description of the present application, "a plurality of" means two or more than two, unless otherwise specified.

[0045] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0046] In the above related technologies of streaming recognition of voice instructions, the SR model sequentially issues each of all streaming results, and the NLP model needs to perform semantic analysis on all streaming results to attempt to extract valid instructions. Due to the long voice in the one-sentence multi-instruction scenario, the number of streaming results is very high, the computational load of the NLP model is relatively large, and the network bandwidth consumption increases. At the same time, if the streaming result is refreshed incorrectly, the NLP model may extract an incorrect instruction, such as "increase the temperature of the air conditioner", which is recognized as "increase the air speed of the air conditioner", thereby causing the instruction to be executed incorrectly and affecting the overall system effect.

[0047] Regarding the recognition technology of one-sentence multi-instructions, the related technology further provides another voice instruction recognition scheme. Specifically, for the voice input of a user, a voice activity detection (VAD) is performed to obtain the voice activity paragraphs therein, and further, voice recognition is performed on each voice activity paragraph. Specifically, in order to guarantee the instruction recognition accuracy, after the voice recognition model recognizes the entire voice activity paragraph, the voice recognition result of the entire paragraph is sent to a natural language understanding model, so that the NLP model extracts instructions from the voice recognition result of the entire paragraph, thereby being able to determine multiple instructions in the voice at one time in combination with the context of each instruction. For example, the voice of the user is "close the sunroof and turn on the air conditioner", and the voice recognition result sent to the natural language understanding model is "close the sunroof and turn on the air conditioner". Although the related technology extracts multiple instructions at one time, there is a problem of low response speed.

[0048] In view of the above technical problems existing in the related technology, VAD is performed on the to-be-processed audio to determine voice activity sub-paragraphs in the to-be-processed audio, wherein the voice activity sub-paragraphs are audio between the start state and the quasi-end state obtained in sequence through detection. Further, in the process of performing voice recognition on the current ith voice activity sub-paragraph, in response to detecting the quasi-end state, an ith streaming voice recognition result corresponding to the ith voice activity sub-paragraph is obtained. The ith streaming voice recognition result is input into a natural language processing (NLP) model, wherein the NLP model outputs an ith instruction corresponding to the ith voice activity sub-paragraph. It can be seen that in the embodiment of the present application, the SR model takes the voice activity sub-paragraph as a unit, and inputs the voice recognition result into the NLP model, so that the NLP model recognizes the semantic recognition result corresponding to the voice activity sub-paragraph at each time of recognition, thereby recognizing multiple instructions contained in the to-be-processed audio. Compared with the related technology, in the embodiment of the present application, the NLP model extracts instructions in units of voice activity sub-paragraphs, which can reduce the computational amount of the NLP model and is beneficial to reducing resource occupation. At the same time, since there is often a slight pause between multiple instructions when issuing one-sentence multi-instructions, the quasi-end state is detected in the embodiment of the present application to simulate the pause between multiple instructions in one sentence. Therefore, the NLP model extracts instructions in units of voice activity sub-paragraphs, which is also beneficial to improving the instruction extraction accuracy.

[0049] The following describes an example of the system architecture of the present application. Figure 1 The example system architecture of the present application is introduced.

[0050] The example system architecture of the present application is introduced. Figure 1 The system architecture 100 of the application environment of the streaming recognition scheme of the voice instruction provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the system architecture 100 includes a voice input unit 101, a voice recognition unit 102, a natural language processing (NLP) unit 103, and a response unit 104. Figure 1As shown, the system architecture of the implementation environment of the embodiments of the present application can include: a terminal 102 and a cloud server 104, and the terminal and the cloud server can be connected through a network. Among them, the streaming speech recognition system of the voice instruction can be deployed on the terminal 102, or can be deployed on the cloud server 104.

[0051] For example, the terminal 102 can be used to collect audio, and / or used to play audio. For example, the terminal 102 includes a vehicle terminal, a computer, a smart phone, a tablet, a smart voice interaction device, a smart home appliance, an aircraft, a wearable smart device, a medical device, etc., but is not limited thereto. The cloud server 104 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers.

[0052] In the exemplary embodiments, with reference to Figure 1 , the terminal 102 with audio collection function collects the audio to be processed. For example, a vehicle terminal or a mobile phone terminal with microphone function, etc.

[0053] For example, with reference to Figure 2 , in the case where the streaming speech recognition system of the voice instruction is deployed on the terminal 102, after the terminal 102 collects the audio to be processed 20, the voice activity detection is performed on the audio to be processed by the VDA processing module 210 in the local streaming speech recognition system of the voice instruction, to determine the voice activity sub-paragraphs in the audio to be processed, such as the i-th voice activity sub-paragraph 22 in Figure 2 . Further, the speech recognition is performed on the current i-th voice activity sub-paragraph based on the SR model 220 in the local streaming speech recognition system of the voice instruction, and the speech recognition result 24 is output; in the recognition process, the speech recognition engine acquires the i-th streaming speech recognition result corresponding to the i-th voice activity sub-paragraph in response to the detection of the quasi-end state, such as the i-th streaming speech recognition result 26 in Figure 2 . Further, the speech recognition engine inputs the i-th streaming speech recognition result into the natural language processing NLP model 230, wherein the NLP model 230 outputs the i-th instruction corresponding to the i-th voice activity sub-paragraph, such as 28 in Figure 2 .

[0054] For example, with reference to Figure 2In the case that the streaming speech recognition system of the voice instruction is deployed on the cloud server 102, after the terminal 102 collects the audio to be processed 20, the terminal 102 transmits the audio to be processed 20 to the streaming speech recognition system of the voice instruction on the cloud, and further performs speech activity detection on the audio to be processed by the VDA processing module 210 to determine the speech activity sub-paragraphs in the audio to be processed. Further, the SR model 220 in the streaming speech recognition system of the voice instruction on the cloud performs speech recognition on the current ith speech activity sub-paragraph. In the recognition process, the speech recognition engine acquires the ith streaming speech recognition result 26 corresponding to the ith speech activity sub-paragraph in response to the detection of the quasi-end state. Further, the speech recognition engine inputs the ith streaming speech recognition result into the natural language processing NLP model 230, wherein the NLP model outputs the ith instruction 28 corresponding to the ith speech activity sub-paragraph.

[0055] It can be understood that part of the streaming speech recognition system of the voice instruction is deployed on the terminal 102, and the other part is deployed on the cloud server 104. In this case, the streaming speech recognition process of the voice instruction will be implemented based on the terminal and the cloud server. For example, the VAD processing module and the SR model are deployed on the terminal 102, and the NLP model is deployed on the cloud server 104. The embodiments of the present application are not limited thereto.

[0056] For example, the instruction obtained after the streaming speech recognition of the voice instruction can be output by the terminal 102.

[0057] It should be noted that Figure 1 The system architecture of the implementation environment of the embodiments of the present application is exemplarily shown, but the system architecture of the implementation environment of the embodiments of the present application is not limited to Figure 1 as shown.

[0058] The streaming speech recognition method of the voice instruction of the embodiments of the present application will be described in detail below. The following embodiments can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments.

[0059] Figure 3 A flowchart of a streaming speech recognition method P300 of a voice instruction provided by the embodiments of the present application is shown. The execution subject of the method P300 can be an electronic device, such as the server 104 or the terminal 102 as shown. Figure 1 .

[0060] In step S310, speech activity detection is performed on the audio to be processed to determine the speech activity sub-paragraphs in the audio to be processed, wherein the speech activity sub-paragraphs are the audio between the start state and the quasi-end state obtained in sequence by detection.

[0061] The audio to be processed can be audio emitted by a user when issuing an instruction, and the audio to be processed can include instruction information. For example, the audio to be processed can be collected by a microphone device. For example, the audio collected by the microphone can be processed by using an echo cancellation technology to obtain real-time multi-channel audio data, so as to extract a pure speech signal from a complex acoustic environment, thereby providing accurate and reliable input information for a subsequent simultaneous interpretation model.

[0062] The VAD technology is used to identify a speech active segment and a non-speech active segment in an audio signal. The speech active segment refers to audio containing speech information, and the non-speech active segment refers to audio not containing speech information. The VAD technology determines the speech active segment and the non-speech active segment in the audio signal by detecting a speech endpoint in the audio signal. In the related art, the following states can be determined by using the VAD technology: START (start state, indicating an endpoint at which speech information starts), CONTINUE (continuous state, indicating that speech information continues), END (end state, indicating an endpoint at which speech information terminates, but not indicating that the entire speech input has been completed), and FINISH (last state, indicating that the entire speech input has been completely ended, and there is no new speech activity). Specifically, audio between START and FINISH that appear in sequence can be determined as a speech active segment, and audio between FINISH and START that appear in sequence can be determined as a non-speech active segment.

[0063] Specifically, in the related art, when the length of the accumulated silence after the valid speech segment reaches a VAD gap, the END state is returned, indicating that the clause ends. The VAD gap is usually in a range of 400 ms to 600 ms. For example, the VAD gap is 400 ms. Figure 4A The time interval between the audio signals corresponding to instruction 1 (close the sunroof window) and instruction 2 (turn on the air conditioner) is 80 ms, which is less than the VAD gap, and thus the END state is not output between the audio signals corresponding to instruction 1 and instruction 2. The length of the accumulated silence after instruction 5 (I am hungry) is 480 ms, which is in the value range of the VAD gap, and thus the END state is output. Therefore, if the VAD technology provided in the related art is used, the five instructions can be divided into the same speech active segment.

[0064] However, when multiple instructions are issued in one sentence (for example, Figure 4A or Figure 4BThe five instructions shown), a small pause often naturally exists between instructions, and the pause between instructions can be used as a natural semantic segmentation point. Based on the audio between adjacent semantic segmentation points, instruction recognition will help improve instruction extraction accuracy. The pause between instructions is short, typically less than the VAD gap, and as described above, the VAD technology provided by the related art cannot divide different instructions into different speech activity paragraphs.

[0065] To solve the above problems, an improved VAD technology is provided in the embodiments of the present application. Specifically, the audio to be processed is input into the VAD processing module. When the VAD processing module detects that the characteristics of the audio signal meet the first preset condition, the start state is output, such as START in the related art. When the VAD processing module detects that the characteristics of the audio signal do not meet the first preset condition and continue for a first preset duration, the pre-end state (PreEND) is output.

[0066] Because the energy, frequency characteristics and changes in the time domain of the audio signal can be used to determine whether the audio paragraph contains speech activity, the characteristics of the audio signal used to determine whether the first preset condition is met in the embodiments of the present application can include at least one of the energy level, spectral characteristics, and change characteristics in the time domain of the audio signal. For example, if it is detected that the energy level of the a-th frame of audio signal exceeds the energy threshold and the energy level of at least one previous frame of audio signal does not exceed the energy threshold, the a-th frame of audio signal can be determined to correspond to the start state (START). If it is detected that the energy level of the b-th frame of audio signal does not exceed the energy threshold and the energy levels of the audio frames in the next first preset duration also do not exceed the energy threshold, the first preset duration is output as the pre-end state (PreEND). Because the pause between multiple instructions in a sentence is short, the VAD stream gap (i.e., the first preset duration) is introduced in the embodiments of the present application, and the value of the VAD stream gap is less than the VAD gap, so that a whole speech activity paragraph as shown in Figure 4A may be divided into multiple sub-paragraphs (specifically referred to as "speech activity sub-paragraphs"). Specifically, the value of the VAD stream gap is related to the actual needs, for example, the value of the VAD stream gap is 50 ms, but the embodiments of the present application do not limit the specific value of the VAD stream gap. For another example, the first preset duration VAD stream gap is a shorter duration than the VAD gap, for example, the first preset duration is less than a first threshold, and the value of the first threshold is exemplarily 200.

[0067] Thus, when the accumulated silence length after the valid speech segment reaches the VAD stream gap, the PreEND state is returned. Specifically, in this embodiment, the audio between the start state and the pre-end state obtained in sequence through detection is determined as a speech activity sub-paragraph. An exemplary reference Figure 4B The time interval between instruction 1 (close sunroof window) and instruction 2 (open air conditioner) corresponding to the audio signals is 80 milliseconds, which is less than the VAD gap but greater than the VAD stream gap (50 milliseconds), so the pre-end state 1 (PreEND-1) can be output, and it can be determined that instruction 1 "close sunroof window" belongs to the speech activity sub-paragraph 1; similarly, the time interval between instruction 2 (open air conditioner) and instruction 3 (close seat massage) corresponding to the audio signals is 60 milliseconds, which is greater than the VAD stream gap (50 milliseconds), so the pre-end state 2 (PreEND-2) can be output, and it can be determined that instruction 2 "open air conditioner" belongs to the speech activity sub-paragraph 2; and so on.

[0068] It can be seen that, compared with the related art, the improved VAD technology provided in the embodiments of the present application can achieve more refined segmentation of audio. In a one-sentence multi-instruction scenario, if the related art is used, multiple instructions in the sentence can be divided into the same speech activity paragraph, as shown in Figure 4A If the improved VAD technology provided in the embodiments of the present application is used, different instructions in the sentence can be divided into different speech activity sub-paragraphs, as shown in Figure 4B

[0069] Referring back to Figure 3 In step S320, in the process of speech recognition on the current ith speech activity sub-paragraph, in response to detecting the pre-end state, an ith stream speech recognition result corresponding to the ith speech activity sub-paragraph is obtained, where i is a positive integer not greater than the total number of speech activity sub-paragraphs in the audio to be processed.

[0070] In the scheme provided in the embodiments of the present application, the audio to be processed is detected through the improved VAD technology described above, and speech activity sub-paragraphs can be sequentially determined from the input audio as the audio is continuously input. As described above, in a one-sentence multi-instruction scenario, there is a natural small pause between different instructions, and the pre-end state can be used to accurately locate the division point between instructions, so as to obtain the speech activity sub-paragraph corresponding to the instruction.

[0071] ​In the following embodiments of the present application, the streaming result of the input NLP model is determined in units of voice activity sub-paragraphs. Specifically, in the process of performing speech recognition on the current i-th voice activity sub-paragraph, in response to detecting the quasi-end state PreEND, indicating that the current sub-paragraph reaches the segmentation point with the next field, the i-th streaming speech recognition result corresponding to the i-th voice activity sub-paragraph can be obtained. Specifically, in response to the detection of the quasi-end state, one implementation of determining the i-th streaming recognition result is as follows:

[0072] In step S11, the speech recognition result of the i-th voice activity sub-paragraph is obtained to obtain the current sub-paragraph speech information. An exemplary reference Figure 5A The speech recognition result of the voice activity sub-paragraph 1 is "close sunroof and window", i.e. the voice activity sub-paragraph 1 is "close sunroof and window"; the speech recognition result of the voice activity sub-paragraph 2 is "turn on air conditioner", i.e. the voice activity sub-paragraph 2 is "turn on air conditioner"; the speech recognition result of the voice activity sub-paragraph 3 is "turn off seat massage", i.e. the voice activity sub-paragraph 2 is "close sunroof and window"; and so on.

[0073] In step S12, all speech recognition results of the input NLP model in the audio to be processed are obtained to obtain the context speech information of the current sub-paragraph. An exemplary reference Figure 5A For the voice activity sub-paragraph 1, since it is the first recognized sub-paragraph in the audio to be processed, there is no all speech recognition results of the input NLP model; for the voice activity sub-paragraph 2, it is the second recognized sub-paragraph in the audio to be processed, so the all speech recognition results of the input NLP model is "close sunroof and window", thus the context speech information of the sub-paragraph 2 is "close sunroof and window"; for the voice activity sub-paragraph 3, it is the third recognized sub-paragraph in the audio to be processed, so the all speech recognition results of the input NLP model is "close sunroof and window turn on air conditioner", thus the context speech information of the sub-paragraph 3 is "close sunroof and window close sunroof and window turn on air conditioner"; and so on.

[0074] In step S13, the current sub-paragraph speech information and the context speech information of the current sub-paragraph are merged to obtain the i-th streaming speech recognition result. An exemplary reference is made to FIG. 5. For the voice activity sub-paragraph 1, its speech recognition result is "close sunroof and window", and the context speech information of the current sub-paragraph is empty, so the first streaming speech recognition result is obtained as "close sunroof and window"; for the voice activity sub-paragraph 2, its speech recognition result is "turn on air conditioner", and the context speech information of the current sub-paragraph is "close sunroof and window", so the second streaming speech recognition result is obtained as "close sunroof and window turn on air conditioner"; and so on. An exemplary reference is made to Figure 5A :

[0075] In response to the VAD detecting the quasi-end state of "voice activity sub-paragraph 1", according to the speech recognition result of the current SR model, the corresponding streaming speech recognition result is determined as "close sunroof and window";

[0076] In response to the VAD detecting the quasi-end state of "voice activity sub-paragraph 2", according to the speech recognition result of the current SR model, the corresponding streaming speech recognition result is determined as "close sunroof and window, open air conditioner";

[0077] In response to the VAD detecting the quasi-end state of "voice activity sub-paragraph 3", according to the speech recognition result of the current SR model, the corresponding streaming speech recognition result is determined as "close sunroof and window, open air conditioner, close seat massage";

[0078] In response to the VAD detecting the quasi-end state of "voice activity sub-paragraph 4", according to the speech recognition result of the current SR model, the corresponding streaming speech recognition result is determined as "close sunroof and window, open air conditioner, close seat massage, set air conditioner to 25 degrees";

[0079] In response to the VAD detecting the quasi-end state of "voice activity sub-paragraph 5", according to the speech recognition result of the current SR model, the corresponding streaming speech recognition result is determined as "close sunroof and window, open air conditioner, close seat massage, set air conditioner to 25 degrees, I am hungry".

[0080] For example, if the five instructions in the above sentence are extracted by the related technology in the manner of inputting the streaming speech recognition result into the NLP model, the streaming speech recognition result is as follows: Figure 5BAs shown in FIG. 5A and FIG. 5B, in a first aspect, the first stream result inputted into the NLP model is "close", the second stream result inputted into the NLP model is "off", and so on. In the related art, the SR model identifies a small number (e.g., one word) of instructions, and then generates a stream result based on the identified instructions and inputs the stream result into the NLP model. The above sentence includes five instructions, which correspond to 29 stream results, and the NLP model performs 29 processing procedures. The related art obviously increases the consumption of computing resources and network bandwidth. In the embodiment of the present application, the NLP model only needs to perform five processing procedures, and thus the embodiment of the present application can obviously reduce the number of times of inputting the NLP model, thereby saving computing resources and network bandwidth. In a second aspect, the stream result provided by the related art cannot accurately extract a short sentence and cannot guarantee semantic integrity. For example, the eighth stream result extracts the instruction "open the window", which is actually incorrect, and the sixteenth stream result also includes the instruction "seat massage", which is not clear in meaning, and thus the accuracy of instruction extraction of the NLP model in the related art will be affected. In the embodiment of the present application, the stream result is determined in response to the PreEND, and the PreEND is used to simulate the semantic segmentation point of the instruction, and thus the semantic integrity can be guaranteed, thereby facilitating the accuracy of instruction extraction.

[0081] As can be seen, the embodiment of the present application simulates the short pause between instructions based on the improved VAD technology to determine the quasi-end state. Further, the stream result for inputting the NLP is determined based on the state, which has the above-mentioned advantages of guaranteeing the response rate of the instruction, saving computing resources and network bandwidth, and improving the accuracy of instruction extraction. In a further embodiment, since the short pause between instructions based on the quasi-end state may have certain instability, in order to reduce the inaccurate instruction recognition caused by inaccurate positioning (for example, if the interval between two adjacent instructions output by the user is too short or there is no pause interval, the quasi-end state cannot be determined based on the improved VAD technology), the embodiment of the present application also provides the following embodiment.

[0082] Figure 6 The flowchart of the stream semantic recognition result determination method provided by the embodiment of the present application can be used as a specific implementation manner of step S320.

[0083] In step S320-1, during the speech recognition of the current ith speech activity sub-paragraph, the subset of speech recognition results that have been generated and not input into the NLP model is continuously compared with the second preset condition before the quasi-end state is detected. In step S320-2, it is determined whether the subset of speech recognition results that have been generated and not input into the NLP model meets the second preset condition.

[0084] Since the speech recognition result is generated in the process of speech recognition, the speech recognition result is not accurate enough, and thus the instruction extraction accuracy is affected. Figure 6In the embodiment shown, for the case that the end condition cannot be accurately detected due to no pause or short pause between instructions, the second preset condition is used to measure whether the information amount of the comparison object (the subset of speech recognition results that have been generated and not input into the NLP model) is sufficient. If the second preset condition is met, it indicates that the current information amount of the comparison object is sufficient, i.e., the information contained in the current comparison object can be used as an instruction, and the streaming result needs to be generated for the NLP model to extract the instruction, even if the condition for triggering the end condition has not been met. If the second preset condition is not met, it indicates that the current information amount of the comparison object is not sufficient, i.e., the information contained in the current comparison object cannot be used as an instruction, and the speech recognition needs to be continued by the SR model, and the condition for triggering the generation of the streaming result has not been met.

[0085] For example, in the case that the type of the speech recognition result is a text string, the second preset condition is whether the length of the text string is greater than a second threshold. Specifically, if the length of the comparison object is greater than the second threshold, it indicates that the second preset condition is met; if the length of the comparison object is less than or equal to the second threshold, it indicates that the second preset condition is not met.

[0086] For example, still taking the five-instruction sentence as an example, assuming that there is no pause between the second instruction and the third instruction, and the current recognition result of the SR model is "open the air conditioner", the specific determination of the comparison object is as follows: during the speech recognition of the current second speech activity sub-paragraph, before the end condition is detected, the speech recognition result that has been generated is "open the air conditioner", and the speech recognition result that has been input into the NLP model is "open the air conditioner", and the subset of speech recognition results that have not been input into the NLP model is "open the air conditioner", so the comparison object is "open the air conditioner".

[0087] In the exemplary embodiment, with reference to Figure 6 If the determination result is that the comparison object does not meet the second preset condition, step S320-3 or step S320-1 can be performed in this case.

[0088] In step S320-3, in response to detecting the end condition, the i-th streaming speech recognition result corresponding to the i-th speech activity sub-paragraph is obtained; in this case, after the i-th streaming speech recognition result is input into the NLP model, the NLP model outputs the i-th instruction corresponding to the i-th speech activity sub-paragraph.

[0089] If the second preset condition is not met, it indicates that the information contained in the current comparison object can still be used as an instruction, and the speech recognition still needs to be continued through the SR model. Steps S320-1 and S320-2 are repeatedly executed until the end state is detected. The determination of the i-th stream speech recognition result in the case of detecting the end state is as described in steps S11-S13, and will not be repeated here.

[0090] In an exemplary embodiment, with reference to Figure 6 If the comparison object meets the second preset condition, step S320-4 of obtaining the i-th stream speech recognition result corresponding to the speech recognition result subset can be performed. After the i-th stream speech recognition result is input into the NLP model, the NLP model outputs one instruction in the instruction set corresponding to the i-th speech activity sub-paragraph.

[0091] Specifically, in response to the comparison object meeting the second preset condition, one implementation of determining the i-th stream speech recognition result is as follows:

[0092] In step S21, the current speech recognition result of the i-th speech activity sub-paragraph is obtained, and the current subset speech information in the i-th speech activity sub-paragraph is obtained. For example, Figure 5A In the process of speech recognition of the semantic activity sub-paragraph 1, "turn off", "turn off the sunroof", and the like can be used as the current subset speech information in the sub-paragraph. The current subset speech information in the sub-paragraph is part of the content belonging to the sub-paragraph, rather than the whole set.

[0093] In step S22, all speech recognition results of the audio to be processed that have been input into the NLP model are obtained, and the context speech information of the current subset is obtained. In step S23, the current subset speech information and the context speech information of the current subset are merged to obtain the i-th stream speech recognition result. For example, Figure 5A In the process of speech recognition of the semantic activity sub-paragraph 3, the current subset speech information of the sub-paragraph is "turn off the seat", and the context speech information of the current subset is "turn off the sunroof, turn on the window, and turn on the air conditioner". Thus, the merged stream recognition result is "turn off the sunroof, turn on the window, turn on the air conditioner, and turn off the seat".

[0094] By the embodiment shown in FIG. 5, in the process of performing speech recognition on the current i-th speech activity sub-paragraph, before the detection of the quasi-end state, by comparing the comparison object (a subset of the speech recognition result that has been generated and has not been input into the NLP model) with the preset condition (the second preset condition), if the current information amount of the comparison object is relatively large, the streaming result can be generated, thereby reducing the inaccurate command recognition caused by inaccurate positioning. For example, if the interval between the adjacent two commands output by the user is too short or there is no pause interval, the improved VAD technology cannot be used to determine the quasi-end state. Specifically, by judging whether the comparison object satisfies the second preset condition, the command segmentation point lacking pause between each other can be positioned, and finally the accuracy of the streaming result and the command extraction is improved.

[0095] As described above, the quasi-end state determined based on the improved VAD technology is used to simulate the short pause between the commands. Further, the streaming result for inputting the NLP is determined based on this state, which has the above-mentioned advantages of ensuring the response rate of the command, saving the computing resource network bandwidth, and improving the accuracy of the command extraction. In further embodiments, due to the instability caused by the short pause between the commands positioned based on the quasi-end state and the large information amount of a single command, in order to reduce the inaccurate command recognition caused by inaccurate positioning, the embodiments of the present application also provide the following embodiments.

[0096] For example, if a pause interval is generated during the output of a certain command by the user, if the improved VAD technology is used to determine the quasi-end state, the "sub-paragraph" recognized may have incomplete semantics. For this case, the embodiments of the present application provide the solution as step S31.

[0097] In step S31, the semantic integrity of the i-th speech activity sub-paragraph is detected before the quasi-end state is detected and before the i-th streaming recognition result is determined. For example, referring to FIG. 6, the semantic integrity of the i-th speech activity sub-paragraph is detected. Figure 5AAs shown in the first embodiment, if there is a pause in the second instruction "turn on the air conditioner", and a sub-paragraph a "turn on" is determined after "turn on" in the process of the improved VAD technology, the semantic integrity of the sub-paragraph a is detected in the embodiment, and it is determined that the semantic of the sub-paragraph a is not complete, so it can be explained that the quasi-end state A is false, that is, the streaming result should not be generated in response to the quasi-state A, but the speech recognition should be continued. Specifically, the semantic integrity is detected again after the next quasi-end state is reached or the recognized but not input NLP model recognition result meets the second preset condition, until the semantic integrity meets the preset requirement. For example, after it is determined that the semantic of the sub-paragraph a is not complete, the speech recognition is continued, the sub-paragraph b "air conditioner" is obtained by reaching the next quasi-end state, the semantic integrity of the combination of the sub-paragraph a and the sub-paragraph b is detected, and the streaming result corresponding to the sub-paragraph a and the sub-paragraph b ("turn on the air conditioner") is determined after it is determined that the semantic integrity meets the preset requirement.

[0098] For example, the information amount of a certain instruction output by the user is large (for example, the sentence is long), and the "sub-paragraph" recognized based on the step S320-1 and the step S320-2 may have a semantic incompleteness. For this case, the embodiment of the present application provides a solution as the step S31'.

[0099] In the step S31', the semantic integrity of the above comparison object (the subset of the speech recognition result that has been generated and has not been input into the NLP model) is detected before the i-th streaming recognition result is determined, when the above comparison object meets the second preset condition. For example, referring to Figure 5A As shown in the first embodiment, if there is a pause in the second instruction "turn on the air conditioner", and a sub-paragraph a "turn on" is determined after "turn on" in the process of the improved VAD technology, the semantic integrity of the sub-paragraph a is detected in the embodiment, and it is determined that the semantic of the sub-paragraph a is not complete, so it can be explained that the quasi-end state A is false, that is, the streaming result should not be generated in response to the quasi-state A, but the speech recognition should be continued. Specifically, the semantic integrity is detected again after the next quasi-end state is reached or the recognized but not input NLP model recognition result meets the second preset condition, until the semantic integrity meets the preset requirement. For example, after it is determined that the semantic of the sub-paragraph a is not complete, the speech recognition is continued, the sub-paragraph b "air conditioner" is obtained by reaching the next quasi-end state, the semantic integrity of the combination of the sub-paragraph a and the sub-paragraph b is detected, and the streaming result corresponding to the sub-paragraph a and the sub-paragraph b ("turn on the air conditioner") is determined after it is determined that the semantic integrity meets the preset requirement.

[0100] By performing semantic integrity detection on the sub-paragraphs in the above embodiments, the semantic integrity of the streaming results can be guaranteed, which is ultimately beneficial to the accuracy of instruction extraction.

[0101] After obtaining the streaming result (the i-th streaming speech recognition result) through the above embodiments, semantic integrity detection can be performed on the streaming result before inputting the i-th streaming recognition result into the NLP model, to ensure the semantic integrity of the information input into the NLP model, which is beneficial to reducing the computational load of the NLP model while guaranteeing accurate extraction of instructions.

[0102] Figure 7 A flowchart of a method for performing semantic integrity detection on a streaming result according to an embodiment of the present application is shown in the figure. Steps S710-S740 in the embodiment shown in the figure can be executed between step S320 and step S330, i.e., after obtaining the i-th streaming speech recognition result in step S320, steps S710-S740 are executed, and after determining that the semantic integrity requirement is met in step S710, step S330 is executed.

[0103] Reference Figure 7 In step S710, semantic integrity detection is performed on the i-th streaming speech recognition result to determine whether its semantic integrity is greater than the latest third threshold value.

[0104] For example, the i-th streaming speech recognition result can be input into a trained large language model to perform semantic integrity analysis on the i-th streaming speech recognition result through the large language model. The output information of the large language model is used to determine whether the semantic integrity of the i-th streaming speech recognition result meets the third preset condition.

[0105] The trained large language model can capture the inherent semantic structure features in the audio, which makes it perform outstanding accuracy and robustness in recognizing and understanding semantic integrity points, thereby enabling the large language model to have the ability to recognize and understand semantic integrity information.

[0106] For example, the i-th stream speech recognition result and the related prompt word are input into the trained large language model to determine whether the semantic integrity of the i-th stream speech recognition result is greater than a third threshold value through the large language model. Specifically, if the semantic integrity of the i-th stream speech recognition result is greater than the third threshold value, it means that the semantic integrity meets the preset requirement and can be processed by the NLP model, and then step S330 is performed. If the semantic integrity of the i-th stream speech recognition result is not greater than the third threshold value, it means that the semantic integrity cannot meet the preset requirement and cannot be processed by the NLP model to avoid inaccurate instruction extraction and waste of computing resources and network bandwidth, and then step S720 is performed: pause inputting information to the NLP model, and wait for the i+1-th stream speech recognition result to perform semantic integrity detection in combination.

[0107] For example, referring to Figure 5A For the first stream speech recognition result "close sunroof window", if the semantic integrity is less than the third threshold value Th3, the second stream speech recognition result "close sunroof window and open air conditioner" is waited for, and step S710 is performed again to determine whether the semantic integrity of the second stream speech recognition result meets the preset requirement.

[0108] For example, referring to Figure 7 It should be noted that, in order to improve the flexibility of the instruction recognition system, the third threshold value can also be dynamically adjusted. Therefore, in addition to performing step S710 again after performing step S720, step S730 is also performed: whether the number n of times that the semantic integrity does not meet the requirement (is not greater than the third threshold value) continuously reaches a fourth threshold value Th4. Since the information for determining the semantic integrity is stream information, the amount of information increases with the increase of i. Therefore, if the number of times that the semantic integrity does not meet the requirement (is not greater than the third threshold value Th3, such as 80%) continuously reaches the fourth threshold value Th4, it means that the amount of information of the current stream result is large. If the stream result with large information still cannot meet the current third threshold value Th3, it means that the requirement for the semantic integrity is too high, and then step S740 can be performed: the third threshold value Th3 is adjusted according to a preset strategy to reduce the requirement for the semantic integrity. The preset strategy can be set according to actual needs. It should be noted that the adjusted third threshold value Th3 should be greater than another threshold value (such as a fifth threshold value, and the value of the fifth threshold value can be 60%) to ensure that the semantic integrity determination works.

[0109] For example, in order to improve the execution efficiency, after step S720, step S730 can be executed first to determine whether the current third threshold value needs to be adjusted, and if it needs to be adjusted, step S740 is executed before step S710 is executed, so that the completeness judgment process of the streaming result is performed in time through the latest third threshold value. Of course, the present application does not limit the execution order of step S710 and step S730 executed again after step S720, and in addition to the restrictive step S730, step S730 and step S710 can also be executed simultaneously, or step S710 can be executed first after step S720 and then step S730 is executed, which all belong to the protection scope of the present application.

[0110] Continue with the above example, taking step S730 after step S720 and then step S710 as an example for illustration.

[0111] Referring to FIG. 5, after determining that the semantic completeness of the first streaming speech recognition result “close sunroof and window” is less than the third threshold value Th3 through step S710, step S730 is executed to determine that the number of times that the semantic completeness is less than the third threshold value Th3 consecutively is 1, and if the fourth threshold value Th4 is 2, the current third threshold value Th3 does not need to be adjusted, and step S740 is still maintained. Then, step S710 is executed to determine whether the semantic completeness of the second streaming speech recognition result is greater than the third threshold value Th3.

[0112] If the semantic completeness of the second streaming speech recognition result is still less than the third threshold value Th3 after executing step S710 again, step S720 is executed to wait for the third streaming speech recognition result “close sunroof and window, turn on air conditioner, and turn off massage seat”. Further, step S730 is executed: the number of times n that the semantic completeness is less than the third threshold value Th3 consecutively is 2, which reaches the fourth threshold value Th4 (the value is 2), so step S740 is executed to reduce the value of the third threshold value Th3. Then, step S710 is executed to determine whether the semantic completeness of the third streaming speech recognition result meets the requirements based on the latest Th3. It can be understood that after adjusting the third threshold value, the number of times n of consecutive semantic incompleteness is reset to zero for the next time as the basis for adjusting the third threshold value Th3.

[0113] Continue to refer to Figure 7 In the above embodiment, the judgment result of step S710 is that the current streaming result does not meet the preset requirements (i.e., it is not greater than the latest third threshold value Th3); the above semantic completeness judgment process is performed between inputting the streaming result into the NLP model in the present application, which can guarantee that the streaming result input into the NLP model has high semantic completeness, thereby being conducive to the accuracy of the NLP processing result, and at the same time, the number of times that the NLP model processes the semantic incomplete streaming result can be reduced, which is conducive to improving the resource utilization rate.

[0114] Next, the case where the determination result of step S710 is that the current streaming result meets the preset requirement (i.e., greater than the latest third threshold Th3) is introduced, in which case, step S330 is executed: inputting the ith streaming speech recognition result into a natural language processing NLP model, wherein the NLP model outputs an ith instruction corresponding to the ith speech activity sub-paragraph.

[0115] Exemplarily, in the embodiment of the application, the NLP model extracts instructions based on the streaming speech recognition result. After extracting the ith instruction, the instruction can be executed based on the scheme as shown in Figure 8 It can be seen that the embodiment of the application can provide more timely instruction response. Since the instructions are continuously extracted during the speech input process, the instruction execution can also be performed in a streaming manner during the speech input process. Therefore, the user can perceive a stronger streaming effect: the first instruction has already started to be executed during the process of inputting subsequent instructions, and the user experience will be better.

[0116] Exemplarily, the NLP model is used to understand and analyze the streaming speech recognition result and extract specific instructions. The NLP model mentioned above can be a pre-trained language model, such as a Bidirectional Encoder Representations from Transformers (BERT) model or a Text-to-Text Transfer Transformer (T5) model, etc. The embodiment of the application does not limit the NLP model, which can be any algorithm model capable of understanding and analyzing the streaming speech recognition result.

[0117] Exemplarily, the process of instruction extraction by the NLP model includes the following steps.

[0118] In step S41, text preprocessing is performed on the ith streaming speech recognition result.

[0119] Exemplarily, word segmentation processing can be performed to divide the input text into words or lexical units. Exemplarily, stop word removal processing can also be performed to remove common meaningless words (such as “of”, “is”, etc.) to reduce noise. Exemplarily, standardization processing can also be performed to convert the text into a unified format, such as lowercasing, removing punctuation marks, etc.

[0120] In step S42, a syntax analysis process is performed. Exemplarily, the dependency relationship between words in a sentence is analyzed, and a syntax structure tree is constructed to help understand the logical structure of the sentence. Exemplarily, named entity recognition is performed to identify proper nouns in the text, such as names, places, times, etc., so as to better understand the context.

[0121] In step S43, a semantic parsing process is performed. Illustratively, intent recognition is conducted to determine the main intent or purpose expressed by the user. For example, "turn on the air conditioner" indicates that the operation the user wants to perform is to turn on the air conditioning device. Slot filling processing can also be conducted, specifically extracting parameter or attribute values related to a specific intent. For example, in "set the temperature to 25 degrees", "25 degrees" is a slot, indicating the target temperature.

[0122] In step S44, the dialogue state is maintained through state tracking, recording the context information of the current dialogue, ensuring the consistency and coherence of multi-round dialogue. And according to the dialogue state and user intent, the next response or operation is decided.

[0123] In step S45, the parsed intent and slot information are mapped to specific commands or application program interface calls that the system can execute. If necessary, confirmation information or further prompts are provided to the user to ensure that the instructions are correctly understood and executed.

[0124] Figure 8 A flowchart of a processing method of the extracted instructions in the embodiments of the present application is provided.

[0125] Reference Figure 8 The i-th streaming speech recognition result is input into the natural language processing module 810, and the instruction is obtained after processing by the natural language processing model 810. Further, the instruction can be input into the instruction execution system 820 to execute the corresponding instruction. For example, for the instruction "turn on the air conditioner to 25 degrees", the control system interface compatible with the vehicle internal communication protocol builds an application program interface request, such as set_air_conditioning(temperature=25); the air conditioner control system receives and executes the command, adjusting the temperature to 25 degrees. Illustratively, a voice feedback mechanism can also be provided to feedback the voice to the user: "the air conditioner has been set to 25 degrees". Illustratively, in the case of extracting the instruction "I am hungry", a voice feedback mechanism can be provided to feedback the relevant voice of nearby restaurants to the user.

[0126] Reference Figure 8, the instruction can be input to the speech synthesis system 830 to provide voice broadcast for the relevant instruction. Assuming that the system needs to broadcast "the air conditioner has been set to 25 degrees", the specific implementation process can be step S51: generating a feedback text according to the instruction, for example, if the instruction is "adjust the air conditioner to 25 degrees", the feedback text can be "the air conditioner has been set to 25 degrees". Step S52: analyze the text and convert it into a phoneme sequence: "air / conditioner / has / set / up / as / 25 / degrees"; then use the pre-trained speech synthesis model to generate the corresponding audio waveform; convert the audio waveform into a format suitable for playing, such as pulse code modulation. Step S53: adjust the volume to adapt to the in-vehicle environment. Exemplarily, audio enhancement technology can also be applied to ensure clear speech.

[0127] The above embodiments generally introduce specific embodiments of the speech instruction streaming recognition method. Next, specific embodiments of the speech instruction streaming recognition method are provided in combination with Figure 9 A specific embodiment of a speech instruction streaming recognition method is provided.

[0128] Reference is made to Figure 9 In step S91, the speech recognition result output by the SR model is obtained.

[0129] Exemplarily, in the process of receiving the audio to be processed, the improved VAD technology described above is used to process the audio. As the speech activity sub-paragraph is generated, the sub-paragraph is recognized by the SR model, wherein the sub-paragraph is recognized as it is generated, for example, after the Start state is determined, the subsequent audio is recognized without waiting for PreEND to recognize the entire sub-paragraph, thereby effectively ensuring the efficiency of instruction extraction and execution response.

[0130] In step S92, it is determined whether the PreEND condition is met.

[0131] If the PreEND condition is met, the streaming result (i.e., the aforementioned ith streaming speech recognition result) is determined. As described above, in the one-sentence multi-instruction scenario, there is a natural small pause between different instructions, and the above-mentioned quasi-end state can be used to accurately locate the segmentation point between instructions, thereby obtaining the speech activity sub-paragraph corresponding to the instruction.

[0132] In the embodiments of the present application, the streaming result input to the NLP model is determined in units of speech activity sub-paragraphs. Specifically, in the process of performing speech recognition on the current ith speech activity sub-paragraph, in response to detecting the quasi-end state PreEND, it is indicated that the segmentation point between the current sub-paragraph and the next field is reached, and then the ith streaming speech recognition result corresponding to the ith speech activity sub-paragraph can be obtained.

[0133] The screenshot implementation of determining the streaming result when the PreEND condition is met is as shown in steps S11-S13, which will not be described again here.

[0134] If the PreEND condition is not met, step S93 can also be performed: determining whether an information amount condition is met.

[0135] The specific implementation of step S93 is as described in the specific implementation of step S320-1 and step S320-2, which will not be repeated here.

[0136] If the information amount condition is met, the streaming result is determined. The screenshot implementation of determining the streaming result when the information amount condition is met is as described in steps S21-S23, which will not be repeated here.

[0137] Reference Figure 9 After the streaming result is determined, step S94 is performed: semantic completeness detection is performed on the streaming result.

[0138] And in step S95, it is determined whether the semantic completeness is greater than Th3.

[0139] The specific implementation of steps S94-S95 is as described in the specific implementation of step S710, which will not be repeated here,

[0140] If it is determined that the semantic completeness of the streaming result is greater than Th3, step S96 is performed: the streaming result is input into an NLP model to extract instructions.

[0141] If it is determined that the semantic completeness of the streaming result is not greater than Th3, step S97 is performed: it is determined whether the number of consecutive semantic incompleteness n reaches Th4.

[0142] If n reaches Th4, step S98 is performed: the value of n is reset to zero, Th3 is lowered and reset, and semantic completeness detection is performed based on the updated Th3 in step S95. If n does not reach Th4, the next streaming result is waited for.

[0143] The speech instruction stream recognition scheme provided in the embodiments of the present application performs VAD on the audio to be processed to determine a speech activity sub-paragraph in the audio to be processed, wherein the speech activity sub-paragraph is audio between a start state and a quasi-end state detected in sequence. Further, in the process of performing speech recognition on the current i th speech activity sub-paragraph, in response to detecting the quasi-end state, an i th stream speech recognition result corresponding to the i th speech activity sub-paragraph is obtained. The i th stream speech recognition result is input into a natural language processing (NLP) model, wherein the NLP model outputs an i th instruction corresponding to the i th speech activity sub-paragraph. It can be seen that in the embodiments of the present application, the SR model takes the speech activity sub-paragraph as a unit, and inputs the speech recognition result into the NLP model, so that the NLP model identifies the semantic recognition result corresponding to the speech activity sub-paragraph in each identification process, thereby identifying multiple instructions contained in the audio to be processed. Compared with the related art, in the embodiments of the present application, the NLP model extracts instructions in units of speech activity sub-paragraphs, which can reduce the calculation amount of the NLP model and is beneficial to reducing resource occupation. At the same time, since there is often a slight pause between multiple instructions when a sentence is used to issue multiple instructions, in the embodiments of the present application, the quasi-end state is detected to simulate the pause between multiple instructions in a sentence. Therefore, the NLP model extracts instructions in units of speech activity sub-paragraphs, which is also beneficial to improving the instruction extraction accuracy.

[0144] The method embodiments of the present application are described above in combination with Figures 1 to 9 The speech instruction stream recognition device embodiments of the present application are described below in combination with Figure 10 The speech instruction stream recognition device embodiments of the present application are described below in combination with

[0145] Figure 10 A schematic block diagram of a speech instruction stream recognition device 1000 provided in the embodiments of the present application is shown in FIG. 1.

[0146] Referring to Figure 10 The speech instruction stream recognition device 1000 provided in the embodiments of the present application includes a speech detection module 1010, a stream result acquisition module 1020, and an instruction extraction module 1030.

[0147] The voice detection module 1010 is configured to perform voice activity detection on the to-be-processed audio to determine a voice activity sub-paragraph in the to-be-processed audio, where the voice activity sub-paragraph is audio detected between a start state and a quasi-end state in sequence. The streaming result acquisition module 1020 is configured to, in a process of performing speech recognition on a current ith voice activity sub-paragraph, acquire an ith streaming speech recognition result corresponding to the ith voice activity sub-paragraph in response to detection of the quasi-end state, where i is a positive integer not greater than a total number of voice activity sub-paragraphs in the to-be-processed audio. The instruction extraction module 1030 is configured to input the ith streaming speech recognition result into a natural language processing (NLP) model, where the NLP model outputs an ith instruction corresponding to the ith voice activity sub-paragraph.

[0148] In an example embodiment, based on the foregoing scheme, the voice detection module 1010 is specifically configured to: input the to-be-processed audio into a voice activity detection (VAD) processing module; in a case where the VAD processing module detects that a feature of an audio signal meets a first preset condition, output a start state; and in a case where the VAD processing module detects that the feature of the audio signal does not meet the first preset condition and lasts for a first preset time length, output a quasi-end state.

[0149] The first preset time length is less than a first threshold.

[0150] The feature of the audio signal includes at least one of an energy level of the audio signal, a spectral feature, and a change characteristic in a time domain.

[0151] In an example embodiment, based on the foregoing scheme, the streaming result acquisition module 1020 includes a comparison unit and a first acquisition unit. The comparison unit is configured to, in a process of performing speech recognition on a current ith voice activity sub-paragraph, continue to compare a subset of speech recognition results that have been generated and have not been input into the NLP model with a second preset condition before detection of the quasi-end state. The first acquisition unit is configured to, if the comparison result is that the subset of speech recognition results does not meet the second preset condition, acquire an ith streaming speech recognition result corresponding to the ith voice activity sub-paragraph in response to detection of the quasi-end state.

[0152] In an example embodiment, based on the above scheme, the first obtaining unit is specifically configured to: obtain a speech recognition result of the i-th speech activity sub-paragraph to obtain current sub-paragraph speech information; obtain all speech recognition results of the audio to be processed that have been input into the NLP model to obtain context speech information of the current sub-paragraph; and combine the current sub-paragraph speech information and the context speech information of the current sub-paragraph to obtain the i-th streaming speech recognition result.

[0153] In an example embodiment, based on the above scheme, the streaming result obtaining module 1020 further includes a second obtaining unit.

[0154] The second obtaining unit is configured to: if the comparison result is that the speech recognition result subset satisfies the second preset condition, obtain the i-th streaming speech recognition result corresponding to the speech recognition result subset; wherein, after the i-th streaming speech recognition result is input into the natural language processing (NLP) model, the NLP model outputs one instruction in the instruction set corresponding to the i-th speech activity sub-paragraph.

[0155] In an example embodiment, based on the above scheme, the second obtaining unit is specifically configured to: obtain a current speech recognition result of the i-th speech activity sub-paragraph to obtain current sub-set speech information in the i-th speech activity sub-paragraph; obtain all speech recognition results of the audio to be processed that have been input into the NLP model to obtain context speech information of the current sub-set; and combine the current sub-set speech information and the context speech information of the current sub-set to obtain the i-th streaming speech recognition result.

[0156] In an example embodiment, based on the above scheme, the type of the speech recognition result is a text string, and the second preset condition includes that the length of the text string is greater than a second threshold.

[0157] In an example embodiment, based on the above scheme, the device further includes a semantic integrity detection module.

[0158] The semantic integrity detection module is configured to: before the i-th streaming speech recognition result is input into the natural language processing (NLP) model by the instruction extraction module, perform semantic integrity detection on the i-th streaming speech recognition result to determine whether the semantic integrity of the i-th streaming speech recognition result is greater than a third threshold; wherein, in the case where the semantic integrity is greater than the third threshold, the i-th streaming speech recognition result is input into the NLP model.

[0159] In an example embodiment, based on the above scheme, the semantic integrity detection module is further configured to, in a case where the semantic integrity is less than or equal to the third threshold, suspend inputting information to the NLP model and wait for performing semantic integrity detection on the (i+1)th stream speech recognition result; in a case where the number of times of performing the semantic integrity detection is greater than a fourth threshold, adjust the third threshold according to a preset strategy to perform semantic integrity detection again based on the adjusted third threshold; and wherein the adjusted third threshold is greater than a fifth threshold.

[0160] In an example embodiment, based on the above scheme, the semantic integrity detection module is specifically configured to input the ith stream speech recognition result into a large language model trained to perform semantic integrity analysis on the ith stream speech recognition result through the large language model.

[0161] In an example embodiment, based on the above scheme, the device further comprises an execution module, wherein the execution module is configured to execute the ith instruction in a process of identifying a subsequent instruction of the ith instruction by the instruction extraction module, and / or generate response information corresponding to the to-be-processed audio according to the ith instruction in the process of identifying the subsequent instruction of the ith instruction by the instruction extraction module, and output the response information.

[0162] It should be understood that the embodiments of the stream recognition device of the voice instruction as shown in Figure 10 correspond to the above-mentioned embodiments of the stream recognition method of the voice instruction, and similar descriptions can be referred to the method embodiments. To avoid repetition, it will not be described here. Specifically, through the information interaction between the modules in the stream recognition device of the voice instruction as shown in Figure 10 , the above-mentioned embodiments of the stream recognition method of the voice instruction can be executed, through the information interaction between the modules in the stream recognition device of the voice instruction as shown in Figure 10 , the above-mentioned embodiments of the stream recognition method of the voice instruction can be executed, and the aforementioned and other operations and / or functions of each module in the device correspond to the method embodiments, which are not described here for brevity.

[0163] The operation and maintenance related apparatus of the software agent of the embodiments of the present application is described above from the perspective of functional modules in combination with the drawings. It should be understood that the functional modules can be implemented in the form of hardware, or in the form of instructions of software, or in the form of a combination of hardware and software modules. Specifically, each step of the method embodiments in the embodiments of the present application can be completed by integrated logic circuits of hardware in a processor and / or instructions of software. The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processing for execution by a processor, or can be executed by a combination of hardware and software modules in the code processing. Alternatively, the software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and a processor reads information in the memory and combines hardware thereof to complete the steps in the above method embodiments.

[0164] The embodiments of the present application also provide an electronic device.

[0165] Figure 11 A schematic block diagram of an electronic device 1100 is provided in the embodiments of the present application. As described above, the operation and maintenance related apparatus of the software agent can be deployed in an electronic device as shown in Figure 11 , so that the electronic device can be used to execute the above-mentioned speech instruction streaming recognition method.

[0166] As shown in Figure 11 , the electronic device 1100 can include:

[0167] The memory 1110 is used to store a computer program 1130 and transmit the program code 1130 to the processor 1120. In other words, the processor 1120 can call and run the computer program 1130 from the memory 1110 to implement the method in the embodiments of the present application.

[0168] For example, the processor 1120 can be used to execute the steps in the above-mentioned speech instruction streaming recognition method according to the instructions in the computer program 1130, or execute the steps in the above-mentioned speech instruction streaming recognition method.

[0169] In some embodiments of the present application, the processor 1120 can include but is not limited to:

[0170] A general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc.

[0171] In some embodiments of the present application, the memory 1110 includes, but is not limited to:

[0172] volatile memory and / or non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), which is used as the external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0173] In some embodiments of the present application, the computer program 1130 can be divided into one or more modules, which are stored in the memory 1110 and executed by the processor 1120 to complete the voice instruction streaming recognition method provided by the present application, or to complete the steps in the voice instruction streaming recognition method described above. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 1130 in the electronic device.

[0174] As Figure 11As shown, the electronic device 1100 can further include:

[0175] a transceiver 1140, which can be connected to the processor 1120 or the memory 1110.

[0176] The processor 1120 can control the transceiver 1140 to communicate with other devices, specifically, can send information or data to other devices, or receive information or data sent by other devices. The transceiver 1140 can include a transmitter and a receiver. The transceiver 1140 can further include an antenna, and the number of antennas can be one or more.

[0177] It should be understood that various components in the electronic device 1100 are connected through a bus system, wherein the bus system includes a data bus, a power supply bus, a control bus, and a status signal bus in addition to a data bus.

[0178] According to an aspect of the present application, a computer storage medium is provided, and the computer storage medium stores a computer program. When the computer program is executed by a computer, the computer can execute the method of the above-mentioned method embodiment. Alternatively, the present application further provides a computer program product containing instructions. When the instructions are executed by a computer, the computer executes the method of the above-mentioned method embodiment.

[0179] According to another aspect of the present application, a computer program product or computer program is provided, and the computer program product or computer program includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of the above-mentioned method embodiment.

[0180] In other words, when implemented using software, it can be implemented wholly or partially in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., Digital Video Disc (DVD)), or a semiconductor medium (e.g., Solid State Disk (SSD)).

[0181] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0183] The modules illustrated as separate components may or may not be physically separate, and the components illustrated as modules may or may not be physical modules, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules in various embodiments of the present application can be integrated in one processing module, or each module can exist physically separately, or two or more modules can be integrated in one module.

[0184] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for streaming recognition of voice commands, characterized in that, The method comprises: performing voice activity detection on the to-be-processed audio to determine a voice activity sub-paragraph in the to-be-processed audio, wherein the voice activity sub-paragraph is audio between a start state and a quasi-end state detected in sequence; in a process of performing speech recognition on a current ith voice activity sub-paragraph, in response to detecting the quasi-end state, obtaining an ith streaming speech recognition result corresponding to the ith voice activity sub-paragraph, wherein i is a positive integer not greater than a total number of voice activity sub-paragraphs in the to-be-processed audio; inputting the ith streaming speech recognition result into a natural language processing (NLP) model, wherein the NLP model outputs an ith instruction corresponding to the ith voice activity sub-paragraph.

2. The method of claim 1, wherein, The voice activity detection on the to-be-processed audio comprises: inputting the to-be-processed audio into a voice activity detection (VAD) processing module, in a case where a feature of an audio signal detected by the VAD processing module meets a first preset condition, outputting a start state; in a case where the feature of the audio signal detected by the VAD processing module does not meet the first preset condition and lasts for a first preset time length, outputting a quasi-end state; wherein the first preset time length is less than a first threshold value; the feature of the audio signal comprises at least one of an energy level, a spectral feature, and a change characteristic in a time domain of the audio signal.

3. The method of claim 1, wherein, In a process of performing speech recognition on a current ith voice activity sub-paragraph, in response to detecting the quasi-end state, obtaining an ith streaming speech recognition result corresponding to the ith voice activity sub-paragraph, comprises: in a process of performing speech recognition on a current ith voice activity sub-paragraph, before detecting the quasi-end state, continuously comparing a subset of speech recognition results that have been generated and not input into the NLP model with a second preset condition; if the comparison result is that the subset of speech recognition results does not meet the second preset condition, in response to detecting the quasi-end state, obtaining an ith streaming speech recognition result corresponding to the ith voice activity sub-paragraph.

4. The method of claim 3, wherein, The obtaining of the ith streaming speech recognition result corresponding to the ith voice activity sub-paragraph comprises: obtaining a speech recognition result of the ith voice activity sub-paragraph to obtain current sub-paragraph speech information; obtaining all speech recognition results in the to-be-processed audio that have been input into the NLP model to obtain context speech information of the current sub-paragraph; merging the current sub-paragraph speech information and the context speech information of the current sub-paragraph to obtain the ith streaming speech recognition result.

5. The method of claim 3, wherein, The method further comprises: if the comparison result is that the subset of speech recognition results meets the second preset condition, obtaining an ith streaming speech recognition result corresponding to the subset of speech recognition results; wherein, after the ith streaming speech recognition result is input into a natural language processing (NLP) model, the NLP model outputs one instruction in an instruction set corresponding to the ith voice activity sub-paragraph.

6. The method of claim 5, wherein, The obtaining of the ith streaming speech recognition result corresponding to the subset of speech recognition results comprises: obtaining a current speech recognition result of the i th speech activity sub-paragraph, to obtain current subset speech information in the i th speech activity sub-paragraph; obtaining all speech recognition results of the NLP model input in the to-be-processed audio, to obtain context speech information of the current subset; merging the current subset speech information and the context speech information of the current subset, to obtain the i th stream speech recognition result.

7. The method according to any one of claims 3 to 6, characterized in that, The type of the speech recognition result is a text string, and the second preset condition includes that the length of the text string is greater than a second threshold.

8. The method according to any one of claims 1 to 6, characterized in that, Before the i th stream speech recognition result is input into the natural language processing NLP model, the method further includes: performing semantic integrity detection on the i th stream speech recognition result to determine whether the semantic integrity degree is greater than a third threshold; wherein, in the case that the semantic integrity degree is greater than the third threshold, the i th stream speech recognition result is input into the natural language processing NLP model.

9. The method of claim 8, wherein, The method further includes: in the case that the semantic integrity is less than or equal to the third threshold, pausing inputting information into the NLP model, and waiting for performing semantic integrity detection on the i+1 th stream speech recognition result; in the case that the continuous substandard times of the semantic integrity degree reaches a fourth threshold, adjusting the third threshold according to a preset strategy, to perform semantic integrity detection again based on the adjusted third threshold; wherein, the adjusted third threshold is greater than a fifth threshold.

10. The method of claim 8, wherein, The performing semantic integrity detection on the i th stream speech recognition result includes: inputting the i th stream speech recognition result into a trained large language model, to perform semantic integrity analysis on the i th stream speech recognition result through the large language model.

11. The method according to any one of claims 1 to 6, characterized in that, The method further includes: in the process of recognizing the subsequent instruction of the i th instruction, executing the i th instruction; and / or, in the process of recognizing the subsequent instruction of the i th instruction, generating response information corresponding to the to-be-processed audio according to the i th instruction, and outputting the response information.

12. A device for streaming recognition of voice commands, characterized in that The device includes: a speech detection module, configured to perform speech activity detection on to-be-processed audio to determine speech activity sub-paragraphs in the to-be-processed audio, wherein the speech activity sub-paragraphs are audio between the start state and the quasi-end state obtained in sequence through detection; a stream result obtaining module, configured to, in the process of performing speech recognition on a current i th speech activity sub-paragraph, obtain an i th stream speech recognition result corresponding to the i th speech activity sub-paragraph in response to detecting a quasi-end state, wherein i is a positive integer not greater than the total number of speech activity sub-paragraphs in the to-be-processed audio; an instruction extraction module, configured to input the i th stream speech recognition result into a natural language processing NLP model, wherein the NLP model outputs an i th instruction corresponding to the i th speech activity sub-paragraph.

13. A computer-readable storage medium, characterized in that, for storing a computer program; The computer program enables a computer to execute the stream speech instruction recognition method in any one of claims 1 to 11.

14. An electronic device, comprising: including a processor and a memory; The memory is configured to store a computer program; The processor is configured to execute the computer program to implement the voice instruction streaming recognition method in any one of claims 1 to 11.

Images