Streaming recognition method and device of voice instruction, medium and equipment

Abstract

The embodiment of the invention provides a streaming recognition method and device of a voice instruction, a computer readable storage medium and electronic equipment, and relates to the technical field of artificial intelligence. The method comprises the steps that voice activity detection is carried out on to-be-processed audio to determine voice activity sub-paragraphs in the to-be-processed audio, and the voice activity sub-paragraphs are audios which are sequentially obtained through detection and are between an initial state and a quasi-ending state; in the process of performing voice recognition on the current ith voice activity sub-paragraph, obtaining an ith streaming voice recognition result corresponding to the ith voice activity sub-paragraph in response to the detected quasi-ending state, i being a positive integer not greater than the total amount of the voice activity sub-paragraphs in the audio to be processed; and the ith streaming speech recognition result is input into a natural language processing (NLP) model, and the NLP model outputs an ith instruction corresponding to the ith speech activity sub-paragraph. According to the embodiment of the invention, computing resources can be saved while the instruction recognition accuracy is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method for streaming recognition of voice commands, a device for streaming recognition of voice commands, a computer-readable storage medium, and an electronic device. Background Art

[0002] Speech Recognition (SR) refers to the technology of converting speech into text. One application of speech recognition includes a streaming recognition system for multiple instructions. Specifically, the input information of the system can be a segment of audio containing multiple instructions. After recognizing the above segment of audio, multiple instructions are output, rather than outputting a single instruction and then recognizing the next instruction. For example, the driver can say: "Turn on seat massage, turn on the air conditioner, and turn up the volume." The multi-instruction recognition system can recognize these three instructions in one sentence and execute the three instructions in sequence.

[0003] In the related art, for the user's voice input, the voice activity segment is first obtained through voice activity detection, and then voice recognition is performed on each voice activity segment. Specifically, in order to improve the response speed, the SR model will promptly send the streaming voice recognition results to the natural language understanding (NLP) model whenever it recognizes part of the voice of the voice activity segment, so that the NLP model will extract instructions for each received voice recognition result. For example, if the user's voice is "close the sunroof, turn on the air conditioner", the voice recognition results sent to the natural language understanding model each time are "off", "off", "close the sky", "close the sunroof", "close the sunroof on", "close the sunroof on", "close the sunroof on air", and "close the sunroof on air".

[0004] However, the related technology has the problem of consuming a large amount of computing resources. Summary of the invention

[0005] The embodiments of the present application provide a method for streaming recognition of voice commands, a device for streaming recognition of voice commands, a computer-readable storage medium, and an electronic device, which can improve the accuracy of command recognition while saving computing resources.

[0006] In a first aspect, an embodiment of the present application provides a method for streaming recognition of voice instructions, the method comprising: performing voice activity detection on an audio to be processed to determine a voice activity subparagraph in the audio to be processed, wherein the voice activity subparagraph is audio between a starting state and a quasi-ending state obtained in sequence through detection; in a process of performing voice recognition on a current i-th voice activity subparagraph, in response to detecting a quasi-ending state, obtaining an i-th streaming voice recognition result corresponding to the i-th voice activity subparagraph, wherein the value of i is a positive integer not greater than the total number of voice activity subparagraphs in the audio to be processed; inputting the i-th streaming voice recognition result into a natural language processing NLP model, wherein the NLP model outputs an i-th instruction corresponding to the i-th voice activity subparagraph.

[0007] In a second aspect, an embodiment of the present application provides a streaming recognition device for voice commands, the device comprising: a voice detection module, a streaming result acquisition module and a command extraction module.

[0008] Among them, the above-mentioned voice detection module is used to perform voice activity detection on the audio to be processed to determine the voice activity sub-paragraph in the above-mentioned audio to be processed, wherein the above-mentioned voice activity sub-paragraph is the audio between the starting state and the quasi-ending state obtained in sequence through detection; the above-mentioned streaming result acquisition module is used to obtain the i-th streaming voice recognition result corresponding to the above-mentioned i-th voice activity sub-paragraph in response to the quasi-ending state obtained by detection during the process of performing voice recognition on the current i-th voice activity sub-paragraph, wherein the value of i is a positive integer not greater than the total number of voice activity sub-paragraphs in the above-mentioned audio to be processed; and the above-mentioned instruction extraction module is used to input the above-mentioned i-th streaming voice recognition result into the natural language processing NLP model, wherein the above-mentioned NLP model outputs the i-th instruction corresponding to the above-mentioned i-th voice activity sub-paragraph.

[0009] In an exemplary embodiment, based on the above scheme, the above voice detection module is specifically used to: input the audio to be processed into the voice activity detection VAD processing module, and output a start state when the above VAD processing module detects that the characteristics of the audio signal meet the first preset condition; output a quasi-end state when the above VAD processing module detects that the characteristics of the audio signal do not meet the first preset condition and last for a first preset time;

[0010] Wherein, the first preset duration is less than the first threshold;

[0011] The characteristics of the audio signal include at least one of the following: energy level, spectrum characteristics, and change characteristics in the time domain of the audio signal.

[0012] In an exemplary embodiment, based on the above scheme, the above streaming result acquisition module includes: a comparison unit and a first acquisition unit; wherein the above comparison unit is used to: in the process of performing speech recognition on the current i-th voice activity sub-paragraph, before detecting a quasi-end state, continue to compare the speech recognition result subset that has been generated and not input into the above-mentioned NLP model with the second preset condition; and the above-mentioned first acquisition unit is used to: if the comparison result is that the above-mentioned speech recognition result subset does not meet the above-mentioned second preset condition, then in response to detecting a quasi-end state, obtain the i-th streaming speech recognition result corresponding to the above-mentioned i-th voice activity sub-paragraph.

[0013] In an exemplary embodiment, based on the above scheme, the above-mentioned first acquisition unit is specifically used to: obtain the speech recognition result of the above-mentioned i-th voice activity sub-paragraph to obtain the speech information of the current sub-paragraph; obtain all speech recognition results that have been input into the above-mentioned NLP model in the above-mentioned audio to be processed to obtain the contextual speech information of the current sub-paragraph; and, merge the above-mentioned current sub-paragraph speech information and the above-mentioned current sub-paragraph contextual speech information to obtain the above-mentioned i-th streaming speech recognition result.

[0014] In an exemplary embodiment, based on the above solution, the above streaming result acquisition module further includes: a second acquisition unit;

[0015] Among them, the second acquisition unit is used for: if the comparison result is that the above-mentioned speech recognition result subset satisfies the above-mentioned second preset condition, then obtaining the i-th streaming speech recognition result corresponding to the above-mentioned speech recognition result subset; wherein, after the above-mentioned i-th streaming speech recognition result is input into the natural language processing NLP model, the above-mentioned NLP model outputs an instruction in the instruction set corresponding to the above-mentioned i-th speech activity sub-paragraph.

[0016] In an exemplary embodiment, based on the above scheme, the above-mentioned second acquisition unit is specifically used to: obtain the current speech recognition result of the above-mentioned i-th voice activity sub-paragraph, and obtain the current subset speech information in the above-mentioned i-th voice activity sub-paragraph; obtain all speech recognition results that have been input into the above-mentioned NLP model in the above-mentioned audio to be processed, and obtain the context speech information of the current subset; and, merge the above-mentioned current subset speech information and the above-mentioned current subset context speech information to obtain the above-mentioned i-th streaming speech recognition result.

[0017] In an exemplary embodiment, based on the above scheme, the type of the above speech recognition result is a text string, and the above second preset condition includes that the length of the text string is greater than a second threshold.

[0018] In an exemplary embodiment, based on the above solution, the above device further includes: a semantic integrity detection module;

[0019] Among them, the above-mentioned semantic integrity detection module is used to: before the above-mentioned instruction extraction module inputs the above-mentioned i-th streaming speech recognition result into the natural language processing NLP model, perform semantic integrity detection on the above-mentioned i-th streaming speech recognition result to determine whether its semantic integrity is greater than the third threshold; wherein, when the above-mentioned semantic integrity is greater than the above-mentioned third threshold, the above-mentioned i-th streaming speech recognition result is input into the natural language processing NLP model.

[0020] In an exemplary embodiment, based on the above scheme, the above semantic integrity detection module is also used to: when the above semantic integrity is less than or equal to the above third threshold, suspend the input of information to the above NLP model, and wait for the semantic integrity detection to be performed on the above i+1th streaming speech recognition result; when the number of consecutive failures of the above semantic completeness to meet the standard reaches a fourth threshold, lower the above third threshold according to a preset strategy to perform the semantic integrity detection again based on the adjusted third threshold; wherein the adjusted third threshold is greater than the fifth threshold.

[0021] In an exemplary embodiment, based on the above scheme, the above semantic integrity detection module is specifically used to: input the above i-th streaming speech recognition result into a trained large language model to perform semantic integrity analysis on the above i-th streaming speech recognition result through the above large language model.

[0022] In an exemplary embodiment, based on the above scheme, the above device also includes: an execution module; wherein the above execution module is used to: execute the above i-th instruction during the process of the above instruction extraction module identifying the subsequent instruction of the above i-th instruction; and / or, during the process of the above instruction extraction module identifying the subsequent instruction of the above i-th instruction, generate response information corresponding to the above-mentioned audio to be processed according to the above i-th instruction, and output the above-mentioned response information.

[0023] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the streaming recognition method of voice commands provided in the first aspect.

[0024] In a sixth aspect, an embodiment of the present application provides a chip for implementing the streaming recognition method of voice commands provided in the first aspect or the second aspect. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip executes the streaming recognition method of voice commands provided in the first aspect.

[0025] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the streaming recognition method of voice instructions provided in the first aspect.

[0026] In an eighth aspect, an embodiment of the present application provides a computer program product, including computer program instructions, wherein the computer program instructions enable a computer to execute the streaming recognition method of voice instructions provided in the first aspect.

[0027] In a ninth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the streaming recognition method of voice instructions provided in the first aspect above.

[0028] In summary, in the streaming recognition scheme of voice instructions provided in the embodiment of the present application, voice activity detection is performed on the audio to be processed to determine the voice activity sub-paragraph in the audio to be processed, wherein the voice activity sub-paragraph is the audio between the starting state and the quasi-end state obtained in sequence through detection. Further, in the process of voice recognition of the current i-th voice activity sub-paragraph, in response to the quasi-end state obtained by detection, the i-th streaming voice recognition result corresponding to the i-th voice activity sub-paragraph is obtained. The i-th streaming voice recognition result is input into the natural language processing NLP model, wherein the NLP model outputs the i-th instruction corresponding to the i-th voice activity sub-paragraph. It can be seen that the SR model in the embodiment of the present application takes the voice activity sub-paragraph as a unit, and inputs the voice recognition result into the NLP model, so that the NLP model uses the semantic recognition result corresponding to the voice activity sub-paragraph in each recognition process, thereby identifying multiple instructions contained in the audio to be processed. Compared with the related art, the NLP model in the embodiment of the present application extracts instructions by taking each voice activity sub-paragraph as a unit, which can reduce the computational amount of the NLP model and help reduce resource usage. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 A schematic diagram of the system architecture of an application environment of a streaming recognition solution for voice commands provided in an embodiment of the present application;

[0031] Figure 2 A flow chart of a method for stream recognition of voice commands provided in an embodiment of the present application;

[0032] Figure 3 A flow chart of a method for stream recognition of voice commands provided in an embodiment of the present application;

[0033] Figure 4A A schematic diagram of VAD detection of a sentence of audio containing multiple instructions provided for related technologies;

[0034] Figure 4B A schematic diagram of improved VAD detection of a sentence audio containing multiple instructions provided in an embodiment of the present application;

[0035] Figure 5A A schematic diagram of determining a streaming speech recognition result provided in an embodiment of the present application;

[0036] Figure 5B A schematic diagram of determining a streaming speech recognition result provided for related technologies;

[0037] Figure 6 A flowchart of a method for determining an i-th streaming speech recognition result provided in an embodiment of the present application;

[0038] Figure 7 A flowchart of a method for detecting the semantic completeness of a streaming speech recognition result provided in an embodiment of the present application;

[0039] Figure 8 A flowchart of a method for processing an identified instruction provided in an embodiment of the present application;

[0040] Fig. 9 A flow chart of a method for stream recognition of voice commands provided in an embodiment of the present application;

[0041] Fig.10 A schematic block diagram of a streaming recognition device for voice commands provided in an embodiment of the present application;

[0042] Fig.11 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0044] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here can be implemented in a sequence other than those illustrated or described here. In the present application embodiment of the present invention, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, but can also determine B according to A and / or other information. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. In the description of the present application, unless otherwise specified, "multiple" refers to two or more than two.

[0045] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0046] In the above streaming recognition related technologies for voice commands, the SR model sends all the streaming results in sequence, and the NLP model needs to perform semantic analysis on all the streaming results to try to extract valid commands. Since the speech in the one-sentence-multiple-command scenario is long and the number of streaming results is high, the computational complexity of the NLP model is relatively large, and the network bandwidth consumption increases. At the same time, if the streaming results are refreshed incorrectly, the NLP model may extract incorrect commands, such as recognizing "increase the air conditioner temperature" as "increase the air conditioner speed", which will cause command execution errors and affect the overall system effect.

[0047] Regarding the recognition technology of multiple instructions in one sentence, the related technology also provides another voice command recognition solution. Specifically, for the user's voice input, the voice activity paragraph is first obtained through voice activity detection (VAD), and further, each voice activity paragraph is subjected to voice recognition. Specifically, in order to ensure the accuracy of command recognition, the voice recognition model recognizes the entire voice activity paragraph, and then the voice recognition result of the entire paragraph is sent to the natural language understanding model, so that the voice recognition result of the entire paragraph of the NLP model is used for command extraction, so that the context of each instruction can be combined to determine multiple instructions in the voice at one time. For example, if the user's voice is "close the sunroof and turn on the air conditioner", the voice recognition result sent to the natural language understanding model is "close the sunroof and turn on the air conditioner". Although the related technology extracts multiple instructions as a whole, it has the problem of low response speed.

[0048] In view of the above technical problems existing in the related art, VAD is performed on the audio to be processed to determine the voice activity sub-paragraph in the audio to be processed, wherein the voice activity sub-paragraph is the audio between the starting state and the quasi-end state obtained in sequence through detection. Further, in the process of performing voice recognition on the current i-th voice activity sub-paragraph, in response to the quasi-end state obtained by detection, the i-th streaming voice recognition result corresponding to the i-th voice activity sub-paragraph is obtained. The i-th streaming voice recognition result is input into the natural language processing NLP model, wherein the NLP model outputs the i-th instruction corresponding to the i-th voice activity sub-paragraph. It can be seen that the SR model in the embodiment of the present application takes the voice activity sub-paragraph as a unit, and inputs the voice recognition result into the NLP model, so that the NLP model uses the semantic recognition result corresponding to the voice activity sub-paragraph in each recognition process, thereby identifying multiple instructions contained in the audio to be processed. Compared with the related art, the NLP model in the embodiment of the present application extracts instructions by taking each voice activity sub-paragraph as a unit, which can reduce the amount of calculation of the NLP model and help reduce resource usage. At the same time, since there are often some pauses between multiple instructions when issuing multiple instructions in one sentence, the embodiment of the present application simulates the pauses between multiple instructions in one sentence by detecting the quasi-end state. Thus, the NLP model extracts instructions by taking each speech activity sub-paragraph as a unit, which is also conducive to improving the accuracy of instruction extraction.

[0049] Below through Figure 1 An exemplary system architecture of an embodiment of the present application is introduced.

[0050] For example, Figure 1 Schematic diagram of a system architecture 100 for an application environment of a streaming recognition solution for voice commands provided in an embodiment of the present application. Figure 1As shown, the system architecture of the implementation environment of the embodiment of the present application may include: a terminal 102 and a cloud server 104, and the terminal and the cloud server may be connected via a network. Among them, the streaming recognition system of the voice command may be deployed in the terminal 102, or may be deployed in the cloud server 104.

[0051] Exemplarily, the terminal 102 can be used to collect audio and / or play audio. Exemplarily, the terminal 102 includes a vehicle terminal, a computer, a smart phone, a tablet, an intelligent voice interaction device, a smart home appliance, an aircraft, a wearable smart device, a medical device, etc., but is not limited thereto. The cloud server 104 can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers.

[0052] In an exemplary embodiment, reference Figure 1 The terminal 102 with audio collection function collects the audio to be processed, such as a vehicle-mounted terminal or a mobile phone terminal with a microphone function.

[0053] Exemplary references Figure 2 In the case where the streaming recognition system for voice commands is deployed on the terminal 102, after the terminal 102 collects the audio to be processed 20, the VDA processing module 210 in the local streaming recognition system for voice commands performs voice activity detection on the audio to be processed to determine the voice activity sub-segment in the audio to be processed, such as Figure 2 In the i-th voice activity sub-paragraph 22. Further, the SR model 220 in the streaming recognition system based on the local voice command performs voice recognition on the current i-th voice activity sub-paragraph and outputs a voice recognition result 24; during the recognition process, the voice recognition engine obtains the i-th streaming voice recognition result corresponding to the i-th voice activity sub-paragraph in response to the detection of the quasi-end state, such as Figure 2 Further, the speech recognition engine inputs the i-th streaming speech recognition result into the natural language processing NLP model 230, wherein the NLP model 230 outputs the i-th instruction corresponding to the i-th speech activity sub-paragraph, such as Figure 2 28 in.

[0054] Exemplary references Figure 2In the case where the streaming recognition system for voice commands is deployed on the cloud server 102, after the terminal 102 collects the audio 20 to be processed, it transmits the audio 20 to be processed to the streaming recognition system for voice commands in the cloud through the network, and further performs voice activity detection on the audio to be processed through the VDA processing module 210 to determine the voice activity sub-paragraph in the audio to be processed. Furthermore, the SR model 220 in the streaming recognition system for voice commands based on the cloud performs voice recognition on the current i-th voice activity sub-paragraph; during the recognition process, the voice recognition engine obtains a quasi-end state in response to the detection, and obtains the i-th streaming voice recognition result 26 corresponding to the i-th voice activity sub-paragraph. Furthermore, the voice recognition engine inputs the i-th streaming voice recognition result into the natural language processing NLP model 230, wherein the NLP model outputs the i-th instruction 28 corresponding to the i-th voice activity sub-paragraph.

[0055] It is understandable that part of the streaming recognition system of voice commands is deployed on the terminal 102, and the other part is deployed on the cloud server 104. In this case, the streaming recognition process of voice commands will be implemented based on the terminal and the cloud server. For example, the VAD processing module and the SR model are deployed on the terminal 102, and the NLP model is deployed on the cloud server 104. The embodiment of the present application does not limit this.

[0056] Exemplarily, the instructions obtained after the streaming recognition of the voice instructions can be output by the terminal 102 .

[0057] It should be noted that Figure 1 The system architecture of the implementation environment of the embodiment of the present application is exemplified, but the system architecture of the implementation environment of the embodiment of the present application is not limited to Figure 1 shown.

[0058] The following describes in detail the streaming recognition method of the voice command of the embodiment of the present application through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0059] Figure 3 The flowchart of a method P300 for stream recognition of voice commands provided in an embodiment of the present application is shown in FIG. Figure 1 The server 104 or the terminal 102 is shown.

[0060] In step S310, voice activity detection is performed on the audio to be processed to determine a voice activity sub-paragraph in the audio to be processed, wherein the voice activity sub-paragraph is the audio between a start state and a quasi-end state obtained in sequence through detection.

[0061] The above-mentioned audio to be processed may be the audio sent by the user when issuing the command, which contains the command information. Exemplarily, the above-mentioned audio to be processed may be collected by a microphone device. Exemplarily, the audio collected by the microphone may be processed by echo cancellation technology to obtain real-time multi-channel audio data so as to extract a pure voice signal from a complex acoustic environment, thereby providing accurate and reliable input information for the subsequent simultaneous interpretation model.

[0062] VAD technology is used to identify voice-active segments and non-voice-active segments in audio signals. Voice-active segments refer to audio containing voice information, and non-voice-active segments refer to audio that does not contain voice information. VAD technology determines voice-active segments and non-voice-active segments in audio signals by detecting voice endpoints in audio signals. In related technologies, the following states can be determined by VAD technology: START (starting state, indicating the endpoint where voice information starts), CONTINUE (continuing state, indicating that voice information continues), END (ending state, indicating the endpoint where voice information ends, but it does not mean that the entire voice input has been completed), FINISH (final state, indicating that the entire voice input has completely ended and there is no new voice activity). Specifically, the audio between START and FINISH that appear in sequence can be determined as a voice-active segment, and the audio between FINISH and START that appear in sequence can be determined as a non-voice-active segment.

[0063] Specifically, in the related art, when the accumulated silence length after the valid speech segment reaches the VAD silence interval (VADgap), the END state is returned, indicating that the clause ends. Usually, the VAD gap ranges from 400ms to 600ms. Figure 4A , the time interval between the audio signals corresponding to instruction 1 (close the sunroof window) and instruction 2 (turn on the air conditioner) is 80 milliseconds, which is less than the above-mentioned VAD gap. Therefore, the end state (END state) will not be output between the audio signals corresponding to instruction 1 and instruction 2. The accumulated silence length after instruction 5 (I’m hungry) reaches 480 milliseconds, which is within the value range of the above-mentioned VAD gap, so the end state (END state) is output. Therefore, if based on the VAD technology provided by the relevant technology, the above-mentioned 5 instructions can be divided into the same voice activity paragraph.

[0064] However, when multiple instructions are given in one sentence (such as Figure 4A or Figure 4BThere are often natural pauses between instructions, and the pauses between instructions can serve as natural semantic segmentation points. Recognizing instructions based on the audio between the adjacent semantic segmentation points will help improve the accuracy of instruction extraction. The pauses between instructions are relatively short, specifically, generally smaller than the VAD gap. As mentioned above, the VAD technology provided by the relevant technology cannot divide different instructions into different voice activity segments.

[0065] In view of the above problems, an improved VAD technology is provided in the embodiment of the present application. Specifically, the audio to be processed is input into the VAD processing module, and when the VAD processing module detects that the characteristics of the audio signal meet the first preset condition, the starting state is output, such as START in the related art. When the VAD processing module detects that the characteristics of the audio signal do not meet the first preset condition and last for a first preset time, the quasi-end state (PreEND) is output.

[0066] Since the energy, frequency characteristics and time domain changes of audio signals can be used to determine whether an audio segment contains voice activities, the characteristics of the audio signal used to determine whether the first preset condition is met in the embodiment of the present application may include at least one of the following: the energy level of the audio signal, the spectrum characteristics, and the change characteristics in the time domain. Exemplarily, if it is detected that the energy level of the a-th frame audio signal exceeds the energy threshold and the energy level of at least one previous frame audio signal does not exceed the energy threshold, the a-th frame audio signal can be corresponding to the start state (START). If it is detected that the energy level of the b-th frame audio signal does not exceed the energy threshold and the energy levels of the audio frames within the next first preset time length do not exceed the energy threshold, the quasi-end state (PreEND) can be output for the first preset time length. Since the pause duration between multiple instructions in a sentence is short, the embodiment of the present application introduces a VAD stream silence interval (VAD stream gap) (i.e., the above-mentioned first preset time length), wherein the value of the VAD stream gap is less than the above-mentioned VAD gap, so that the above-mentioned VAD gap can be used. Figure 4A The entire voice activity segment shown is divided into multiple sub-segments (specifically referred to as "voice activity sub-segments"). Specifically, the value of VAD stream gap is related to actual needs. For example, the VAD stream gap value is 50ms, but the embodiment of the present application does not limit the specific value of VAD stream gap. For another example, the first preset duration VAD stream gap is a shorter duration than the above-mentioned VAD gap, for example, the first preset duration is less than the first threshold, and the exemplary value of the first threshold is 200.

[0067] Therefore, when the accumulated silence length after the valid speech segment reaches the VAD stream gap, the state is returned to the PreEND state. Specifically, in this embodiment, the audio between the start state and the quasi-end state obtained by detection is determined as the speech activity sub-segment. Figure 4B , the time interval between the audio signals corresponding to instruction 1 (close the sunroof window) and instruction 2 (turn on the air conditioner) is 80 milliseconds, which is smaller than the above-mentioned VAD gap but larger than the above-mentioned VAD stream gap (50 milliseconds). Therefore, the quasi-end state 1 (PreEND-1) can be output, thereby determining that instruction 1 "close the sunroof window" belongs to voice activity sub-paragraph 1; similarly, the time interval between the audio signals corresponding to instruction 2 (turn on the air conditioner) and instruction 3 (turn off seat massage) is 60 milliseconds, which is larger than the above-mentioned VAD stream gap (50 milliseconds). Therefore, the quasi-end state 2 (PreEND-2) can be output, thereby determining that instruction 2 "turn on the air conditioner" belongs to voice activity sub-paragraph 2; and so on.

[0068] It can be seen that compared with the related technology, the improved VAD technology provided in the embodiment of the present application can achieve more detailed segmentation of audio. In the scenario of multiple instructions in one sentence, if the related technology is used, multiple instructions in the sentence may be divided into the same voice activity paragraph, such as Figure 4A As shown; if the improved VAD technology provided in the embodiment of the present application is adopted, different instructions in the sentence can be divided into different voice activity sub-paragraphs, such as Figure 4B shown.

[0069] Continue to refer Figure 3 In step S320, during the process of performing speech recognition on the current i-th voice-active sub-paragraph, in response to detecting a quasi-end state, an i-th streaming speech recognition result corresponding to the i-th voice-active sub-paragraph is obtained, wherein the value of i is a positive integer not greater than the total number of voice-active sub-paragraphs in the audio to be processed.

[0070] In the solution provided by the embodiment of the present application, the audio to be processed is detected by the improved VAD technology, and with the continuous input of audio, the voice activity sub-paragraphs can be determined from the input audio one by one. As mentioned above, in a scenario with multiple instructions in one sentence, there is a natural small pause between different instructions. The quasi-end state can accurately locate the segmentation point between instructions, thereby obtaining the voice activity sub-paragraph corresponding to the instruction.

[0071] In the following embodiments of the present application, the streaming results of the input NLP model are determined in units of voice activity sub-paragraphs. Specifically, in the process of performing speech recognition on the current i-th voice activity sub-paragraph, in response to the detection of the quasi-end state PreEND, it is indicated that the segmentation point between the current sub-paragraph and the next field is reached, and the i-th streaming speech recognition result corresponding to the i-th voice activity sub-paragraph can be obtained. Specifically, in response to the detection of the quasi-end state, an implementation method for determining the i-th streaming recognition result is as follows:

[0072] In step S11, the speech recognition result of the i-th speech activity sub-paragraph is obtained to obtain the speech information of the current sub-paragraph. Figure 5A , the voice recognition result of voice activity sub-paragraph 1 is "Close the sunroof window", that is, voice activity sub-paragraph 1 is: "Close the sunroof window"; the voice recognition result of voice activity sub-paragraph 2 is "Turn on the air conditioner", that is, voice activity sub-paragraph 2 is: "Turn on the air conditioner"; the voice recognition result of voice activity sub-paragraph 3 is "Turn on the air conditioner", that is, voice activity sub-paragraph 2 is: "Turn off seat massage"; and so on.

[0073] In step S12, all speech recognition results that have been input into the NLP model in the audio to be processed are obtained to obtain the contextual speech information of the current sub-paragraph. Figure 5A For speech active sub-paragraph 1, since it is the first sub-paragraph recognized in the audio to be processed, all speech recognition results that have been input into the NLP model do not exist yet; for speech active sub-paragraph 2, it is the second sub-paragraph recognized in the audio to be processed, so all speech recognition results that have been input into the NLP model are “close the sunroof window”, so the contextual speech information of sub-paragraph 2 is: “close the sunroof window”; for speech active sub-paragraph 3, it is the third sub-paragraph recognized in the audio to be processed, so all speech recognition results that have been input into the NLP model are “close the sunroof window and turn on the air conditioner”, so the contextual speech information of sub-paragraph 3 is: “close the sunroof window, close the sunroof window, turn on the air conditioner”; and so on.

[0074] In step S13, the current sub-paragraph voice information and the context voice information of the current sub-paragraph are combined to obtain the i-th streaming voice recognition result. Exemplarily, referring to FIG5 , for voice active sub-paragraph 1, its voice recognition result is “close the sunroof window”, and the context voice information of the current sub-paragraph is empty, then the first streaming voice recognition result is obtained: “close the sunroof window”; for voice active sub-paragraph 2, its voice recognition result is “turn on the air conditioner”, and the context voice information of the current sub-paragraph is “close the sunroof window”, then the second streaming voice recognition result is obtained: “close the sunroof window and turn on the air conditioner”; and so on. Exemplary reference Figure 5A :

[0075] In response to VAD detecting the quasi-end state of “voice activity sub-paragraph 1”, according to the speech recognition result of the current SR model, determining that the corresponding streaming speech recognition result is: “close the sunroof window”;

[0076] In response to VAD detecting the quasi-end state of “voice activity sub-paragraph 2”, according to the speech recognition result of the current SR model, the corresponding streaming speech recognition result is determined to be: “close the sunroof window and turn on the air conditioner”;

[0077] In response to VAD detecting the quasi-end state of “voice activity sub-paragraph 3”, according to the speech recognition result of the current SR model, it is determined that the corresponding streaming speech recognition result is: “close the sunroof, turn on the air conditioner, turn off the seat massage”;

[0078] In response to VAD detecting the quasi-end state of “voice activity sub-paragraph 4”, according to the speech recognition result of the current SR model, it is determined that the corresponding streaming speech recognition result is: “Close the sunroof, turn on the air conditioner, turn off the seat massage, and adjust the air conditioner to 25 degrees”;

[0079] In response to VAD detecting the quasi-end state of "voice activity sub-paragraph 5", according to the speech recognition result of the current SR model, the corresponding streaming speech recognition result is determined as: "Close the sunroof, turn on the air conditioner, turn off seat massage, adjust the air conditioner to 25 degrees, I'm hungry."

[0080] For example, if the five instructions in the above sentence are extracted by the relevant technology, the streaming speech recognition result of the NLP model is as follows: Figure 5BAs shown. Referring to 5A and 5B, it can be seen that, on the first aspect, the streaming result of the first input into the NLP model is "off", and the streaming result of the second input into the NLP model is "closed"... In the related art, after the SR model recognizes a small number (such as a word), it generates a streaming result based on the recognition result and inputs it into the NLP model. The five instructions in the above sentence correspond to 29 streaming results, and then the NLP model executes 29 processing steps. The related art significantly increases the consumption of computing resources and network bandwidth; while the NLP model of the same instruction in the present application only needs to execute five processing steps, so the embodiment of the present application can significantly reduce the number of times the NLP model is input, thereby achieving the effect of saving computing resources and network bandwidth. Secondly, the streaming results provided by the related technology cannot accurately summarize sentences and ensure semantic integrity. For example, the instruction "open the window" extracted by the 8th streaming result is actually wrong, and the 16th streaming result also contains the instruction "seat massage" with unclear meaning. Therefore, the instruction extraction accuracy of the NLP model in the related technology will be affected; in the embodiment of the present application, the streaming results are determined in response to PreEND, and PreEND is used to simulate the semantic segmentation point of the instruction, so the semantic integrity can be guaranteed accordingly, which is beneficial to the accuracy of instruction extraction.

[0081] It can be seen that the quasi-end state determined by the improved VAD technology provided in the embodiment of the present application is used to simulate the short pauses between instructions. Further based on this state, the streaming results for inputting NLP are determined, which has the advantages of saving computing resources and network bandwidth while ensuring the response rate of the instructions as mentioned above, and can also improve the accuracy of instruction extraction. In a further embodiment, since there may be certain instability in the short pauses between instructions based on the quasi-end state positioning, in order to reduce the inaccurate instruction recognition caused by inaccurate positioning (for example, if the interval between two adjacent instructions output by the user is too short or there is no pause interval, the end state cannot be determined based on the above-mentioned improved VAD technology), the embodiment of the present application also provides the following embodiments.

[0082] Figure 6 A flow chart of a method for determining a streaming semantic recognition result provided in an embodiment of the present application may be used as a specific implementation of step S320.

[0083] In step S320-1, in the process of performing speech recognition on the current i-th voice activity sub-paragraph, before the quasi-end state is detected, the speech recognition result subset that has been generated and not input into the NLP model is continuously compared with the second preset condition. And, in step S320-2, it is determined whether the speech recognition result subset that has been generated and not input into the NLP model satisfies the second preset condition.

[0084] Because Figure 6The embodiment shown is for the situation where the quasi-end state cannot be accurately detected due to reasons such as no pauses or short intervals between instructions. The specific second preset condition mentioned above is used to measure whether the amount of information of the comparison object (a subset of speech recognition results that has been generated and not input into the NLP model) is sufficient. If the second preset condition is met, it can be said that the current amount of information of the comparison object is sufficient, that is, the information contained in the current comparison object can be used as an instruction. Even if the conditions for triggering the quasi-end state have not been met, a streaming result still needs to be generated for the NLP model to extract instructions. If the second preset condition is not met, it means that the current amount of information of the comparison object is not sufficient, that is, the information contained in the current comparison object cannot be used as an instruction, and speech recognition still needs to be continued through the SR model, and the conditions for triggering the generation of streaming results have not been met.

[0085] Exemplarily, when the type of the speech recognition result is a text string, the second preset condition is whether the length of the text string is greater than a second threshold. Specifically, if the length of the comparison object is greater than the second threshold, it means that the second preset condition is met; if the length of the comparison object is less than or equal to the second threshold, it means that the second preset condition is not met.

[0086] For example, still taking the above sentence and five instructions as an example, if there is no pause between the second instruction and the third instruction, and if the current recognition result of the SR model is "turn on the air conditioner", then the specific method of determining the above comparison object is: in the process of voice recognition of the current second voice activity sub-paragraph, before the quasi-end state is detected, the voice recognition result that has been generated and is "open the sunroof window, turn on the air conditioner", and the voice recognition result that has been input into the NLP model is "open the sunroof window", then the subset of voice recognition results that has not been input into the NLP model is: "turn on the air conditioner", so the comparison object can be determined as "turn on the air conditioner".

[0087] In an exemplary embodiment, reference Figure 6 If the judgment result is that the comparison object does not satisfy the second preset condition, then step S320-3 or step S320-1 may be executed in this case.

[0088] In step S320-3, in response to detecting a quasi-end state, the i-th streaming speech recognition result corresponding to the i-th voice activity sub-paragraph is obtained; in this case, after the i-th streaming speech recognition result is input into the NLP model, the NLP model outputs the i-th instruction corresponding to the i-th voice activity sub-paragraph.

[0089] As mentioned above, if the second preset condition is not met, it means that the information contained in the current comparison object can still be used as an instruction, and the SR model is still required to continue speech recognition. Before obtaining the quasi-end state mark, steps S320-1 and S320-2 are repeatedly executed until the quasi-end state is detected. In particular, the method for determining the i-th streaming speech recognition result when the quasi-end state is detected is the same as steps S11 to S13, which will not be repeated here.

[0090] In an exemplary embodiment, reference Figure 6 If the judgment result is that the above-mentioned comparison object satisfies the above-mentioned second preset condition, then step S320-4 can be executed in this case: obtain the i-th streaming speech recognition result corresponding to the speech recognition result subset; in this case, after the i-th streaming speech recognition result is input into the NLP model, the NLP model outputs an instruction in the instruction set corresponding to the i-th speech activity sub-paragraph.

[0091] Specifically, in response to the comparison object satisfying the second preset condition, an implementation method of determining the i-th stream recognition result is as follows:

[0092] In step S21, the current speech recognition result of the i-th voice activity sub-paragraph is obtained to obtain the current subset speech information in the i-th voice activity sub-paragraph. Figure 5A In the process of speech recognition of semantic activity sub-paragraph 1, "close", "close skylight", etc. can be used as the current subset speech information in the sub-paragraph. Among them, the current subset speech information in the sub-paragraph belongs to part of the content of the sub-paragraph, not the whole set.

[0093] In step S22, all speech recognition results that have been input into the NLP model in the audio to be processed are obtained to obtain contextual speech information of the current subset; and in step S23, the speech information of the current subset and the contextual speech information of the current subset are combined to obtain the i-th streaming speech recognition result. Figure 5A , and then in the process of speech recognition of semantic activity sub-paragraph 3, the current subset speech information of the sub-paragraph is obtained as "close the seat", then the context speech information of the current subset is "close the sunroof and windows, turn on the air conditioner", so the merged streaming recognition result is: close the sunroof and windows, turn on the air conditioner, and close the seat.

[0094] Through the embodiment shown in FIG5, in the process of performing speech recognition on the current i-th voice activity sub-paragraph, before detecting the quasi-end state, by comparing the comparison object (a subset of speech recognition results that have been generated and not input into the NLP model) with the preset condition (the second preset condition), if the current amount of information of the above comparison object is large, the streaming result can be generated, thereby reducing the inaccurate instruction recognition caused by inaccurate positioning. For example, if the interval between two adjacent instructions output by the user is too short or there is no pause interval, the end state cannot be accurately determined based on the above improved VAD technology. Specifically, by judging whether the above comparison object meets the second preset condition, the instruction segmentation points that lack pauses between each other can be located, which is ultimately beneficial to improving the accuracy of streaming results and instruction extraction.

[0095] As mentioned above, the quasi-end state determined by the improved VAD technology provided in the embodiment of the present application is used to simulate the short pauses between instructions. Further based on this state, the streaming results for inputting NLP are determined, which has the advantages of saving computing resources and network bandwidth while ensuring the response rate of instructions as mentioned above, and can also improve the accuracy of instruction extraction. In a further embodiment, there may be certain instabilities due to reasons such as the short pauses between instructions located based on the quasi-end state and the large amount of information in a single instruction. In order to reduce inaccurate instruction recognition caused by inaccurate positioning, the embodiment of the present application also provides the following embodiments.

[0096] For example, if a pause occurs during a command output by the user, the identified "sub-paragraph" may be semantically incomplete if the end state is determined based on the improved VAD technology. In view of this situation, the present application provides a solution such as step S31.

[0097] In step S31, after the quasi-end state is detected and before the i-th stream recognition result is determined, the semantic integrity check is performed on the i-th voice activity sub-paragraph. Figure 5AAs shown in , if there is a pause in the second instruction "turn on the air conditioner", if after the above-mentioned improvement, a quasi-end state A is determined after "turn on" in the VAD technology processing process, that is, a sub-paragraph a "turn on" is obtained, then in this embodiment, the semantic integrity check of sub-paragraph a can be determined, and the semantics of the sub-paragraph a can be determined to be incomplete, so that the quasi-end state A can be explained as false, that is, the streaming result should not be generated in response to the quasi-state A, but the speech recognition is continued. Specifically, after reaching the next quasi-end state or the recognition result that has been recognized but not input into the NLP model meets the second preset condition, the semantic integrity check is performed again until the semantic integrity meets the preset requirement. For example, after determining that the semantics of sub-paragraph a is incomplete, the speech recognition is continued, and the next quasi-end state is reached to obtain sub-paragraph b "air conditioner", then the sub-paragraph a and sub-paragraph b are combined for semantic integrity check, and after determining that the semantic integrity meets the preset requirement, the streaming result corresponding to sub-paragraph a and sub-paragraph b ("turn on the air conditioner") is determined: close the sunroof window and turn on the air conditioner.

[0098] For example, if a command output by the user has a large amount of information (such as a long sentence), the "sub-paragraph" identified based on steps S320-1 and S320-2 may have incomplete semantics. In view of this situation, the present application embodiment provides a solution such as step S31'.

[0099] In step S31', when the comparison object (a subset of speech recognition results that has been generated and not input into the NLP model) meets the second preset condition and before determining the i-th stream recognition result, the semantic integrity check is performed on the comparison object. Figure 5A As shown in , if the first instruction "close the sunroof window" is a long sentence, after the above-mentioned step S320-1 and step S320-2 are processed, it is determined that the comparison object meets the second preset condition after "open the sunroof window", thereby obtaining a sub-paragraph c "open the sunroof window". By performing a semantic integrity check on the sub-paragraph c in this embodiment, it can be determined that the semantics of the sub-paragraph c is incomplete, so the streaming result corresponding to the sub-paragraph c should not be generated, but the speech recognition is continued. Specifically, after reaching the next quasi-end state or the recognition result that has been recognized but not input into the NLP model meets the second preset condition, the semantic integrity check is performed again until the semantic integrity meets the preset requirement. For example, after determining that the semantics of the sub-paragraph c is incomplete, the speech recognition is continued, and the sub-paragraph d "window" is obtained when the next quasi-end state is reached, then the sub-paragraph c and the sub-paragraph d are combined for semantic integrity check, and after determining that the semantic integrity meets the preset requirement, the streaming results corresponding to the sub-paragraph c and the sub-paragraph d ("open the sunroof window") are determined.

[0100] By performing semantic integrity detection on the sub-paragraphs in the above embodiment, the semantic integrity of the streaming results can be guaranteed, which is ultimately beneficial to the accuracy of instruction extraction.

[0101] After obtaining the streaming results (the i-th streaming speech recognition results) through the above embodiment, before inputting the i-th streaming recognition results into the NLP model, a semantic integrity check can also be performed on the streaming results to ensure the semantic integrity of the information input into the NLP model. This not only ensures accurate extraction of instructions, but also helps to reduce the computational complexity of the NLP model.

[0102] Figure 7 A flow chart of a method for performing semantic integrity detection on a streaming result provided in an embodiment of the present application. In the embodiment shown in the figure, steps S710 to S740 can be performed between steps S320 and S330, that is, after executing step S320 to obtain the i-th streaming speech recognition result, steps S710 to S740 are performed, and after determining in step S710 that the semantic integrity requirement is met, step S330 is performed.

[0103] refer to Figure 7 In step S710, a semantic integrity check is performed on the i-th streaming speech recognition result to determine whether its semantic integrity is greater than the latest third threshold.

[0104] Exemplarily, the i-th stream speech recognition result can be input into a trained large language model to perform semantic integrity analysis on the i-th stream speech recognition result through the large language model, wherein the output information of the large language model is used to determine whether the semantic integrity of the i-th stream speech recognition result meets the third preset condition.

[0105] Among them, the trained large language model is able to capture the intrinsic semantic structural features in the audio, enabling it to demonstrate excellent accuracy and robustness in identifying and understanding semantically complete points, thus enabling the large language model to have the ability to identify and understand semantically complete information.

[0106] Exemplarily, the i-th streaming speech recognition result and the related prompt words are input into a trained large language model to determine whether the semantic integrity of the i-th streaming speech recognition result is greater than a third threshold through the large language model. Specifically, if the semantic integrity of the i-th streaming speech recognition result is greater than the third threshold, it means that the semantic integrity meets the preset requirements and can be processed by the NLP model, then step S330 is executed. If the speech integrity of the i-th streaming speech recognition result is not greater than the third threshold, it means that the semantic integrity does not meet the preset requirements and cannot be processed by the NLP model to avoid inaccurate instruction extraction and waste of computing resources and network bandwidth, then step S720 is executed: suspend inputting information to the NLP model, and wait for the i+1-th streaming speech recognition result to be combined to perform semantic integrity detection.

[0107] For example, reference Figure 5A For the first streaming speech recognition result “Close the sunroof window”, execute step S710. If the semantic integrity is less than the third threshold Th3, wait for the second streaming speech recognition result “Close the sunroof window and turn on the air conditioner”, and execute step S710 again to determine whether the semantic integrity of the second streaming speech recognition result meets the preset requirements.

[0108] refer to Figure 7 It should be noted that, for the flexibility of the instruction recognition system, the third threshold can also be adjusted dynamically. Therefore, after executing step S720, in addition to executing step S710 again, step S730 is also executed: whether the number of times n that the semantic integrity is continuously not up to standard (not greater than the third threshold) reaches the fourth threshold Th4. Since the information for semantic integrity judgment is streaming information, the amount of information increases with the increase of i value. Therefore, if the number of times that the semantic integrity is continuously not greater than the third threshold Th3 (such as 80%) reaches the fourth threshold Th4, it means that the amount of information of the current streaming result is large; if the streaming result of the larger information still cannot meet the current third threshold Th3, it means that the current requirement for semantic integrity may be too high, then step S740 can be executed: lower the third threshold Th3 according to the preset strategy to reduce the requirement for semantic integrity. The preset strategy can be set according to actual needs. It should be noted that the adjusted third threshold Th3 should be greater than another threshold (such as the fifth threshold, and the specific value of the fifth threshold can be 60%), so as to ensure that the semantic integrity judgment works.

[0109] Exemplarily, in order to improve execution efficiency, after executing step S720, step S730 may be executed first to determine whether the current third threshold value needs to be adjusted. If adjustment is required, step S740 is executed before step S710, so as to timely perform the completeness judgment process on the streaming result through the latest third threshold value. Of course, the embodiment of the present application does not limit the execution order of step S710 and step S730 executed again after step S720. In addition to the restrictive step S730, step S730 and step S710 may also be executed simultaneously, or step S710 may be executed first after step S720 and then step S730, all of which belong to the protection scope of the embodiment of the present application.

[0110] Continuing with the above example, the following is an example of executing step S730 first after step S720 and then executing step S710.

[0111] Referring to FIG5 , after determining that the semantic integrity of the first streaming speech recognition result “Close the sunroof window” is less than the third threshold Th3 through step S710, step S730 is executed: to determine that the number of times the semantic integrity is continuously not greater than the third threshold Th3 is 1, if the fourth threshold Th4 is 2, then step S740 does not need to be executed at present, and the current third threshold Th3 is still used. Step S710 is then executed to determine whether the semantic integrity of the second streaming speech recognition result is greater than the third threshold Th3.

[0112] If after executing step S710 again, it is determined that the semantic completeness of the second streaming speech recognition result is still not greater than the third threshold value Th3, then execute step S720 to wait for the third streaming speech recognition result "Close the sunroof, turn on the air conditioner, and turn off the massage seat." Further, execute step S730: if the number of times n that the semantic completeness is continuously not greater than the third threshold value Th3 is 2, then the fourth threshold value Th4 (value is 2) is reached, so execute step S740 to lower the value of the third threshold value Th3. Then execute step S710 to determine whether the semantic completeness of the third streaming speech recognition result meets the requirements based on the latest Th3. It can be understood that after adjusting the third threshold, the number of consecutive semantic incompleteness n is reset to zero, so as to be used as the basis for adjusting the third threshold value Th3 next time.

[0113] Continue to refer Figure 7 In the above embodiment, the judgment result of step S710 is introduced as the situation that the current streaming result does not meet the preset requirements (i.e., not greater than the latest third threshold Th3); in the embodiment of the present application, the above semantic completeness judgment process is performed before the streaming results are input into the NLP model. This can ensure that the streaming results input into the NLP model have a high semantic completeness, which is beneficial to the accuracy of the NLP processing results. At the same time, it can reduce the number of times the NLP model processes semantically incomplete streaming results, which is beneficial to improving resource utilization.

[0114] Next, the case where the judgment result of step S710 is that the current streaming result meets the preset requirements (i.e., is greater than the latest third threshold Th3) is introduced. In this case, step S330 is executed: input the i-th streaming speech recognition result into the natural language processing NLP model, where the NLP model outputs the i-th instruction corresponding to the i-th speech activity sub-paragraph.

[0115] Exemplarily, in the embodiment of the present application, the NLP model extracts instructions based on the streaming speech recognition result. After extracting the i-th instruction, the instruction can be executed based on the solution as shown in Figure 8 At the same time, the NLP model performs subsequent instruction extraction based on the streaming speech result. It can be seen that the embodiment of the present application can provide a more timely instruction response. Since the instructions are continuously extracted along with the speech input process, the instruction execution can also be performed in a streaming manner during the speech input process. Therefore, the user can perceive a stronger streaming effect: the first instruction has started to be executed during the subsequent instruction input of the speech, and the user experience will be better.

[0116] Exemplarily, the NLP model is used to understand and parse the streaming speech recognition result and extract specific instructions. Among them, the above NLP model can be a pre-trained language model, such as a Bidirectional Encoder Representations from Transformers (BERT), or a T5 (Text-to-Text Transfer Transformer) model, etc. The embodiment of the present application does not limit the NLP model, and it can be any algorithm model that can understand and parse the streaming speech recognition result.

[0117] Exemplarily, the process of the NLP model extracting instructions includes the following steps.

[0118] In step S41, text preprocessing is performed on the i-th streaming speech recognition result.

[0119] Exemplarily, word segmentation processing can be performed to split the input text into words or lexical units. Exemplarily, stop word removal processing can also be performed to remove common meaningless words (such as "de", "shi", etc.) to reduce noise. Exemplarily, normalization processing can also be performed to convert the text into a unified format, such as lowercasing, removing punctuation marks, etc.

[0120] In step S42, a syntactic analysis process is executed. Exemplarily, the dependency relationship between words in a sentence is analyzed to construct a syntactic structure tree to help understand the logical structure of the sentence. Exemplarily, named entity recognition is performed to identify proper nouns in the text, such as person names, place names, time, etc., in order to better understand the context.

[0121] In step S43, a semantic parsing process is performed. Exemplarily, intent recognition is performed to determine the main intent or purpose expressed by the user. For example, "turn on the air conditioner" indicates that the operation the user wants to perform is to turn on the air conditioner. Slot filling processing can also be performed to specifically extract parameters or attribute values ​​related to a specific intent. For example, in "set the temperature to 25 degrees", "25 degrees" is a slot, indicating the target temperature.

[0122] In step S44, the state of the conversation is maintained by state tracking, the context information of the current conversation is recorded, and the consistency and coherence of multiple rounds of conversations are ensured. In addition, the next response or operation is determined based on the conversation state and user intention.

[0123] In step S45, the parsed intent and slot information are mapped to specific commands or application program interface calls that can be executed by the system. If necessary, confirmation information or further prompts are provided to the user to ensure that the instructions are correctly understood and executed.

[0124] Figure 8 A flowchart of a method for processing extracted instructions in an embodiment of the present application is provided.

[0125] refer to Figure 8 , the i-th streaming speech recognition result is input into the natural language processing module 810, and the instruction is obtained after being processed by the natural language processing model 810. Further, the instruction can be input into the instruction execution system 820 to execute the corresponding instruction. For example, for the instruction "turn on the air conditioner to 25 degrees", the control system interface compatible with the vehicle's internal communication protocol constructs an application interface request, such as set_air_conditioning(temperature=25); the air conditioning control system receives and executes the command to adjust the temperature to 25 degrees. Exemplarily, a voice feedback mechanism can also be provided to feedback the voice to the user: "The air conditioner is set to 25 degrees." Exemplarily, when the instruction "I'm hungry" is extracted, a voice feedback mechanism can be provided to feedback the relevant voice of nearby restaurants to the user.

[0126] refer to Figure 8, the instruction can be input into the speech synthesis system 830 to provide voice broadcast for the relevant instructions. Assuming that the system needs to broadcast "the air conditioner has been set to 25 degrees", the specific implementation process can be step S51: generate feedback text according to the instruction. For example, if the instruction is "adjust the air conditioner to 25 degrees", the feedback text can be: "The air conditioner has been set to 25 degrees". Step S52: Analyze the text and convert it into a phoneme sequence: "air / adjustment / has / been / set / to / 25 / degrees"; then use the pre-trained speech synthesis model to generate the corresponding audio waveform; convert the audio waveform into a format suitable for playback, such as pulse code modulation. Step S53: Adjust the volume to adapt to the in-car environment. Exemplarily, audio enhancement technology can also be applied to ensure clear speech.

[0127] The above embodiments generally introduce the specific embodiments of the streaming recognition method of voice commands. Fig. 9 A specific implementation of a method for streaming recognition of voice commands is provided.

[0128] refer to Fig. 9 In step S91, the speech recognition result output by the SR model is obtained.

[0129] Exemplarily, in the process of receiving the audio to be processed, the audio is processed by the improved VAD technology. As the speech activity sub-paragraph is generated, the sub-paragraph is recognized by the SR model, wherein the sub-paragraph is recognized while being generated. For example, after determining the Start state, the subsequent audio is recognized without waiting for PreEND to recognize the entire sub-paragraph, thereby effectively ensuring the efficiency of instruction extraction and execution response.

[0130] In step S92, it is determined whether the PreEND condition is satisfied.

[0131] If the PreEND condition is met, the streaming result (i.e., the aforementioned streaming speech recognition result of the ith sentence) is determined. As mentioned above, in a scenario where there are multiple instructions in one sentence, there are naturally small pauses between different instructions. The above quasi-end state can accurately locate the segmentation point between instructions, thereby obtaining the speech activity sub-paragraph corresponding to the instruction.

[0132] In the embodiment of the present application, the streaming result of the input NLP model is determined in units of voice activity sub-paragraphs. Specifically, in the process of performing voice recognition on the current i-th voice activity sub-paragraph, in response to the detection of obtaining the quasi-end state PreEND, it indicates that the division point between the current sub-paragraph and the next field has been reached, and the i-th streaming voice recognition result corresponding to the i-th voice activity sub-paragraph can be obtained.

[0133] The implementation method of determining the screenshot of the streaming result when the PreEND condition is met is as shown in steps S11 to S13, which will not be repeated here.

[0134] If the PreEND condition is not met, step S93 may be further executed: determining whether the information volume condition is met.

[0135] Among them, the specific implementation of step S93 is the same as the specific implementation of step S320-1 and step S320-2, which will not be repeated here.

[0136] If the information volume condition is met, the streaming result is determined. The implementation method of determining the screenshot of the streaming result when the information volume condition is met is as shown in steps S21 to S23, which will not be repeated here.

[0137] refer to Fig. 9 After determining the streaming result, execute step S94: perform semantic completeness detection on the streaming result.

[0138] And in step S95, it is determined whether the semantic completeness is greater than Th3.

[0139] The specific implementation of steps S94 to S95 is similar to the specific implementation of step S710, and will not be described in detail here.

[0140] If it is determined that the semantic completeness of the streaming result is greater than Th3, step S96 is executed: the streaming result is input into the NLP model to extract instructions.

[0141] After determining that the semantic completeness of the streaming result is not greater than Th3, step S97 is executed: determining whether the number of consecutive semantic incompleteness n reaches Th4.

[0142] If n reaches Th4, then step S98 is executed: the value of n is reset to zero, Th3 is adjusted down and reset, so as to perform semantic integrity detection based on the updated Th3 in step S95. If n does not reach Th4, then the next streaming result is waited for.

[0143] The streaming recognition scheme of voice instructions provided in the embodiment of the present application performs VAD on the audio to be processed to determine the voice activity sub-paragraphs in the audio to be processed, wherein the voice activity sub-paragraphs are the audio between the starting state and the quasi-end state obtained in sequence through detection. Further, in the process of performing voice recognition on the current i-th voice activity sub-paragraph, in response to the detection of the quasi-end state, the i-th streaming voice recognition result corresponding to the i-th voice activity sub-paragraph is obtained. The i-th streaming voice recognition result is input into the natural language processing NLP model, wherein the NLP model outputs the i-th instruction corresponding to the i-th voice activity sub-paragraph. It can be seen that the SR model in the embodiment of the present application takes the voice activity sub-paragraph as a unit, and inputs the voice recognition result into the NLP model, so that the NLP model uses the semantic recognition result corresponding to the voice activity sub-paragraph in each recognition process, thereby identifying multiple instructions contained in the audio to be processed. Compared with the related art, the NLP model in the embodiment of the present application extracts instructions by taking each voice activity sub-paragraph as a unit, which can reduce the computational amount of the NLP model and help reduce resource usage. At the same time, since there are often some pauses between multiple instructions when issuing multiple instructions in one sentence, the embodiment of the present application simulates the pauses between multiple instructions in one sentence by detecting the quasi-end state. Thus, the NLP model extracts instructions by taking each speech activity sub-paragraph as a unit, which is also conducive to improving the accuracy of instruction extraction.

[0144] Combination of the above Figures 1 to 9 , describes the method embodiment of the present application, and the following is combined with Fig.10 An embodiment of a streaming recognition device for voice commands of the present application is described.

[0145] Fig.10 A schematic block diagram of a streaming recognition device 1000 for voice commands provided in an embodiment of the present application.

[0146] refer to Fig.10 The streaming recognition device 1000 for voice commands provided in the embodiment of the present application includes: a voice detection module 1010 , a streaming result acquisition module 1020 and a command extraction module 1030 .

[0147] Among them, the above-mentioned voice detection module 1010 is used to perform voice activity detection on the audio to be processed to determine the voice activity sub-paragraph in the above-mentioned audio to be processed, wherein the above-mentioned voice activity sub-paragraph is the audio between the starting state and the quasi-ending state obtained in sequence through detection; the above-mentioned streaming result acquisition module 1020 is used to obtain the i-th streaming voice recognition result corresponding to the above-mentioned i-th voice activity sub-paragraph in response to the quasi-ending state obtained by detection during the process of performing voice recognition on the current i-th voice activity sub-paragraph, wherein the value of i is a positive integer not greater than the total number of voice activity sub-paragraphs in the above-mentioned audio to be processed; and the above-mentioned instruction extraction module 1030 is used to input the above-mentioned i-th streaming voice recognition result into the natural language processing NLP model, wherein the above-mentioned NLP model outputs the i-th instruction corresponding to the above-mentioned i-th voice activity sub-paragraph.

[0148] In an exemplary embodiment, based on the above scheme, the above voice detection module 1010 is specifically used to: input the audio to be processed into the voice activity detection VAD processing module, and output a start state when the above VAD processing module detects that the characteristics of the audio signal meet the first preset condition; output a quasi-end state when the above VAD processing module detects that the characteristics of the audio signal do not meet the first preset condition and continue for a first preset time;

[0149] Wherein, the first preset duration is less than the first threshold;

[0150] The characteristics of the audio signal include at least one of the following: energy level, spectrum characteristics, and change characteristics in the time domain of the audio signal.

[0151] In an exemplary embodiment, based on the above scheme, the streaming result acquisition module 1020 includes: a comparison unit and a first acquisition unit; wherein the comparison unit is used to: in the process of performing speech recognition on the current i-th voice activity sub-paragraph, before detecting a quasi-end state, continue to compare the speech recognition result subset that has been generated and not input into the above-mentioned NLP model with the second preset condition; and the first acquisition unit is used to: if the comparison result is that the above-mentioned speech recognition result subset does not meet the above-mentioned second preset condition, then in response to detecting a quasi-end state, obtain the i-th streaming speech recognition result corresponding to the above-mentioned i-th voice activity sub-paragraph.

[0152] In an exemplary embodiment, based on the above scheme, the above-mentioned first acquisition unit is specifically used to: obtain the speech recognition result of the above-mentioned i-th voice activity sub-paragraph to obtain the speech information of the current sub-paragraph; obtain all speech recognition results that have been input into the above-mentioned NLP model in the above-mentioned audio to be processed to obtain the contextual speech information of the current sub-paragraph; and, merge the above-mentioned current sub-paragraph speech information and the above-mentioned current sub-paragraph contextual speech information to obtain the above-mentioned i-th streaming speech recognition result.

[0153] In an exemplary embodiment, based on the above solution, the above streaming result acquisition module 1020 further includes: a second acquisition unit;

[0154] Among them, the second acquisition unit is used for: if the comparison result is that the above-mentioned speech recognition result subset satisfies the above-mentioned second preset condition, then obtaining the i-th streaming speech recognition result corresponding to the above-mentioned speech recognition result subset; wherein, after the above-mentioned i-th streaming speech recognition result is input into the natural language processing NLP model, the above-mentioned NLP model outputs an instruction in the instruction set corresponding to the above-mentioned i-th speech activity sub-paragraph.

[0155] In an exemplary embodiment, based on the above scheme, the above-mentioned second acquisition unit is specifically used to: obtain the current speech recognition result of the above-mentioned i-th voice activity sub-paragraph, and obtain the current subset speech information in the above-mentioned i-th voice activity sub-paragraph; obtain all speech recognition results that have been input into the above-mentioned NLP model in the above-mentioned audio to be processed, and obtain the context speech information of the current subset; and, merge the above-mentioned current subset speech information and the above-mentioned current subset context speech information to obtain the above-mentioned i-th streaming speech recognition result.

[0156] In an exemplary embodiment, based on the above scheme, the type of the above speech recognition result is a text string, and the above second preset condition includes that the length of the text string is greater than a second threshold.

[0157] In an exemplary embodiment, based on the above solution, the above device further includes: a semantic integrity detection module;

[0158] Among them, the above-mentioned semantic integrity detection module is used to: before the above-mentioned instruction extraction module inputs the above-mentioned i-th streaming speech recognition result into the natural language processing NLP model, perform semantic integrity detection on the above-mentioned i-th streaming speech recognition result to determine whether its semantic integrity is greater than the third threshold; wherein, when the above-mentioned semantic integrity is greater than the above-mentioned third threshold, the above-mentioned i-th streaming speech recognition result is input into the natural language processing NLP model.

[0159] In an exemplary embodiment, based on the above scheme, the above semantic integrity detection module is also used to: when the above semantic integrity is less than or equal to the above third threshold, suspend the input of information to the above NLP model, and wait for the semantic integrity detection to be performed on the above i+1th streaming speech recognition result; when the number of executions of the above semantic integrity detection is greater than the fourth threshold, lower the above third threshold according to a preset strategy to perform the semantic integrity detection again based on the adjusted third threshold; wherein the adjusted third threshold is greater than the fifth threshold.

[0160] In an exemplary embodiment, based on the above scheme, the above semantic integrity detection module is specifically used to: input the above i-th streaming speech recognition result into a trained large language model to perform semantic integrity analysis on the above i-th streaming speech recognition result through the above large language model.

[0161] In an exemplary embodiment, based on the above scheme, the above device also includes: an execution module; wherein the above execution module is used to: execute the above i-th instruction during the process of the above instruction extraction module identifying the subsequent instruction of the above i-th instruction; and / or, during the process of the above instruction extraction module identifying the subsequent instruction of the above i-th instruction, generate response information corresponding to the above-mentioned audio to be processed according to the above i-th instruction, and output the above-mentioned response information.

[0162] It should be understood that if Fig.10 The embodiment of the streaming recognition device for voice commands shown in the figure may correspond to the embodiment of the streaming recognition method for voice commands described above, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, by Fig.10 The information interaction between the modules in the streaming recognition device for voice commands shown in FIG. 1 can execute the embodiment of the streaming recognition method for voice commands described above, by Fig.10 The information interaction between the various modules in the streaming recognition device for voice commands shown can execute the embodiments of the streaming recognition method for voice commands described above, and the aforementioned and other operations and / or functions of the various modules in the device The method embodiments corresponding to the functions are not repeated here for the sake of brevity.

[0163] The operation and maintenance related devices of the software agent of the embodiment of the present application are described above from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or a combination of hardware and software modules in the decoding processor to execute. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0164] An embodiment of the present application also provides an electronic device.

[0165] Fig.11 1 is a schematic block diagram of an electronic device 1100 provided in an embodiment of the present application. As mentioned above, the operation and maintenance related devices of the software agent can be deployed in Fig.11 The electronic device shown can be used to execute the above-mentioned streaming recognition method of voice commands.

[0166] like Fig.11 As shown, the electronic device 1100 may include:

[0167] The memory 1110 and the processor 1120, the memory 1110 is used to store the computer program 1130 and transmit the program code 1130 to the processor 1120. In other words, the processor 1120 can call and run the computer program 1130 from the memory 1110 to implement the method in the embodiment of the present application.

[0168] For example, the processor 1120 may be used to execute the steps in the above-mentioned method for streaming recognition of voice instructions according to the instructions in the computer program 1130, or to execute the steps in the above-mentioned method for streaming recognition of voice instructions.

[0169] In some embodiments of the present application, the processor 1120 may include but is not limited to:

[0170] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0171] In some embodiments of the present application, the memory 1110 includes but is not limited to:

[0172] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0173] In some embodiments of the present application, the computer program 1130 may be divided into one or more modules, which are stored in the memory 1110 and executed by the processor 1120 to complete the method for streaming recognition of voice instructions provided by the present application, or complete the steps in the method for streaming recognition of voice instructions. The one or more modules may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program 1130 in the electronic device.

[0174] like Fig.11As shown, the electronic device 1100 may further include:

[0175] The transceiver 1140 may be connected to the processor 1120 or the memory 1110 .

[0176] The processor 1120 may control the transceiver 1140 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 1140 may include a transmitter and a receiver. The transceiver 1140 may further include an antenna, and the number of antennas may be one or more.

[0177] It should be understood that the various components in the electronic device 1100 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0178] According to one aspect of the present application, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0179] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of the above method embodiment.

[0180] In other words, when software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instruction is loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integration. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a digital video disc (Digital Video Disc, DVD)), or a semiconductor medium (for example, a solid-state drive (Solid State Disk, SSD)), etc.

[0181] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0182] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0183] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0184] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for stream recognition of voice commands, characterized in that: The method comprises: Performing voice activity detection on the audio to be processed to determine a voice activity sub-segment in the audio to be processed, wherein the voice activity sub-segment is the audio between a start state and a quasi-end state obtained sequentially through detection; In the process of performing speech recognition on the current i-th voice-active subparagraph, in response to detecting a quasi-end state, obtaining an i-th streaming speech recognition result corresponding to the i-th voice-active subparagraph, wherein the value of i is a positive integer not greater than the total amount of voice-active subparagraphs in the audio to be processed; The i-th streaming speech recognition result is input into a natural language processing (NLP) model, wherein the NLP model outputs an i-th instruction corresponding to the i-th voice activity sub-paragraph.

2. The method according to claim 1, characterized in that The performing voice activity detection on the audio to be processed comprises: Input the audio to be processed into a voice activity detection (VAD) processing module, and output a start state when the VAD processing module detects that the characteristics of the audio signal meet a first preset condition; and output a quasi-end state when the VAD processing module detects that the characteristics of the audio signal do not meet the first preset condition and last for a first preset time. Wherein, the first preset duration is less than a first threshold; The characteristics of the audio signal include at least one of the following: energy level, spectrum characteristics, and change characteristics of the audio signal in the time domain.

3. The method according to claim 1, characterized in that In the process of performing speech recognition on the current i-th voice activity sub-paragraph, in response to detecting a quasi-end state, obtaining an i-th streaming speech recognition result corresponding to the i-th voice activity sub-paragraph, including: In the process of performing speech recognition on the current i-th speech activity subparagraph, before the quasi-end state is detected, the subset of speech recognition results that have been generated and not input into the NLP model is continuously compared with the second preset condition; If the comparison result is that the speech recognition result subset does not satisfy the second preset condition, then in response to detecting a quasi-end state, an i-th streaming speech recognition result corresponding to the i-th voice activity sub-paragraph is obtained.

4. The method according to claim 3, characterized in that The obtaining of the i-th streaming speech recognition result corresponding to the i-th voice activity subparagraph includes: Acquire the speech recognition result of the i-th speech activity sub-paragraph to obtain the speech information of the current sub-paragraph; Obtain all speech recognition results that have been input into the NLP model in the audio to be processed, and obtain contextual speech information of the current sub-paragraph; The current sub-paragraph speech information and the context speech information of the current sub-paragraph are combined to obtain the i-th streaming speech recognition result.

5. The method according to claim 3, characterized in that: The method further comprises: If the comparison result is that the speech recognition result subset satisfies the second preset condition, obtaining the i-th streaming speech recognition result corresponding to the speech recognition result subset; Among them, after the i-th streaming speech recognition result is input into the natural language processing NLP model, the NLP model outputs an instruction in the instruction set corresponding to the i-th voice activity sub-paragraph.

6. The method according to claim 5, characterized in that The obtaining of the i-th streaming speech recognition result corresponding to the speech recognition result subset includes: Acquire the current speech recognition result of the i-th voice activity subparagraph, and obtain the current subset speech information in the i-th voice activity subparagraph; Obtain all speech recognition results in the audio to be processed that have been input into the NLP model to obtain contextual speech information of the current subset; The current subset speech information and the context speech information of the current subset are combined to obtain the i-th streaming speech recognition result.

7. The method according to any one of claims 3 to 6, characterized in that The type of the speech recognition result is a text string, and the second preset condition includes that the length of the text string is greater than a second threshold.

8. The method according to any one of claims 1 to 6, characterized in that Before inputting the i-th streaming speech recognition result into a natural language processing (NLP) model, the method further includes: Performing a semantic integrity check on the i-th streaming speech recognition result to determine whether its semantic integrity is greater than a third threshold; Wherein, when the semantic completeness is greater than the third threshold, the i-th streaming speech recognition result is input into a natural language processing (NLP) model.

9. The method according to claim 8, characterized in that The method further comprises: When the semantic integrity is less than or equal to the third threshold, suspend inputting information to the NLP model and wait for semantic integrity detection to be performed on the i+1th streaming speech recognition result; When the number of times that the semantic integrity fails to meet the standard continuously reaches a fourth threshold, lowering the third threshold according to a preset strategy, so as to perform the semantic integrity check again based on the adjusted third threshold; Among them, the adjusted third threshold is greater than the fifth threshold.

10. The method according to claim 8, characterized in that The performing semantic integrity detection on the i-th streaming speech recognition result comprises: The i-th streaming speech recognition result is input into a trained large language model so as to perform semantic integrity analysis on the i-th streaming speech recognition result through the large language model.

11. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: In the process of identifying the subsequent instruction of the i-th instruction, executing the i-th instruction; and / or, In the process of identifying the subsequent instruction of the i-th instruction, response information corresponding to the audio to be processed is generated according to the i-th instruction, and the response information is output.

12. A streaming recognition device for voice commands, characterized in that: The device comprises: A voice detection module, used for performing voice activity detection on the audio to be processed to determine a voice activity sub-paragraph in the audio to be processed, wherein the voice activity sub-paragraph is the audio between a start state and a quasi-end state obtained in sequence through detection; A streaming result acquisition module, used for acquiring an i-th streaming speech recognition result corresponding to the i-th speech activity subparagraph in response to detecting a quasi-end state during speech recognition of the current i-th speech activity subparagraph, wherein the value of i is a positive integer not greater than the total amount of the speech activity subparagraphs in the audio to be processed; An instruction extraction module is used to input the i-th streaming speech recognition result into a natural language processing (NLP) model, wherein the NLP model outputs the i-th instruction corresponding to the i-th voice activity sub-paragraph.

13. A computer-readable storage medium, characterized in that: For storing computer programs; The computer program enables a computer to execute the streaming recognition method of voice instructions as described in any one of claims 1 to 11.

14. An electronic device, wherein: including a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program to implement the streaming recognition method of voice instructions as described in any one of claims 1 to 11.

Images