Voice interaction method, voice interaction device, server, and storage medium

Abstract

The application discloses a voice interaction method, a voice interaction device, a server and a storage medium. The voice interaction method comprises the following steps: performing voice recognition on a received user voice request of a vehicle cabin to obtain an identified text; in the case that the identified text comprises a wake-up word and the wake-up word is used as a salutation in the identified text, issuing a wake-up instruction to the vehicle so that the vehicle wakes up a voice assistant and a user to perform voice interaction according to the wake-up instruction. Thus, when the user communicates in the vehicle cabin, even if the same or similar instruction description as the wake-up word appears in the communication content, the voice interaction method in the application can accurately identify whether the user's voice request is a salutation to the voice assistant, thereby avoiding false wake-up, enabling the user to communicate naturally and smoothly, i.e. without deliberately avoiding the content related to the wake-up word in the communication process, and improving the user experience.

Description

Technical Field

[0001] This invention relates to the field of voice interaction technology, and in particular to a voice interaction method, a voice interaction device, a server, and a storage medium. Background Technology

[0002] Currently, in-vehicle voice assistants typically require activation before commands can be issued, which is inconvenient. Wake-up-free technology aims to solve this problem, allowing users to issue commands directly to the voice assistant without waking it up, thus improving the user experience. Wake-up-free technology requires determining whether a voice message is a command to the in-vehicle assistant. In practice, users may utter valid commands during in-car conversations, but these commands are not directed to the assistant, easily leading to false wake-ups. Furthermore, the ASR (Automatic Speech Recognition) module may misidentify some voice messages as valid commands, also causing false wake-ups. False wake-ups result in a poor user experience and require improvement. Summary of the Invention

[0003] This invention aims to at least solve one of the technical problems existing in the prior art. Therefore, one object of this invention is to provide a voice interaction method that can recognize voice requests within a vehicle cabin to avoid false wake-ups and improve user experience.

[0004] The voice interaction method according to the present invention includes: performing voice recognition on a received user voice request from a vehicle cabin to obtain recognized text; and, if it is determined that the recognized text includes a wake-up word and the wake-up word is used as a form of address in the recognized text, issuing a wake-up command to the vehicle so that the vehicle can wake up the voice assistant and the user for voice interaction according to the wake-up command.

[0005] Therefore, when users communicate in the vehicle cabin, even if the content of the conversation contains instructions that are the same as or similar to the wake word, the voice interaction method in this invention can accurately identify whether the user's voice request is a name for the voice assistant, thereby avoiding false wake-ups and allowing users to have natural and smooth conversations. In other words, during the conversation, there is no need to deliberately avoid content related to the wake word, thus improving the user experience.

[0006] Determining whether the wake-up word is used as a form of address in the recognized text includes: determining the matching results of the recognized text with multiple preset rules; and, if the recognized text matches a target rule among the multiple preset rules, determining whether the wake-up word is used as a form of address in the recognized text based on the matching results. Thus, by quickly matching the content corresponding to the recognized text using preset rules, the wake-up efficiency of the voice assistant is improved.

[0007] Furthermore, determining whether the wake-up word is used as a form of address in the identified text further includes: when the identified text does not match any of the multiple preset rules, determining whether the wake-up word is used as a form of address in the identified text based on the positional encoding features and part-of-speech encoding features of the wake-up word in the identified text. Thus, combining positional encoding features and part-of-speech encoding features can quickly and accurately identify whether text is a form of address, improving the wake-up efficiency of the voice assistant.

[0008] Furthermore, after determining whether the wake-up word is used as a form of address in the identified text based on its positional encoding features and part-of-speech encoding features, the method further includes: if the wake-up word is determined not to be used as a form of address in the identified text based on its positional encoding features and part-of-speech encoding features, determining the confusion level of the identified text; and if the confusion level is greater than a target confusion level, determining that the wake-up word is not used as a form of address in the identified text. Thus, the identified text can be analyzed in detail based on multiple judgment conditions, thereby accurately determining whether the identified text is a form of address, ensuring timely and accurate wake-up of the voice assistant.

[0009] Furthermore, after determining the confusion level of the identified text, the method further includes: if the confusion level is not greater than a target confusion level, determining the keyword weights in the identified text; wherein the keyword weights are used to characterize the proportion of target words in the word segmentation of the identified text; if the keyword weights are greater than the target weights, determining that the wake-up word is a form of address in the identified text. Thus, by further combining keyword weights, the identified text can be analyzed in detail, thereby accurately determining that the identified text is a form of address, ensuring that the voice assistant is woken up promptly and accurately.

[0010] Furthermore, determining the keyword weights in the identified text includes: identifying keywords that intersect the subwords of the identified text with the core word dictionary; dividing the number of keywords by the sum of the number of subwords and the number of keywords to obtain a preliminary proportion; and normalizing the preliminary proportion to obtain the keyword weights. Thus, by specifically combining keyword weights, the identified text is analyzed in detail, thereby accurately determining that the identified text is a form of address, ensuring timely and accurate activation of the voice assistant.

[0011] After determining that the identified text includes a wake-up word, the method further includes: if it is determined that the wake-up word is not used as a form of address in the identified text, not issuing a command to instruct the voice assistant to be woken up. Therefore, by ending the judgment process promptly when the voice assistant does not need to be woken up, the judgment time of the wake-up process can be shortened, and the judgment efficiency can be improved.

[0012] Determining that the identified text includes a wake word involves: extracting candidate words from the identified text using sequence labeling; and identifying the wake word from the candidate words using pinyin comparison. This allows for accurate identification of the wake word, ensuring timely activation of the voice assistant.

[0013] The present invention also proposes a voice interaction device, comprising: a text recognition module for performing voice recognition on a received user voice request from a vehicle cabin to obtain recognized text; and a sending module for issuing a wake-up command to the vehicle when it is determined that the recognized text includes a wake-up word and the wake-up word is used as a form of address in the recognized text, so that the vehicle can wake up the voice assistant and the user for voice interaction according to the wake-up command.

[0014] Therefore, by setting up this voice interaction device, when users communicate in the vehicle cabin, even if the content of the conversation contains instructions that are the same as or similar to the wake word, the voice interaction method of this invention can accurately identify whether the user's voice request is a name for the voice assistant, thereby avoiding false wake-ups. This allows users to have a natural and smooth conversation, that is, during the conversation, there is no need to deliberately avoid content related to the wake word, thus improving the user experience.

[0015] This invention also proposes a server comprising a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements any of the methods described above. This satisfies the wake-up requirements of the voice assistant and ensures accurate wake-up of the voice assistant.

[0016] This invention also proposes a non-volatile computer-readable storage medium for a computer program, characterized in that, when the computer program is executed by one or more processors, it implements any of the methods described above. This satisfies the wake-up requirements of a voice assistant and ensures accurate wake-up of the voice assistant.

[0017] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0018] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the voice interaction method in conjunction with the following drawings, wherein:

[0019] Figure 1 This is a diagram illustrating the main framework of the voice interaction method according to the present invention;

[0020] Figure 2 This is a schematic diagram of a classification model for the voice interaction method according to the present invention;

[0021] Figure 3 This is a schematic diagram of the main steps of the voice interaction method according to the present invention;

[0022] Figure 4 This is a schematic diagram illustrating the application of the voice interaction method according to the present invention;

[0023] Figure 5 This is one of the flowcharts illustrating the voice interaction method according to the present invention;

[0024] Figure 6 This is a second flowchart illustrating the voice interaction method according to the present invention;

[0025] Figure 7 This is the third flowchart illustrating the voice interaction method according to the present invention;

[0026] Figure 8 This is the fourth flowchart illustrating the voice interaction method according to the present invention;

[0027] Figure 9 This is the fifth flowchart illustrating the voice interaction method according to the present invention;

[0028] Figure 10 This is a schematic diagram illustrating the application of the voice interaction method according to the present invention;

[0029] Figure 11 This is one of the flowcharts illustrating the voice interaction method according to the present invention;

[0030] Figure 12 This is a second flowchart illustrating the voice interaction method according to the present invention;

[0031] Figure 13 This is the third flowchart illustrating the voice interaction method according to the present invention;

[0032] Figure 14 This is the fourth flowchart illustrating the voice interaction method according to the present invention;

[0033] Figure 15 This is the fifth flowchart illustrating the voice interaction method according to the present invention;

[0034] Figure 16 This is a schematic diagram illustrating the application of the voice interaction method according to the present invention;

[0035] Figure 17 This is one of the flowcharts illustrating the voice interaction method according to the present invention;

[0036] Figure 18 This is a second flowchart illustrating the voice interaction method according to the present invention;

[0037] Figure 19This is the third flowchart illustrating the voice interaction method according to the present invention;

[0038] Figure 20 This is the fourth flowchart illustrating the voice interaction method according to the present invention;

[0039] Figure 21 This is the fifth flowchart illustrating the voice interaction method according to the present invention;

[0040] Figure 22 This is a schematic diagram of a voice interaction device according to the present invention;

[0041] Figure 23 This is a schematic diagram of a server according to the present invention. Detailed Implementation

[0042] The voice interaction method of the present invention is described in detail below. Examples of the voice interaction method are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The voice interaction method described below with reference to the accompanying drawings is exemplary and is only used to explain the present invention, and should not be construed as limiting the present invention.

[0043] This invention proposes a voice interaction method, such as... Figures 1-4 As shown, when a passenger in the vehicle cabin makes a voice output: after the user's voice request is converted into recognized text, the text and its accompanying address are received into the rule engine, where it undergoes initial screening. The rule engine adapts the prefixes and some action words of the user's query and constructs the rule engine using a trie to reduce perturbation to the model. The input to the rule engine is "text query (ASR output result + address)," and the output is the query result matched by the rule engine, which is used for decision fusion for final judgment. Further, text queries not matched by the rule engine are input into the classification model, where they are judged and analyzed, and the output is the probability of a single query corresponding to a category, which is used for decision fusion for final judgment. Further still, the input query and the classification model result are sent to the decision engine, which accepts the result based on different situations and outputs a decision on whether the query should be spoken to the user. Thus, after the user outputs a voice request, the voice content is converted into recognized text, which can be accurately recognized by the rule engine, classification model, and decision engine to ensure accurate wake-up of the voice assistant and avoid false wake-ups.

[0044] Please see Figure 4 This describes the execution method of the voice interaction method of the present invention.

[0045] The voice interaction method proposed in this invention, such as Figure 5As shown, it includes the following steps:

[0046] A100: Perform speech recognition on the received user voice request from the vehicle cabin to obtain the recognized text. Specifically, a microphone or other voice acquisition component may be installed in the vehicle cabin to acquire user voice requests from within the cabin. These user voice requests may originate from various audio zones within the vehicle cabin, including but not limited to the driver and front passenger zones, the front passenger seat zone, the zone on the left side of the second row behind the driver, the zone in the middle of the second row behind the driver, the zone on the right side of the second row behind the driver, the zone on the left side of the third row behind the driver, and the zone on the right side of the third row behind the driver. Further, after receiving the user voice request, perform recognition to obtain the corresponding text.

[0047] Specifically, the speech recognition method can employ ASR technology (a technology that converts human speech into text), so that when a user makes a speech output from any location in the vehicle cabin, the recognized text can be obtained based on the user's voice request.

[0048] A200: If it is determined that the recognized text includes a wake-up word and the wake-up word is used as a form of address in the recognized text, a wake-up command is issued to the vehicle so that the vehicle can wake up the voice assistant and the user for voice interaction according to the wake-up command.

[0049] The wake word can be a system default noun or a manually set noun, and it refers to the voice assistant. This allows users to use the wake word when requesting the voice assistant to perform a function, such as "Xiao P," "Xiao Y," or other nouns. This invention will use "Xiao P" as an example for further explanation.

[0050] Furthermore, when the wake word appears in the user's voice request and is used as a form of address, the wake word can refer to the voice assistant. For example, if the wake word is "Xiao P" and the user's voice request is "Xiao P, please open the car window for me," then "Xiao P" is used as a form of address and refers to the voice assistant in the voice request. This allows the vehicle to wake up the voice assistant according to the wake-up command, enabling the voice assistant to communicate with the user and thus facilitate the voice assistant to perform functions or make adjustments according to the user's voice commands.

[0051] It should be noted that in actual use, words identical to the wake-up word may appear in the user's communication in different contexts. That is, when the wake-up word is not used as an appellation, the wake-up word in the user's voice request does not refer to the voice assistant. For example, a user in a vehicle cockpit is engaged in daily communication, such as saying "Let's go to Little P Park" or "Open Little P's dressing up". Although the voice request includes the wake-up word, the wake-up word is not used as an appellation. That is, the user who issues this voice request does not intend to wake up the voice assistant. Therefore, by setting the voice interaction method in the present invention, when it is determined that the recognized text includes the wake-up word and the wake-up word serves as an appellation in the recognized text, the wake-up of the voice assistant is achieved, thereby avoiding the situation of mis-waking up the voice assistant.

[0052] Thus, when the user is communicating in the vehicle cockpit, even if there are command descriptions identical or similar to the wake-up word in the communication content, the voice interaction method in the present invention can accurately identify whether the user's voice request is an appellation for the voice assistant, thereby avoiding mis-waking up, enabling the user to communicate naturally and smoothly. That is, during the conversation, there is no need to deliberately avoid content related to the wake-up word, enhancing the user experience.

[0053] As Figure 6 shown, determining that the recognized text includes the wake-up word includes:

[0054] A210. Extract candidate words from the recognized text by means of sequence labeling. It should be noted that this step extracts words that may be appellations from the text, which is not a simple text matching because ASR may mis-recognize an appellation as other words with similar pronunciations. For example, "Little P" may be mis-recognized as "Little Pi", "Little Pei", "Little Pi" etc. In the present invention, the method of sequence labeling is adopted to extract candidate appellation words, effectively solving the problem of ASR errors.

[0055] A220. Determine the wake-up word from the candidate words by means of pinyin comparison. That is, the candidate appellation words extracted in the first step in A220 may not be valid appellation words, so further screening is required. A method based on pinyin can be adopted to screen for "Little P" or words pronounced as "xiao pi".

[0056] Furthermore, after determining the wake-up word, further disambiguate the wake-up word. That is, the appellation word in the recognized text may be an appellation, or it may be a mention or a proper noun, so it needs to be distinguished, that is, disambiguated. Specifically, disambiguation can be performed based on the following steps.

[0057] Furthermore, as Figure 7 shown, to determine whether the wake-up word serves as an appellation in the recognized text, preliminary screening can be first performed based on a rule engine, and the execution examples of the rule engine are A230, A231 including:

[0058] A230. Determine the matching results between the recognized text and multiple preset rules. In other words, when designing voice interaction, multiple preset rules can be pre-defined, and these preset rules can filter and identify the recognized text. These preset rules can be those default to the vehicle or flexibly set by the user according to their needs, such as selecting specific sentences as the matching content for the preset rules. Specifically, they can be set according to the actual functional needs of the vehicle, such as "Xiao P, please turn on the air conditioning," "Xiao P, please open the driver's side window," or other types or methods of preset rules.

[0059] A231. When the recognized text matches the target rules among multiple preset rules, determine whether the wake word is used as a form of address in the recognized text based on the matching results. That is, after recognizing the text, perform matching analysis on the content of the recognized text to determine whether the wake word is the user's address to the voice assistant, i.e., whether the user is calling the voice assistant. Among the preset rules, prefixes and some action words of the user can be adapted, and preset rules are constructed using a trie to reduce perturbation to the model. Specifically, the input is: text query (ASR output result + address), and the output is: the query result matched by the preset rules, used for decision fusion for final judgment.

[0060] Therefore, when the identified text matches the target rule among multiple preset rules, the voice request command can be output, thereby waking up the voice assistant so that the voice assistant can perform the corresponding function operation based on the voice request. Specifically, phrases like "Xiao P, speak louder" or "Xiao P, lower your volume" conform to the sentence structure "Xiao P's XX," and we classify them as negative examples. However, phrases like "Xiao P's lab" and "Xiao P changing clothes" are related to business scenarios and are not easily distinguishable. Introducing such data would significantly disturb the training model; this part is handled by rules, and the corresponding result is accepted if a rule is matched.

[0061] Therefore, after the rule engine is executed, the classification module is filtered in conjunction with step A232.

[0062] Furthermore, determining whether the wake word functions as a form of address in the identified text also includes:

[0063] A232. When the recognized text does not match any of the preset rules among multiple preset rules, determine whether the wake word is used as a form of address in the recognized text based on the positional encoding features and part-of-speech encoding features of the wake word in the recognized text.

[0064] In other words, during the actual execution process, after the identified text is matched against multiple preset rules, if the target rules in these preset rules do not match the content of the identified text—meaning the identified text is not within the user's preset rules—then the content of the identified text can be further analyzed using the positional encoding features and part-of-speech encoding features of the wake word in the identified text to determine whether the wake word is a form of address. Specifically, the positional encoding features and part-of-speech encoding features can be set and implemented based on the POS Embedding function (part-of-speech embedding) and the Salutation Embedding function (address embedding), respectively.

[0065] At this point, the wake word can be determined based on its positional encoding features and part-of-speech encoding features. Positional encoding features refer to the location of the wake word within the recognized text. This can be determined by matching the text according to the usual sentence order, such as the wake word appearing at the beginning, middle, or end of a sentence. Wake words at the beginning and end of a sentence tend to be more like titles or appellations, such as "Little P, turn the volume up a bit" or "Please turn the volume up a bit, Little P." In this type of recognized text, "Little P" is at the beginning and end of the sentence, and both are titles. Alternatively, in the sentence "Enter Little P's studio," "Little P" is in the middle of the sentence and is not a title or appellation.

[0066] Furthermore, part-of-speech encoding features can identify the content of the recognized text to determine whether it is a combination of a form of address and an action. Specifically, for example, if a user outputs a voice request like "Xiao P, turn the volume up a bit" or "Xiao P, turn the volume down a bit," "Xiao P" can be identified as a form of address, followed by an action. However, in voice requests like "Xiao P Lab" or "Xiao P Playground," "Xiao P" is the name of the lab and playground, not a form of address, and the following phrases are not explicit actions. Based on this, it can be determined whether the user's voice command is waking up the voice assistant. Additionally, wake-up words in the recognized text can also be determined based on a Masked Language Model, i.e., masked encoding features can also be selected, both of which can be used to identify forms of address.

[0067] The process involves first encoding the identified text to obtain mask encoding features, position encoding features, and part-of-speech encoding features. These features are then fused to obtain an embedding matrix. The embedding matrix is ​​then subjected to self-attention transformation to obtain a similarity matrix. Logistic regression is performed on the similarity matrix to obtain the classification confidence score. If the classification confidence score is greater than the target confidence score, the wake word is determined to be a form of address in the identified text.

[0068] Alternatively, the identified text can be input into the encoding layer of a classification model to obtain the masked encoding features, the positional encoding features of the wake word, and the part-of-speech encoding features of the identified text. The encoding layer includes a masked speech model, a title encoding function for encoding the positional relationship of the wake word, and a part-of-speech encoding function for encoding verbs and prepositions in the identified text. The masked encoding features, positional encoding features, and part-of-speech encoding features are then input into the embedding layer of the classification model to obtain the embedding matrix output by the embedding layer. The embedding matrix is ​​then input into the self-attention layer of the classification model to obtain the similarity matrix output by the self-attention layer. The similarity matrix is ​​then input into the logistic regression layer of the classification model to obtain the classification confidence score output by the logistic regression layer. Based on the classification confidence score, it is determined whether the wake word functions as a title in the identified text.

[0069] Specifically, when performing confidence level determination using a classification model, a threshold can be set first. If the probability (score) given by the model is greater than the set threshold, the judgment result is output; otherwise, the process proceeds to the next step. For example, "Little P opens the car window" will give a high-confidence positive example score, and "Little P, you're great" will give a high-confidence negative example score.

[0070] After executing the classification model based on step A232, the decision fusion engine can be further executed in conjunction with steps A233, A234, A235, and A236.

[0071] Furthermore, such as Figure 8 As shown, after determining whether the wake word functions as a form of address in the recognized text based on its positional encoding features and part-of-speech encoding features, the method further includes:

[0072] A233. If, based on positional encoding features and part-of-speech encoding features, it is determined that the wake word is not used as a form of address in the identified text, then the confusion level of the identified text is determined. In other words, after identifying the wake word as not being used as a form of address based on the classification model, the confusion level of the identified text is determined to further determine whether the wake word is a form of address, thereby improving the accuracy of the output results.

[0073] The confusion level of the identified text can be determined based on a language model. For example, a 3-gram and a 4-gram language model can be prepared in advance, and the identified text can be input into the language model to calculate the confusion level.

[0074] A234. When the confusion level is greater than the target confusion level, determine that the wake word is not used as a form of address in the recognized text. For example, phrases like "Xiao P's voice is louder" or "Xiao P's volume is lower" fit the sentence structure "Xiao P's XX," and we classify them as negative examples. Other examples include phrases like "Xiao P's lab" and "Xiao P changing clothes," which are related to business scenarios and are not easily distinguishable. Introducing such data would significantly perturb the training model; these parts are handled by rules, and results that match the rules are accepted.

[0075] Furthermore, such as Figure 9 As shown, after determining the obfuscation level of the identified text, the method further includes:

[0076] A235. When the confusion level is not greater than the target confusion level, determine the keyword weights in the identified text; whereby the keyword weights represent the proportion of the target word in the segmented text. Therefore, when it is impossible to determine whether the wake-up word is a form of address based on the confusion level, the keyword weights in the identified text can be determined, and based on these keyword weights, it can be further determined whether the wake-up word functions as a form of address in the identified text.

[0077] A236. When the keyword weight is greater than the target weight, the wake word is determined to be the form of address in the recognized text. Therefore, the wake word can be determined as the form of address based on the keyword weight, thus activating the voice assistant.

[0078] Therefore, in the specific implementation, the confusion degree of the wake word in the recognized text can be calculated in advance based on the 3-gram and 4-gram language models. If the calculated weighted confusion degree is greater than the target confusion degree, a negative example result is output; otherwise, the judgment is made in combination with the keyword weight.

[0079] Furthermore, the keyword weights in the identified text are determined, including:

[0080] A2361, identify the keywords that intersect the sub-words and core word dictionaries of the identified text; divide the number of keywords by the sum of the number of sub-words and the number of keywords to obtain the preliminary proportion; normalize the preliminary proportion to obtain the keyword weight.

[0081] In other words, when determining keyword weights, a core word dictionary is prepared in advance. The proportion of each keyword in the sum of the number of sub-words and the total number of keywords is calculated to obtain a preliminary proportion of the keywords. This preliminary proportion is then normalized to obtain the keyword weights. For example, the core word dictionary includes AA, AB, CD, DE...; query sub-words: AA, AB, DD; the intersection of the core word dictionary and the sub-words in the query is X = 2(AA AB).

[0082] Therefore, the weight of the keywords can be obtained. When the keyword weight is greater than the target weight, the wake word is determined as a form of address in the recognized text, thus waking up the voice assistant. For example, "Xiao P, please navigate me to Huolu Mountain Forest Park." Because the model training data cannot cover all open slots, this module improves the recall of wake words for open slots.

[0083] Therefore, rules tightly coupled with business logic were incorporated into the decision fusion stage, avoiding the introduction of too much difficult-to-judge data into the model and reducing the model's decision-making pressure. In cases of low model confidence, a pre-prepared language model and core word dictionary fusion discrimination process was added, increasing overall recall.

[0084] After determining that the identified text includes a wake word, the method further includes:

[0085] If it is determined that the wake word is not used as a form of address in the recognized text, no instruction to wake up the voice assistant will be issued. In other words, if the wake word in the user's voice request is not used as a form of address to wake up the voice assistant, the voice assistant does not need to be woken up, and therefore no instruction to wake up the voice assistant will be issued.

[0086] This invention proposes other voice interaction methods; please refer to [link / reference]. Figure 10 Furthermore, this voice interaction method shares many similarities with the voice interaction method proposed in the first aspect above in terms of specific execution steps. The difference lies in the wake-free step performed when the voice command is a simple address, and in the determination of valid commands in NLU (Natural Language Processing), which further improves the accuracy of wake-free operation.

[0087] like Figure 11 As shown, A110 performs speech recognition on the received user voice request from the vehicle cabin to obtain the recognized text. Specifically, a microphone or other voice acquisition components are installed in the vehicle cabin to acquire user voice requests from within the cabin. These user voice requests can originate from the driver's seat, front passenger seat, or rear seats. Further, after receiving the user voice request, the content of the speech is recognized to identify the corresponding recognized text for that round of the voice request.

[0088] Specifically, the speech recognition method can employ ASR technology (a technology that converts human speech into text), so that when a user makes a speech output from any location in the vehicle cabin, the recognized text can be obtained based on the user's voice request.

[0089] A240. If it is determined that the text to be recognized in this round includes a form of address and is a pure form of address, obtain the text to be recognized in the previous round; wherein, the text to be recognized in the previous round is the text to be recognized corresponding to the user's voice request in the previous round, and the text to be recognized in the previous round does not include a form of address.

[0090] In other words, after the user issues a voice command, if the address in the recognized text is a pure address, then the recognized text contains only the address and no other content, such as "Little P". Once this condition is determined, the previous round of recognized text can be retrieved. That is, after each round of retrieving the recognized text, the recognized text can be temporarily stored so that after further retrieving and recognizing the recognized text, the previous round of recognized text can be retrieved based on the recognition result. Then, the previous round of recognized text can be judged to determine whether it does not contain an address.

[0091] A250: If the previously recognized text is determined to be a valid instruction, a wake-up command is issued to the vehicle so that the vehicle can activate the voice assistant and interact with the user via voice. That is, if it is further determined that the previously recognized text did not include a greeting, and the recognized text is a valid instruction, it can be concluded that the user's voice command aims to obtain assistance from the voice assistant.

[0092] For example, if the recognized text obtained in this round is "Xiao P", which is a pure form of address, and the previous round of recognized text is obtained, which is "Please open the driver's side window", the previous round of recognized text does not include any form of address, and it contains a clear action to be performed. Therefore, the recognized text is a valid instruction. At this time, the voice assistant can be activated to execute the request instruction corresponding to the previous round of recognized text to open the driver's side window, thus meeting the user's needs.

[0093] Therefore, when a user requests to wake up the voice assistant inside the car, even if the user first speaks the corresponding action and then utters the wake-up word for the voice assistant, the voice interaction method of this invention can still determine that the user has a request to wake up the voice assistant based on the content of the voice request, thus achieving accurate wake-up of the voice assistant. In other words, even if the user outputs the voice request in an unconventional order of performing the action first and then using the wake-up word, the voice assistant can still be woken up promptly and effectively, thereby meeting the user's wake-up needs in different types of application scenarios and improving the user experience.

[0094] like Figure 12 As shown, the method further includes the following steps before issuing a wake-up command to the vehicle:

[0095] A251. Determine that the time interval between the current round of identified text and the previous round of identified text is less than the target duration. That is, before issuing a wake-up command to the vehicle, the time interval between the two rounds of identified text can be judged and analyzed. If the time interval between the two rounds is less than the target duration, it can be determined that the identified text in both rounds was issued by the user based on the same need.

[0096] In practice, the target duration can be set to 3 seconds. This means that if the current round of recognized text contains a form of address within 3 seconds of the previous round's text being received, a dependency relationship is established between the two rounds of recognized text. In other words, if the time interval between two rounds of recognized text is too long, they may not be necessarily related. For example, if the user's previous recognized text was "Please open the driver's side window," and the current recognized text is a simple address like "Little P," but the time interval exceeds the target duration (e.g., 1 hour), then there is clearly no connection between them, and the voice assistant does not need to be activated.

[0097] Therefore, by setting the interval between the two rounds of text recognition, the voice assistant can be prevented from being woken up accidentally, thus improving the accuracy of wake-up.

[0098] like Figure 13 As shown, after obtaining the text to be recognized in this round, the method also includes:

[0099] A252. If it is determined that the text to be recognized in this round does not include a title, cache the text to be recognized in this round, the time information of the text to be recognized in this round, and the audio region of the user's voice request to be recognized in this round.

[0100] In this case, the identified text does not include a title or address, meaning that this identified text may serve as a valid instruction for the next round of identification. Therefore, this identified text can be temporarily cached. Correspondingly, the time information and corresponding vocal range of this identified text (excluding the title or address) are identified and cached so that the identified text, along with its corresponding time information and vocal range, can be applied in the next round.

[0101] It should be noted that the sound zone refers to the area of ​​each seat in the vehicle cabin, such as the driver's sound zone, the passenger's sound zone, and the rear seat sound zone.

[0102] Specifically, for example, the recognized text is "Please open the passenger window", and the time corresponding to the recognized text is 9:30:15, and the corresponding audio range is passenger.

[0103] like Figure 14 As shown, the method also includes:

[0104] A253. If it is determined that the time interval between the current round of recognized text and the previous round of recognized text is not less than the target duration, or if the previous round of recognized text is an invalid instruction, no instruction to wake up the voice assistant shall be issued to the vehicle.

[0105] In other words, no wake-up command will be issued to the vehicle if either of the following conditions is met: the time interval between the current and previous recognized texts is not less than the target duration, or the previous recognized text is an invalid command. Specifically, for example, if the target duration is set to 3 seconds, and the current recognized text obtained within 3 seconds of the previous recognized text contains a greeting, but the voice command in the previous recognized text is "Do you want to listen to music?", then the recognized text is clearly invalid and no wake-up operation is needed. Alternatively, if the target duration is set to 3 seconds, and the voice command in the previous recognized text is "Please open the passenger window", then the recognized text is clearly valid, but the time interval between the two recognized texts is greater than 3 seconds, such as up to 15 minutes, then no wake-up command is needed for the voice assistant.

[0106] Therefore, by setting the interval between the two rounds of text recognition and combining the content of the previous round of text recognition, the voice assistant can be further prevented from being woken up accidentally, thus improving the accuracy of wake-up.

[0107] It should be noted that the time interval between the two rounds of text recognition refers to the end time of the previous round of text recognition and the start time of the current round of text recognition. For example, if the time period of the previous round of text recognition is t1 to t2 and the time period of the current round of text recognition is t3 to t4, then the time interval between the two rounds of text recognition is t3-t2.

[0108] After determining that the text to be identified in this round includes salutations, the method also includes:

[0109] If it is determined that the text being recognized in this round is not a mere address and is a valid instruction, a wake-up command is issued to the vehicle. That is, when it is determined that the text being recognized in this round is a mere address, the command analysis and recognition text can be performed based on the rule engine, classification module, and decision fusion in the above-mentioned voice interaction method. The text contains an address and also includes the valid instruction requested by the customer, thereby clarifying that the customer has a clear need to summon the voice assistant and execute the corresponding function, thus ensuring accurate wake-up of the voice assistant.

[0110] like Figure 15 As shown, determining that the text identified in the previous round is a valid instruction includes:

[0111] A254. The intent of the previously identified text is determined to be within the list of valid intents, and the length of the previously identified text is greater than the target length. Furthermore, the previously identified text is determined to point to a specific object and method of operation. In other words, the conditions for determining the previously identified text as a valid instruction include that the length of the previously identified text is greater than the target length, and that there is a specific object and method of operation.

[0112] The text recognized in the previous round must be longer than the target length, such as more than 5 characters, or other target lengths can be set. The objects of operation can be in-vehicle functional components, such as windows, air conditioning, lights, and audio systems, and the operation methods can include opening, closing, raising, and lowering.

[0113] For example, in the text "Xiao P, please turn on the air conditioner", the length of the text being recognized is greater than the target length, the object of the operation is the air conditioner, and the operation method is to turn it on; or, in the text "Xiao P, please turn up the volume", the length of the text being recognized is greater than the target length, the object of the operation is the audio system, and the operation method is to turn up the volume.

[0114] Therefore, it can be determined that the user's output command is a valid command, which enables the voice assistant to be activated.

[0115] The following are some specific scenarios:

[0116] In scenario one: the instruction buffered in the previous voice zone 1 was "turn on the driver's air conditioning". The system did not recognize the address, so the instruction was not executed and was directly stored in the cache. Then, in the current voice zone 1, the instruction "Xiao P" was received. The system recognized the instruction as a pure address and retrieved the previous instruction from the same voice zone. The time interval between the previous instruction and the current instruction is 6.5-4.0=2.5s, which is less than 3s, so it is a valid cached instruction. Therefore, the wake-up judgment process is entered. After the NLU stage, the system finally judges the instruction as a valid instruction, so the wake-up is triggered. The system executes the instruction and clears the cache in voice zone 1.

[0117] In scenario two: In the previous round, when voice zone 1 received the instruction "turn on the driver's air conditioning", the system did not recognize the address, so the instruction was not executed and was directly stored in the cache; then in the current round, when voice zone 1 received the instruction "Xiao P", the system recognized the instruction as a pure address, so it retrieved the previous round's instruction from the same voice zone cache; the time interval between the previous round's instruction and the current round's instruction is 7.5-4.0=3.5s, which is greater than 3s, so it is not an effective cached instruction, so it does not enter the wake-up judgment process, and at the same time, the system cache of voice zone 1 is cleared.

[0118] In scenario three: In the previous round, when voice zone 1 received the instruction "Is the car window open?", the system did not recognize the address, so the instruction was not executed and was directly stored in the cache. Then, in the current round, when voice zone 1 received the instruction "Xiao P", the system recognized the instruction as a simple address and retrieved the previous round's instruction from the same voice zone cache. The time interval between the previous round's instruction and the current round's instruction is 6.5 - 4.0 = 3.5 seconds, which is less than 3 seconds, so it is a valid cached instruction. After NLU, the system finally determined that the previous round's instruction was an invalid instruction, so it did not trigger the wake-up and cleared the system cache for voice zone 1.

[0119] For example, in scenario four: In the previous round, when audio zone 1 received the instruction "Open the driver's side window," the system did not recognize the address, so the instruction was not executed and was directly stored in the cache. Then, in the current round, when audio zone 2 received the instruction "Xiao P," the system recognized the instruction as a simple address, so it retrieved the previous round's instruction from the cache in the same audio zone. Since there was no cached instruction in the current audio zone 2, the process ended. (The system ensures that the audio zone cache is independent and will not affect the cache of audio zone 1).

[0120] In scenario five: In the previous round, when audio zone 1 received the instruction "open the driver's side window," the system did not recognize the address, so the instruction was not executed and was directly stored in the cache. Then, in the current round, when audio zone 2 received the instruction "Xiao P," the system recognized the instruction as a pure address and retrieved the previous round's instruction from the same audio zone cache. Since there was no cached instruction in the current audio zone 2, the process ended. In the next round, when audio zone 1 received the instruction "Xiao P," the system recognized the instruction as a pure address and retrieved the previous round's instruction from the same audio zone cache. The time interval between the previous round's instruction and the current round's instruction was 6.5 - 4.0 = 2.5 seconds, which was less than 3 seconds, indicating that the cached instruction was valid. Therefore, the wake-up judgment process was initiated. After NLU final determination, the system determined that the instruction was valid and triggered the wake-up process. The system executed the instruction and cleared the cache in audio zone 1.

[0121] In Scenario 6: In the previous round, when audio zone 1 received the instruction "Turn on the driver's side air conditioning," the system did not recognize the address, so the instruction was not executed and was directly stored in the cache. Then, in the current round, audio zone 1 received the instruction "Xiao P." The system recognized the instruction as a simple address and retrieved the previous round's cached instruction from the same audio zone. The time interval between the previous and current instructions is 6.5 - 4.0 = 2.5 seconds, less than 3 seconds, indicating a valid cached instruction. Therefore, the system enters the wake-up judgment process. After the NLU stage, the system ultimately determines the instruction is valid, triggering the wake-up, executing the instruction, and clearing the audio zone 1 cache. In the next round, when audio zone 1 received the instruction "Xiao P close all windows," the system recognized the address and entered the address wake-up judgment process. After the NLU stage, the system ultimately determines the instruction is valid, triggering the address wake-up execution instruction and clearing the audio zone 1 cache.

[0122] Determine whether the wake word is used as a form of address in the text being identified in this round, including:

[0123] If the text identified in this round contains a wake word, determine the matching result between the text identified in this round and multiple preset rules; if the text identified in this round matches the target rule among the multiple preset rules, determine whether the wake word is used as a form of address in the text identified in this round based on the matching result.

[0124] In determining whether the wake-up word is used as a form of address in the current round of text recognition, the rules A230 and A231 mentioned above can be combined. Specifically, this involves A230 determining the matching results between the recognized text and multiple preset rules. In other words, when designing voice interaction, multiple preset rules can be pre-defined, and these preset rules can filter and identify the recognized text. These preset rules can be those that are built into the vehicle by default, or they can be flexibly set by the user according to their needs. For example, specific sentences can be selected as the matching content of the preset rules. Specifically, they can be set according to the actual functional needs of the vehicle, such as "Xiao P, please turn on the air conditioning" or "Xiao P, please open the driver's side window," or they can be other types or methods of preset rules.

[0125] When the recognized text matches the target rules among multiple preset rules, the system determines whether the wake word is used as a form of address in the recognized text based on the matching results. That is, after recognizing the text, the content of the recognized text is matched and analyzed to determine whether the wake word is the user's address to the voice assistant, i.e., whether the user is calling the voice assistant. The preset rules can be adapted to user prefixes and some action words, and are constructed using a trie to reduce perturbation to the model. Specifically, the input is a text query (the result of ASR output + the form of address), and the output is the query result matched by the preset rules, used for decision fusion for the final judgment.

[0126] Therefore, when the identified text matches the target rule among multiple preset rules, the voice request command can be output, thereby waking up the voice assistant so that the voice assistant can perform the corresponding function operation based on the voice request. Specifically, phrases like "Xiao P, speak louder" or "Xiao P, lower your volume" conform to the sentence structure "Xiao P's XX," and we classify them as negative examples. However, phrases like "Xiao P's lab" and "Xiao P changing clothes" are related to business scenarios and are not easily distinguishable. Introducing such data would significantly disturb the training model; this part is handled by rules, and the corresponding result is accepted if a rule is matched.

[0127] In addition, determining whether the wake word is used as a form of address in the text being identified in this round also includes:

[0128] If the identified text does not match any of the multiple preset rules, the system determines whether the wake-up word is used as a form of address in the identified text based on its positional encoding and part-of-speech encoding features. This step can be combined with step A232 in the classification model described above; the execution logic is the same. In other words, during the specific execution process, after the identified text is matched against multiple preset rules, if none of the corresponding target rules match the content of the identified text (meaning the identified text is not within the user's preset rules), then the content of the identified text can be further analyzed using the positional encoding and part-of-speech encoding features of the wake-up word to determine whether the wake-up word is a form of address. Specifically, the positional encoding features and part-of-speech encoding features can be set and implemented based on POS Embedding (function) and Salutation Embedding (address embedding), respectively.

[0129] At this point, the wake word can be determined based on its positional encoding features and part-of-speech encoding features. Positional encoding features refer to the location of the wake word within the recognized text. This can be determined by matching the text according to the usual sentence order, such as the wake word appearing at the beginning, middle, or end of a sentence. Wake words at the beginning and end of a sentence tend to be more like titles or appellations, such as "Little P, turn the volume up a bit" or "Please turn the volume up a bit, Little P." In this type of recognized text, "Little P" is at the beginning and end of the sentence, and both are titles. Alternatively, in the sentence "Enter Little P's studio," "Little P" is in the middle of the sentence and is not a title or appellation.

[0130] Furthermore, part-of-speech encoding features can identify the content of the recognized text to determine whether it is a combination of a form of address and an action. Specifically, for example, if a user outputs a voice request like "Xiao P, turn the volume up a bit" or "Xiao P, turn the volume down a bit," "Xiao P" can be identified as a form of address, followed by an action. However, in voice requests like "Xiao P Lab" or "Xiao P Playground," "Xiao P" is the name of the lab and playground, not a form of address, and the following phrases are not explicit actions. Based on this, it can be determined whether the user's voice command is waking up the voice assistant. Additionally, wake-up words in the recognized text can also be determined based on a Masked Language Model, i.e., masked encoding features can also be selected, both of which can be used to identify forms of address.

[0131] The process involves first encoding the identified text to obtain mask encoding features, position encoding features, and part-of-speech encoding features. These features are then fused to obtain an embedding matrix. The embedding matrix is ​​then subjected to self-attention transformation to obtain a similarity matrix. Logistic regression is performed on the similarity matrix to obtain the classification confidence score. If the classification confidence score is greater than the target confidence score, the wake word is determined to be a form of address in the identified text.

[0132] Alternatively, the identified text can be input into the encoding layer of a classification model to obtain the masked encoding features, the positional encoding features of the wake word, and the part-of-speech encoding features of the identified text. The encoding layer includes a masked speech model, a title encoding function for encoding the positional relationship of the wake word, and a part-of-speech encoding function for encoding verbs and prepositions in the identified text. The masked encoding features, positional encoding features, and part-of-speech encoding features are then input into the embedding layer of the classification model to obtain the embedding matrix output by the embedding layer. The embedding matrix is ​​then input into the self-attention layer of the classification model to obtain the similarity matrix output by the self-attention layer. The similarity matrix is ​​then input into the logistic regression layer of the classification model to obtain the classification confidence score output by the logistic regression layer. Based on the classification confidence score, it is determined whether the wake word functions as a title in the identified text.

[0133] Specifically, when performing confidence level determination using a classification model, a threshold can be set first. If the probability (score) given by the model is greater than the set threshold, the judgment result is output; otherwise, the process proceeds to the next step. For example, "Little P opens the car window" will give a high-confidence positive example score, and "Little P, you're great" will give a high-confidence negative example score.

[0134] Furthermore, after determining whether the wake-up word functions as a form of address in the current round of text recognition based on its positional and part-of-speech encoding features, the method further includes:

[0135] Specifically, after determining that the wake-up word is not used as a form of address in the current round of text recognition based on positional encoding features and part-of-speech encoding features, the confusion level of the current round of text recognition is determined. Specifically, after identifying the wake-up word as not being used as a form of address based on the classification model, the confusion level of the recognized text is determined to further determine whether the wake-up word is indeed a form of address, thereby improving the accuracy of the output results. The confusion level of the recognized text can be determined based on a language model, such as by preparing 3-gram and 4-gram language models in advance, inputting the recognized text into these models, and then calculating the confusion level. This step can refer to step A233 above.

[0136] If the confusion level is greater than the target confusion level, the wake word is determined not to be used as a form of address in the current round of text recognition. For example, phrases like "Xiao P's voice is louder" or "Xiao P's volume is lower" fit the sentence structure "Xiao P's XX," and are therefore classified as negative examples. Other examples include phrases like "Xiao P's lab" and "Xiao P changing clothes," which are related to business scenarios and are difficult to distinguish. Introducing such data would significantly perturb the training model. This part is handled by rules; if a rule is matched, the corresponding result is accepted. This step can be referenced from step A234 above.

[0137] If the confusion level is not greater than the target confusion level, determine the keyword weights in the current round of text recognition; whereby the keyword weights represent the proportion of the target words in the word segmentation of the current round of text recognition. Thus, when it is impossible to determine whether the wake-up word is a form of address based on the confusion level, the keyword weights of the recognition text can be determined, and based on the keyword weights, it can be further determined whether the wake-up word is used as a form of address in the recognition text. This step can refer to step A235 above.

[0138] If the keyword weight is greater than the target weight, the wake-up word is determined as the form of address in the current round of text recognition. Therefore, the wake-up word can be determined as the form of address based on the keyword weight, thus waking up the voice assistant. This step can be referred to step A236 above.

[0139] Therefore, in the specific implementation, the confusion degree of the wake word in the recognized text can be calculated in advance based on the 3-gram and 4-gram language models. If the calculated weighted confusion degree is greater than the target confusion degree, a negative example result is output; otherwise, the judgment is made in combination with the keyword weight.

[0140] Further, please refer to Figure 16 This invention also proposes other voice interaction methods. These methods share many similarities with the voice interaction methods proposed in the second aspect above in terms of specific execution steps, but differ in that they further improve the accuracy of the wake-up-free step in determining whether the currently recognized text matches a preset text in the whitelist. For example... Figure 17 As shown, the voice interaction method includes:

[0141] A110. Perform speech recognition on the received user voice request from the vehicle cabin to obtain the recognized text for that round. Specifically, a microphone or other voice acquisition components may be installed in the vehicle cabin to acquire user voice requests from the driver's seat, front passenger seat, or rear seats. Further, after receiving the user voice request, recognize the content of the speech to identify the corresponding recognized text for that round.

[0142] Specifically, the speech recognition method can employ ASR technology (a technology that converts human speech into text), so that when a user makes a speech output from any location in the vehicle cabin, the recognized text can be obtained based on the user's voice request.

[0143] A260. If it is determined that the text identified in this round is credible, does not contain a salutation, matches the preset text in the whitelist, and is a valid instruction, a wake-up command is issued to the vehicle so that the vehicle can wake up the voice assistant and interact with the user via voice according to the wake-up command.

[0144] The preset text in the whitelist is pre-set, such as the default setting in the vehicle system, or it can be subjectively set by the user according to their own needs. The preset text is text that can be used to perform the corresponding function. That is, when the recognized text corresponds to the preset text, the voice assistant can be woken up, thereby simplifying the wake-up process.

[0145] Therefore, after obtaining the text to be recognized in this round, its credibility is determined. If credibility is determined, it is further determined whether the text includes a form of address. If not, the text is matched against preset texts in a whitelist. Once the content of the text matches the preset texts in the whitelist, the voice assistant can be activated and the vehicle can be controlled to perform the function corresponding to the recognized text. Thus, by matching the recognized text with preset texts, the voice assistant can be activated, simplifying the activation steps and providing a more natural and convenient voice interaction method.

[0146] like Figure 18 As shown, the preset text in the whitelist is determined in the following way;

[0147] A2611. Obtain multiple historical recognized texts within the target time period. In other words, the preset texts in the whitelist can be collected and mined from multiple historical recognized texts. For example, when performing a wake-up judgment in the current round, the recognized texts from the previous round and previous rounds can be collected and used as the preset texts for this round. Thus, the construction of the whitelist does not require spending a lot of time considering the input of preset texts. At the same time, continuously updating or caching new recognized texts helps ensure the applicability of the preset texts and better meets user needs.

[0148] A2612. From multiple historical identified texts, select those with a length greater than the target length and a frequency greater than the target frequency or a frequency ranking higher than the target ranking as candidate texts. That is, in the construction of the whitelist, the length of the historical identified texts can be used as a filter, and if the length of a historical identified text is greater than the target length, it meets one of the conditions for being selected as a preset text. A historical identified text with an occurrence frequency greater than the target frequency or a frequency ranking higher than the target ranking is also considered as a condition for being selected as a preset text. Specifically, the usage frequency of multiple historical identified texts is statistically analyzed to form a list S. L The instructions are sorted by frequency and can be selectively retained, such as retaining the top 30% of instructions, resulting in an instruction list S. F Instruction List S F The historical identified text in the column is used as candidate text in the whitelist.

[0149] A2613. Select candidate texts that match the target sentence structure as preset texts. That is, after obtaining the instruction list S... F The instruction list S can be used. F Each instruction in the process is judged to conform to the target sentence pattern, thereby ensuring that only compliant and well-organized instructions are selected, resulting in the final instruction list, i.e., the final whitelist.

[0150] like Figure 19 As shown, candidate texts that match the target sentence structure are used as preset texts, including:

[0151] A2614. When the candidate text includes sequentially arranged control action verbs and specific controlled objects, the candidate text shall be used as the preset text. That is, the condition for the candidate text to be used as the preset text includes sequentially arranged control action verbs and specific controlled objects.

[0152] The control action words can include "open," "close," "increase," and "decrease," and the specific controlled objects can be functional components inside the vehicle, such as windows, air conditioning, lights, and audio systems. Examples arranged sequentially include "Please turn on the air conditioning," where "open" is the control action word and "air conditioning" is the specific controlled object, or "Please increase the audio volume," where "increase" is the control action word and "audio system" is the specific controlled object. This type of candidate text can be used as preset text and clearly aligns with the user's intent to activate the voice assistant.

[0153] A2615, or if the candidate text includes control action words, degree modifiers and specific controlled objects arranged in sequence, the candidate text shall be used as the preset text.

[0154] Similarly, control action words can include opening, closing, raising, lowering, etc., and degree modifiers can indicate the degree of operation of the control action word, such as half. The specific control object can be a functional component inside the vehicle, such as the driver's side window, air conditioning, headlights, etc. The corresponding sentence structure is <control action> <degree modifier> <specific control point>.

[0155] It should be noted that degree modifiers should be standardized and clear. For example, "half" is more specific and clear than "a point" or "a part". Also, for the controlled object, there may be multiple descriptions, such as "car window" or "driver's side window", where the driver's side window is a specific control point relative to the car window.

[0156] For example, in some examples, the instruction is "open the car window" and the matching result is "open-car window", and the corresponding instruction sentence structure is <control action><general control point>. This sentence structure does not conform to a regular instruction and is not used as preset text. Or, the instruction is "open the car window a little" and the matching result is "open-a little-car window", and the corresponding instruction sentence structure is <control action>a little<general control point>. This sentence structure does not conform to a regular instruction and is not used as preset text. Or, the instruction is "open the driver's side window halfway" and the matching result is "open-half-driver's side window", and the corresponding instruction sentence structure is <control action><degree modifier><specific control point>. This sentence structure conforms to a regular instruction and can be used as preset text.

[0157] like Figure 20 As shown, after obtaining the text to be recognized in this round, the method also includes:

[0158] A262. If it is determined that the text identified in this round is credible, and that the text identified in this round includes a form of address and is not a purely formal form of address, and that the text identified in this round is a valid instruction, a wake-up instruction is issued to the vehicle.

[0159] In other words, once it is determined that the identified text is credible, and the identified text in this round is not an impure address and is a valid instruction that meets the wake-up requirements, a wake-up command can be issued to control the voice assistant to wake up, and then the corresponding components of the vehicle can be controlled according to the wake-up command.

[0160] The reliability can be determined based on the ASR reliability assessment model. Specifically, multiple features are constructed based on the decoding confidence of the ASR module and the Top N decoding results:

[0161] 1. Length of the final result: 4;

[0162] 2. The number of TopN results;

[0163] 3; Similarity between Top N results: Calculate the average Jaccard similarity of Top N results. The formula for calculating Jaccard similarity is: sim(X,Y)=||X∩Y|| / ||X∪Y||.

[0164] For example, the similarity between "open car window" and "open window" is 3 / 4 = 0.75; the similarity between "open car window" and "open car window" is 3 / 4; the average similarity is (0.75 + 0.75) / 2 = 0.75.

[0165] In addition, when constructing the ASR reliability judgment model, the standard deviation of the Top N length is: std([4,3,3]) = 0.47; - whether the final result contains letters: no (0); - whether the Top N contains empty strings: no (0); - the length of the same prefix in the Top N results: 0; - the length of the same suffix in the Top N results: 1.

[0166] The above features are summarized as: [4,3,0.75,0.47,0,0,0,1]; Feature expansion: adding second-order features (pairwise multiplication); Final feature summary: [4,3,0.75,0.47,0,0,0,1,16,12,3.0,1.88,0,0,0,4,12,9,2.25,1.41,0,0,0,3,3.0,2.25,0.56,0] .35,0.0,0.0,0.0,0.75,1.88,1.41,0.35,0.22,0.0,0.0,0.0,0.47,0,0,0.0,0.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,3,0.75,0.47,0,0,0,1).

[0167] In feature selection, one approach is to use L1 regularization: standardize the features, train an L1 regularized LR (Logistic Regression) model using all features, obtain the coefficients for each feature, and remove features with coefficients close to 0. Alternatively, a random forest approach can be used: train a random forest model using all features, and the model will provide an assessment of the importance of each feature, removing features below a set threshold.

[0168] And model training, using the final feature set to train an LR model.

[0169] like Figure 21 As shown, after obtaining the text to be recognized in this round, the method also includes:

[0170] A2631. If it is determined that the text identified in this round is credible and that the text identified in this round includes a form of address and is a pure form of address, the text identified in the previous round is obtained; wherein, the text identified in the previous round is the text identified in the previous round corresponding to the user's voice request, and the text identified in the previous round is unreliable, or the text identified in the previous round is credible and does not include a form of address and does not match the preset text.

[0171] A2632. If it is determined that the time interval between the current round of identified text and the previous round of identified text is less than the target duration, and the previous round of identified text is a valid instruction, a wake-up instruction is issued to the vehicle.

[0172] In other words, during the process of determining a voice request, if the text being recognized in the current round is deemed credible and includes a form of address, especially a purely formal form of address, the text being recognized in the previous round can be evaluated. Even if the text being recognized in the previous round does not include a form of address and does not match the preset text, if the text being recognized in the previous round is a valid instruction, the voice assistant can be activated, and the voice assistant can then control the vehicle-related components using the instructions in the text being recognized in the previous round.

[0173] Specifically, the execution steps can be referred to as steps A251 to A254 in the voice interaction method of the second aspect above.

[0174] In step A263, the text identified in the previous round is determined to be a valid instruction.

[0175] Specifically, the criteria for determining that the intent of the previously identified text is within the list of valid intents, that the length of the previously identified text is greater than the target length, and that the previously identified text points to a specific operation object and operation method. In other words, the conditions for determining that the previously identified text is a valid instruction include that the length of the previously identified text is greater than the target length, and that there is a specific operation object and operation method.

[0176] The text recognized in the previous round must be longer than the target length, such as more than 5 characters, or other target lengths can be set. The objects of operation can be in-vehicle functional components, such as windows, air conditioning, lights, and audio systems, and the operation methods can include opening, closing, raising, and lowering.

[0177] For example, in the text "Xiao P, please turn on the air conditioner", the length of the text being recognized is greater than the target length, the object of the operation is the air conditioner, and the operation method is to turn it on; or, in the text "Xiao P, please turn up the volume", the length of the text being recognized is greater than the target length, the object of the operation is the audio system, and the operation method is to turn up the volume.

[0178] Therefore, it can be determined that the user's output command is a valid command, which enables the voice assistant to be activated.

[0179] Determine whether the wake word is used as a form of address in the text being identified in this round, including:

[0180] If the text to be recognized in this round includes a wake word, the matching results between the recognized text and multiple preset rules are determined. In other words, when designing voice interaction, multiple preset rules can be pre-defined, and these preset rules can filter and identify the recognized text. These preset rules can be those that are default to the vehicle itself, or they can be flexibly set by the user according to their needs. For example, specific sentences can be selected as the matching content for the preset rules. Specifically, they can be set according to the actual functional needs of the vehicle, such as "Xiao P, please turn on the air conditioning," or "Xiao P, please open the driver's side window," or they can be other types or methods of preset rules. Reference can be made to A230 and A231 in the aforementioned rule engine.

[0181] If the identified text matches the target rules among multiple preset rules, the wake-up word is determined based on the matching results to determine whether it is used as a form of address in the identified text. In other words, after identifying the text, the content of the identified text is matched and analyzed to determine whether the wake-up word is the user's address to the voice assistant, i.e., whether the user is calling the voice assistant. The preset rules can be adapted to user prefixes and some action words, and are constructed using a trie to reduce perturbation to the model. Specifically, the input is a text query (the result of ASR output + the form of address), and the output is the query result matched by the preset rules, used for decision fusion for final judgment.

[0182] Therefore, when the identified text matches the target rule among multiple preset rules, the voice request command can be output, thereby waking up the voice assistant so that the voice assistant can perform the corresponding function operation based on the voice request. Specifically, phrases like "Xiao P, speak louder" or "Xiao P, lower your volume" conform to the sentence structure "Xiao P's XX," and we classify them as negative examples. However, phrases like "Xiao P's lab" and "Xiao P changing clothes" are related to business scenarios and are not easily distinguishable. Introducing such data would significantly disturb the training model; this part is handled by rules, and the corresponding result is accepted if a rule is matched.

[0183] Specifically, refer to A230 and A231 in the above-mentioned voice interaction methods, and determine them based on the rule engine.

[0184] Determining whether the wake word is used as a form of address in the text being identified in this round also includes:

[0185] If the identified text does not match any of the multiple preset rules in this round of identification, the positional encoding features and part-of-speech encoding features of the wake-up word in the identified text in this round are used to determine whether the wake-up word is used as a form of address in the identified text. In other words, during the specific execution process, after the identified text is matched against multiple preset rules, if none of the corresponding target rules match the content of the identified text, meaning the identified text is not within the user's preset rules, then the content of the identified text can be further analyzed using the positional encoding features and part-of-speech encoding features of the wake-up word in the identified text to determine whether the wake-up word is used as a form of address. Specifically, the positional encoding features and part-of-speech encoding features can be set and implemented based on POS Embedding (function) and Salutation Embedding (address embedding), respectively.

[0186] Specifically, the implementation can be referenced from step A232 in the above-mentioned voice interaction method, which is based on a classification model.

[0187] After determining whether the wake-up word functions as a form of address in the current round of text recognition based on its positional and part-of-speech encoding features, the method further includes:

[0188] If, based on positional encoding features and part-of-speech encoding features, it is determined that the wake word is not used as a form of address in the current round of text recognition, then the confusion level of the current round of text recognition is determined. Specifically, after identifying the wake word as not being used as a form of address based on the classification model, the confusion level of the recognized text is determined to further determine whether the wake word is a form of address, thereby improving the accuracy of the output results. The confusion level of the recognized text can be determined based on a language model, such as by preparing 3-gram and 4-gram language models in advance, inputting the recognized text into these models, and then calculating the confusion level. This step can refer to step A233 above.

[0189] If the confusion level is greater than the target confusion level, the wake word is determined not to be used as a form of address in the current round of text recognition. For example, phrases like "Xiao P's voice is louder" or "Xiao P's volume is lower" fit the sentence structure "Xiao P's XX," and are therefore classified as negative examples. Similarly, phrases like "Xiao P's lab" and "Xiao P changing clothes" are related to business scenarios and are difficult to distinguish. Introducing such data would significantly disturb the training model. This part is handled by rules; if a rule is matched, the corresponding result is accepted. This step can be referenced from step A234 above.

[0190] If the confusion level is not greater than the target confusion level, determine the keyword weights in the current round of recognition text; wherein, the keyword weights are used to represent the proportion of the target words in the word segmentation of the current round of recognition text; thus, when it is impossible to determine whether the wake word is a form of address based on the confusion level, the keyword weights of the recognition text can be determined, and based on the keyword weights, it can be further determined whether the wake word is a form of address in the recognition text. This step can refer to step A235 above.

[0191] If the keyword weight is greater than the target weight, the wake-up word is determined as the form of address in the current round of text recognition. Therefore, the wake-up word can be determined as the form of address based on the keyword weight, thus waking up the voice assistant. This step can be referred to step A236 above.

[0192] Therefore, in the specific implementation, the confusion degree of the wake word in the recognized text can be calculated in advance based on the 3-gram and 4-gram language models. If the calculated weighted confusion degree is greater than the target confusion degree, a negative example result is output; otherwise, the judgment is made in combination with the keyword weight.

[0193] The present invention also proposes a voice interaction device 1.

[0194] like Figure 22 As shown, the voice interaction device 1 according to the present invention includes: a text recognition module 10 and a sending module 20.

[0195] The text recognition module 10 is used to perform voice recognition on the received user voice request from the vehicle cabin to obtain the recognized text.

[0196] The sending module 11 is used to send a wake-up command to the vehicle when it is determined that the recognized text includes a wake-up word and the wake-up word is used as a form of address in the recognized text, so that the vehicle can wake up the voice assistant and the user for voice interaction according to the wake-up command.

[0197] According to the voice interaction device 1 of the present invention, when a user communicates in the vehicle cabin, even if the content of the communication contains instructions that are the same as or similar to the wake word, the voice interaction method of the present invention can accurately identify whether the user's voice request is a name for the voice assistant, thereby avoiding false wake-up and allowing the user to have a natural and smooth conversation. That is, during the conversation, there is no need to deliberately avoid content related to the wake word, thus improving the user experience.

[0198] The voice interaction device 1 also includes a first determining module, wherein, when determining whether the wake word is used as a form of address in the recognized text, the first determining module is used to determine the matching result of the recognized text with multiple preset rules;

[0199] Furthermore, it is also used to determine whether the wake word is used as a form of address in the identified text when the identified text matches the target rule among multiple preset rules, based on the matching result.

[0200] When determining whether a wake word is used as a form of address in the identified text, the first determining module is used to determine whether a wake word is used as a form of address in the identified text based on the positional encoding features and part-of-speech encoding features of the wake word in the identified text, when the identified text does not match any of the preset rules among multiple preset rules.

[0201] After determining whether the wake word functions as a form of address in the recognized text based on its positional and part-of-speech encoding features, the first determination module is used to:

[0202] Determine the confusion level of the recognized text if the wake word is not used as a form of address in the recognized text based on positional encoding features and part-of-speech encoding features.

[0203] In addition, it is also used to determine that the wake word is not used as a form of address in the recognized text when the confusion level is greater than the target confusion level.

[0204] After determining the level of obfuscation of the identified text, the first determining module is used to:

[0205] Under the condition that the confusion level is not greater than the target confusion level, the keyword weights in the identified text are determined; whereby the keyword weights are used to characterize the proportion of target words in the word segmentation of the identified text.

[0206] In addition, it is also used to determine the wake word as a form of address in the recognized text when the keyword weight is greater than the target weight.

[0207] When determining the keyword weights in the identified text, the first determining module is used for:

[0208] Identify keywords that intersect the sub-words of the identified text with the core word dictionary;

[0209] Additionally, the initial proportion is obtained by dividing the number of keywords by the sum of the number of sub-words and the number of keywords;

[0210] The initial proportions were normalized to obtain the keyword weights.

[0211] After determining that the recognition text includes a wake word, the sending module 11 is further configured to:

[0212] If it is determined that the wake word is not used as a form of address in the recognized text, no instruction to wake up the voice assistant will be issued.

[0213] The voice interaction device 1 also includes a second determining module. When it is determined that the recognized text includes a wake word, the second determining module is used to extract candidate words from the recognized text by sequence labeling.

[0214] And, it is used to identify the wake word from the candidate words by comparing the pinyin.

[0215] like Figure 23 As shown, the present invention also proposes a server 2, including a memory 21 and a processor 22. The memory 21 stores a computer program, which, when executed by the processor, implements the various processes of the above-mentioned voice interaction method and achieves the same technical effect. To avoid repetition, it will not be described in detail here.

[0216] This application also provides a non-volatile computer-readable storage medium for a computer program. When the computer program is executed by one or more processors 22, it implements the various processes of the above-described voice interaction method and achieves the same technical effect. To avoid repetition, it will not be described in detail here.

[0217] Those skilled in the art will understand that all or part of the processes in the above methods can be implemented by a computer program instructing related software. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), etc.

[0218] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0219] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more steps for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which this application pertains.

[0220] Although this application has been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A voice interaction method, characterized in that, include: The received user voice requests from the vehicle cabin are processed by voice recognition to obtain the recognized text for this round. If it is determined that the recognized text includes a wake word, and the wake word is used as a form of address in the recognized text, the previous round of recognized text is obtained; wherein, the previous round of recognized text is the recognized text corresponding to the previous round of user voice request, and the previous round of recognized text does not include a form of address; If the previously identified text is determined to be a valid instruction, a wake-up command is sent to the vehicle so that the vehicle can wake up the voice assistant and the user for voice interaction according to the wake-up command. If it is determined that the text to be recognized in this round does not include a title, the text to be recognized in this round, the time information of the text to be recognized in this round, and the audio region of the user's voice request to be recognized in this round are cached so that the text to be recognized in this round, along with the corresponding time information and audio region, can be cached for the next round of application. Determining that the identified text includes a wake word includes: extracting candidate words from the identified text through sequence labeling; and determining the wake word from the candidate words through pinyin comparison.

2. The voice interaction method according to claim 1, characterized in that, Determining whether the wake word is used as a form of address in the recognized text includes: Determine the matching results between the identified text and multiple preset rules; If the identified text matches the target rule among the multiple preset rules, the wake word is determined as a form of address in the identified text based on the matching result.

3. The voice interaction method according to claim 2, characterized in that, The step of determining whether the wake word is used as a form of address in the recognized text further includes: If the identified text does not match any of the preset rules, the system determines whether the wake word is used as a form of address in the identified text based on the positional encoding features and part-of-speech encoding features of the wake word in the identified text.

4. The voice interaction method according to claim 3, characterized in that, After determining whether the wake-up word is used as a form of address in the identified text based on its positional encoding features and part-of-speech encoding features, the method further includes: If, based on the location encoding features and the part-of-speech encoding features, it is determined that the wake word is not used as a form of address in the identified text, the confusion level of the identified text is determined. If the level of confusion is greater than the target level of confusion, it is determined that the wake word is not used as a form of address in the identified text.

5. The voice interaction method according to claim 4, characterized in that, After determining the obfuscation level of the identified text, the method further includes: If the confusion level is not greater than the target confusion level, the keyword weights in the identified text are determined; wherein, the keyword weights are used to characterize the proportion of target words in the word segmentation of the identified text; If the keyword weight is greater than the target weight, the wake word is determined to be a form of address in the identified text.

6. The voice interaction method according to claim 5, characterized in that, Determining the keyword weights in the identified text includes: Determine the keywords that are the intersection of the sub-words of the identified text and the core word dictionary; Divide the number of keywords by the sum of the number of sub-words and the number of keywords to obtain a preliminary proportion; The initial proportions are normalized to obtain the keyword weights.

7. The voice interaction method according to any one of claims 1-6, characterized in that, After determining that the identified text includes a wake word, the method further includes: If it is determined that the wake word is not used as a form of address in the recognized text, no instruction to wake up the voice assistant will be issued.

8. A voice interaction device, characterized in that, include: The text recognition module is used to perform voice recognition on the received user voice requests from the vehicle cabin to obtain the recognized text for this round. The sending module is used to obtain the previous round of recognition text when it is determined that the recognition text includes a wake-up word and the wake-up word is used as a form of address in the recognition text; wherein the previous round of recognition text is the recognition text corresponding to the previous round of user voice request, and the previous round of recognition text does not include a form of address; If the previously identified text is determined to be a valid instruction, a wake-up command is sent to the vehicle so that the vehicle can wake up the voice assistant and the user for voice interaction according to the wake-up command. If it is determined that the text to be recognized in this round does not include a title, the text to be recognized in this round, the time information of the text to be recognized in this round, and the audio region of the user's voice request to be recognized in this round are cached so that the text to be recognized in this round, along with the corresponding time information and audio region, can be cached for the next round of application. The audio zones refer to the areas of each seat in the vehicle cabin, including the driver's seat audio zone, the passenger seat audio zone, and the rear seat audio zone.

9. A server, characterized in that, The server includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the method according to any one of claims 1-7.

10. A non-volatile computer-readable storage medium for a computer program, characterized in that, When the computer program is executed by one or more processors, it implements the method according to any one of claims 1-7.

Images