Voice fusion self-developed NLU cloud arbitration method

Abstract

The invention belongs to the technical field of vehicle-mounted voice control, and particularly relates to a voice fusion self-research NLU cloud arbitration method. The vehicle terminal uploads a voice stream to the cloud terminal, and the cloud terminal analyzes the voice into a text result through a three-party voice service; the self-research model takes a JSON character string formed by the processing result as a semantic result, the cloud judges whether rejection is carried out or not according to a topic field value in the result, and if rejection is carried out, a three-party service result is used; if not, judging whether the semantic result of the segment exists in the configuration file, if so, adopting a self-research model result, and if not, adopting a three-party service result; if the preceding semantic exists, judging whether the preceding semantic has a context relationship with the current semantic or not, and if the preceding semantic has the context relationship with the current semantic, performing semantic conversion, and storing a modified semantic result into a context container; if the association does not exist, directly issuing a semantic result of the self-developed model to the vehicle-mounted terminal, and storing the semantic result into a context container; according to the invention, after the self-developed NLU and the three-party voice capability are fused, fusion arbitration is carried out, and issued semantics are correctly processed.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle-mounted voice control, and specifically relates to a voice fusion self-developed NLU cloud arbitration method. Background Art

[0002] Many mid-to-high-end car platforms are equipped with voice control functions, which are convenient for the driver or co-driver to use while driving. At present, in-vehicle voice is completely dependent on third-party services. Although it can meet the needs of common in-vehicle voice control, the voice control functions of various car manufacturers are similar, and the product differentiation and competitive advantage is not obvious. However, although the coverage of existing third-party voice skills has achieved the control of common voice functions, the actual voice skill execution effect may not meet the user's linkage needs in actual scenarios. In addition, some uncommon expressions are not fully covered by third-party voice skills. If users of existing skills have their own preferred expressions, they will not be covered, which will cause the experience to be unintelligent. Summary of the invention

[0003] In order to overcome the above problems, the present invention provides a voice fusion self-developed NLU cloud arbitration method, which can integrate the self-developed NLU with the three-party voice capabilities for fusion arbitration, correctly process the sent semantics, adapt to cloud platformization, increase portability, improve self-developed capabilities, and ensure that the semantic results are correctly sent.

[0004] A voice fusion self-developed NLU cloud arbitration method includes the following contents:

[0005] Step 1: The vehicle uploads the voice stream to the cloud and creates a context container for this conversation in the cloud. The context container is used to store semantics, including topic, nlu, and skill fields.

[0006] Step 2: The cloud transmits the voice stream to the three-party voice service. The three-party voice service analyzes the voice stream, converts the voice stream into text results, and returns them to the cloud.

[0007] Step 3: After the cloud obtains the text, it transmits it to the self-developed model built into the vehicle computer. The self-developed model is the open source BERT model. After receiving the text, the self-developed model processes the text and repackages the processing result into a JSON string as a semantic result and returns it to the cloud. The string has three fields, namely topic, skill and nlu. The topic field is used to determine whether the model has successfully parsed the semantics. When the topic value is dm.rt.output, it means that the model has not parsed the semantics, that is, it refuses to recognize; when the topic value is dm.output, it means that the model has successfully parsed the semantics; the skill field represents the domain to which the received text belongs; nlu is the specific action to be performed after receiving the text;

[0008] Step 4: The cloud receives the semantic result of the self-developed model and determines whether it is rejected based on the value of the topic field in the result. If it is rejected, the text result transmitted by the third-party service is used. If it is not rejected, that is, when the topic value is dm.output, it means that the model has successfully parsed the semantics. At this time, the cloud will read the configuration file to determine whether the value of the skill field in the segment semantic result exists in the configuration file. If it exists, the semantic result of the self-developed model is used. If not, the text result of the third-party service is used.

[0009] Step 5. If the semantic result of the self-developed model is used, the previous semantics are searched from the context container of step 1. If there is no previous semantics in the context container, the semantic result of the self-developed model is directly sent to the vehicle end; if there is previous semantics, according to the value of the nlu field in the semantic result returned by the self-developed model, it is determined whether the previous semantics has a contextual relationship with the current semantics. If there is a correlation, a semantic conversion is performed, that is, the skill value in the current semantic result is changed to the skill value in the previous semantics, and the modified semantic result is stored in the context container; if there is no correlation, the semantic result of the self-developed model is directly sent to the vehicle end, and the semantic result is stored in the context container, and the text result of the third-party service is discarded.

[0010] Before step 4, after the cloud receives the text returned by the third-party service, if the self-developed model has not returned the semantic result, the cloud waits for the self-developed model to return the semantic result, and the cloud does not perform the next step of processing; if the self-developed model does not return within the set time, the text result of the third-party service is directly used, and the context of the third-party service will not be recorded in the context container of the cloud.

[0011] In step 3, the skill field represents the field to which the received text belongs. When the skill value is vehicle control, it means that the received text belongs to the vehicle control field. When the skill value is vehicle system control, it means that the received text belongs to the vehicle system control field.

[0012] In step 6, based on the value of the nlu field in the semantic result returned by the self-developed model, it is judged whether the previous semantics has a contextual relationship with the current semantics. Specifically, it is judged whether the previous semantics contains the value of the nlu field this time. If it is contained, it means that the previous semantics and the current semantics have a contextual relationship.

[0013] The training process of the self-developed model in step 3 is as follows:

[0014] The BERT model has 6 layers of encoders. The required training data in the following formats are input into the BERT model for training until the accuracy of the model output results reaches 96%. The model training is considered completed, where the accuracy of the model output results is the number of correct output results / total number of input data*100%.

[0016] The training data format is as follows:

[0017] {"text":"(I|)(wear too much|wear too much|)(too|) hot; the temperature is (too|)(high|hot); (let

[0018] |)(cooler|lower temperature|lower temperature|cooler temperature);(turn on|switch|change to|) cooling mode; it's so hot; I'm (sweating|smoking);","contain":"heat; temperature; air temperature; cooler; cooler; cooler; cooler; cooler","reject":"air conditioning; car windows;","scene":"cooling mode-open"}.

[0019] Beneficial effects of the present invention:

[0020] After acquiring the text, the present invention asynchronously calls the self-developed model, and after receiving the results returned by the self-developed model and the third-party service, arbitrates the results and rewrites the results according to the context to ensure that the semantics can be sent down correctly. DETAILED DESCRIPTION

[0021] The present invention will be further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention.

[0022] In the description of the present invention, unless otherwise clearly specified and limited, the terms "connected", "connected", and "fixed" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0023] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0024] In the description of this embodiment, the terms "upper", "lower", "left", "right" and other directions or positional relationships are only for the convenience of description and simplification of operation, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used to distinguish in the description and have no special meaning.

[0025] Example 1

[0026] A voice fusion self-developed NLU cloud arbitration method includes the following contents:

[0027] Step 1: The vehicle terminal (Dlife6.0 system) uploads the voice stream to the cloud and creates a context container for this conversation in the cloud. The context container is used to store semantics, including topic, nlu and skill fields.

[0028] Step 2: The cloud transmits the voice stream to the three-party voice service. The three-party voice service analyzes the voice stream, converts the voice stream into text results, and returns them to the cloud.

[0029] Step 3: After the cloud obtains the text, it transmits it to the self-developed model built into the vehicle computer. The self-developed model is the open source BERT model. After receiving the text, the self-developed model processes the text and repackages the processing result into a JSON string as a semantic result and returns it to the cloud. The string has three fields, namely topic, skill and nlu. The topic field is used to determine whether the model has successfully parsed the semantics. When the topic value is dm.rt.output, it means that the model has not parsed the semantics, that is, it refuses to recognize; when the topic value is dm.output, it means that the model has successfully parsed the semantics; the skill field represents the domain to which the received text belongs; nlu is the specific action to be performed after receiving the text;

[0030] Step 4: The cloud receives the semantic result of the self-developed model and determines whether it is rejected based on the value of the topic field in the result. If it is rejected, the text result transmitted by the third-party service is used. If it is not rejected, that is, when the topic value is dm.output, it means that the model has successfully parsed the semantics. At this time, the cloud will read the configuration file to determine whether the value of the skill field in the segment semantic result exists in the configuration file. If it exists, the semantic result of the self-developed model is used. If not, the text result of the third-party service is used.

[0031] Step 5: If the semantic result of the self-developed model is used, the previous semantics are searched from the context container of step 1. If there is no previous semantics in the context container, the semantic result of the self-developed model is directly sent to the vehicle end; if there is a previous semantics, according to the value of the nlu field in the semantic result returned by the self-developed model, it is determined whether the previous semantics has a contextual relationship with the current semantics. If there is a correlation, a semantic conversion is performed, that is, the skill value in the current semantic result is changed to the skill value in the previous semantics, and the modified semantic result is stored in the context container; if there is no correlation, the semantic result of the self-developed model is directly sent to the vehicle end, and the semantic result is stored in the context container, and the text result of the third-party service is discarded;

[0032] Before step 4, after the cloud receives the text returned by the third-party service, if the self-developed model has not returned the semantic result, the cloud waits for the self-developed model to return the semantic result, and the cloud does not perform the next step of processing; if the self-developed model does not return within the set time, the text result of the third-party service is directly used, and the context of the third-party service will not be recorded in the context container of the cloud.

[0033] In step 3, the skill field represents the field to which the received text belongs. When the skill value is vehicle control, it means that the received text belongs to the vehicle control field. When the skill value is vehicle system control, it means that the received text belongs to the vehicle system control field.

[0034] In step 6, based on the value of the nlu field in the semantic result returned by the self-developed model, it is judged whether the previous semantics has a contextual relationship with the current semantics. Specifically, it is judged whether the previous semantics contains the value of the nlu field this time. If it is contained, it means that the previous semantics and the current semantics have a contextual relationship.

[0035] The training process of the self-developed model in step 3 is as follows:

[0036] The BERT model has 6 layers of encoders. The required training data in the following formats are input into the BERT model for training until the accuracy of the model output results reaches 96%. The model training is considered completed, where the accuracy of the model output results is the number of correct output results / total number of input data*100%.

[0038] The training data format is as follows:

[0039] {"text":"(I|)(wear too much|wear too much|)(too|) hot; the temperature is (too|)(high|hot); (let

[0040] |)(cooler|lower temperature|lower temperature|cooler temperature);(turn on|switch|change to|) cooling mode; it's so hot; I'm (sweating|smoking);","contain":"heat; temperature; air temperature; cooler; cooler; cooler; cooler; cooler","reject":"air conditioning; car windows;","scene":"cooling mode-open"}.

[0041] When the user says: "I'm too hot, I'm a little too dressed", "Turn on the cooling mode", etc., the words contain keywords such as "hot" and "cooling". At this time, we think that a series of actions such as opening the window or turning on the air conditioner can be used to cool down the car. We call all these behaviors and actions the cooling mode, so we convert this problem into data according to the needs, for example:

[0042] Input: I feel a little hot -> Execution: open the car window, turn on the air conditioner -> Name these actions: cooling mode-open -> Output the final result: cooling mode-open.

[0043] The model first extracts the features of the training data, converts them into a three-dimensional tensor, and inputs the three-dimensional tensor into a 6-layer decoder for training. After the training set data is trained, the model will be tested. The test data is as follows (this is just an example. The test set will have a large amount of test data of this type. The statistical model calculates the results of these data. If the accuracy rate reaches 96%, it can be considered that the model training goal has been achieved and can be used): "It's a bit cold today", and the expected result is returned: "Cooling mode". If the result returned by the model is the same as the expected result, it means that the model has been trained successfully.

[0044] Example 2

[0045] 1. The vehicle uploads the voice stream (binary stream) to the Pentium cloud, and creates a context container for this conversation on the Pentium cloud to store semantics (semantics include topic, nlu, and skill fields, which are grouped into an object. A queue is created in memory. A queue is a data structure used to store a series of Java objects of the same type, with a first-in, first-out feature).

[0046] 2. The Pentium Cloud transmits the voice stream to the three-party voice service. The three-party voice service analyzes the voice stream, converts the voice stream into text, and returns it to the Pentium Cloud;

[0047] 3. The Pentium cloud obtains the text and asynchronously calls the self-developed model. The self-developed model uses the open source BERT model. The BERT model has a 6-layer encoder. The training data format is as follows (the following is a data example, and multiple data of this type are required):

[0048] {"text":"(I|)(wear too much|wear too much|)(too|) hot; temperature (too|)(high|hot); (let|)(cool down|cool down|lower temperature|cool down|cooler); (turn on|switch|change to|) cooling mode; it's (too|) hot; I'm (sweating|smoking);","contain":"heat; temperature; air temperature; cooler; cooler; cooler; cooler; cooler; cooler","reject":"air conditioning; car windows;","scene":"cooling mode-open"}, the model first extracts the features of the training data, converts them into a three-dimensional tensor, and inputs the three-dimensional tensor into a 6-layer decoder for training. After the training set data is trained, the model will be tested. The test data is as follows (this is just an example. The test set will have a large amount of test data, all of this type. The statistical model calculates the results of these data. If the accuracy rate reaches 96%, it can be considered that the model training goal has been achieved and can be used): "It's a bit cold today", and returns the expected result: "Cooling mode". If the result returned by the model is the same as the expected result, it means that the model has been trained and qualified. Thus, Pentium's self-developed model is formed. Pentium Cloud transmits the text to the model. After the self-developed model receives the text, it calculates through the self-developed model to get the result. The model repackages the result into a JSON string. This text is the semantics. This string will have three fields, namely topic, skill, and nlu. The topic field is used to determine whether the model has successfully parsed the semantics. When the topic value is dm.rt.output, it means that the model has not parsed the semantics, that is, it rejects recognition; when the topic value is dm.output, it means that the model has successfully parsed the semantics; the skill field indicates which field the semantics belongs to, for example, the skill value is vehicle control, which represents the vehicle control field, and the skill value is vehicle system control, which represents the vehicle system control field; nlu is the specific action to be performed. Finally, the self-developed model will return the JSON string result to the Pentium cloud.

[0049] 4. If Pentium Cloud receives the semantics returned by the third-party service, and the self-developed model has not returned the result, it will wait for the result of the self-developed model synchronously.

[0050] 5. Pentium Cloud receives the result of the self-developed model and determines whether to reject the recognition based on the value of the topic field in the result returned by the self-developed model. If so, the semantic result of the third-party service is used. If the self-developed model outputs the correct semantic result, Pentium Cloud will read the configuration file (this configuration file is manually configured, and the configuration content specifies which fields need to use the semantics of the self-developed model) and determine whether the value exists in the configuration file based on the value of the skill field in the result returned by the self-developed model. If so, the result of the self-developed model is used. If not, the semantic result of the third-party service is used.

[0051] 6. If the domain uses the results of the self-developed model, the semantics of the previous context are obtained from the context container in step 1 (if this is the first round of dialogue, the semantics of the previous context does not exist in the queue). If there is no semantics of the previous context in the context container, the result is directly sent to the vehicle end. When there is a contextual semantic, the value of the nlu field in the result returned by the model is used to determine whether the contextual semantic has a contextual relationship with the current semantic (for example, if the contextual semantic is to ask "whether to open the window", and the current semantic is "yes", it means that the contextual semantic has a contextual relationship with the current semantic). If there is a correlation, a semantic conversion is performed, and the skill value needs to be changed to the skill value in the contextual semantic (if the returned semantic is a general semantic, such as yes, no, good, confirm, in this semantic case, topic is a global domain value. If there is a correlation with the contextual semantic, the topic value needs to be changed from the full-text domain value to the value of the specific domain), and the modified result is stored in the context container; if there is no correlation, the semantic result of the self-developed model is directly sent to the vehicle end, and the semantic result is stored in the context container. The semantic results of the third-party service are discarded.

[0052] 7. If the self-developed model does not return within the corresponding time, the semantic result of the third-party service will be directly used, and the context of the third-party service will not be recorded in the context container of Pentium Cloud.

[0053] The preferred embodiments of the present invention are described in detail above. However, the protection scope of the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention. These simple variations all belong to the protection scope of the present invention.

[0054] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0055] In addition, various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the present invention, they should also be regarded as the contents disclosed by the present invention.

Claims

1. A voice fusion self-developed NLU cloud arbitration method, characterized in that: It includes the following: Step 1: The vehicle uploads the voice stream to the cloud and creates a context container for this conversation in the cloud. The context container is used to store semantics, including topic, nlu, and skill fields. Step 2: The cloud transmits the voice stream to the three-party voice service. The three-party voice service analyzes the voice stream, converts the voice stream into text results, and returns them to the cloud. Step 3: After the cloud obtains the text, it transmits it to the self-developed model built into the vehicle computer. The self-developed model is the open source BERT model. After receiving the text, the self-developed model processes the text and repackages the processing result into a JSON string as a semantic result and returns it to the cloud. The string has three fields, namely topic, skill and nlu. The topic field is used to determine whether the model has successfully parsed the semantics. When the topic value is dm.rt.output, it means that the model has not parsed the semantics, that is, it refuses to recognize; when the topic value is dm.output, it means that the model has successfully parsed the semantics; the skill field represents the domain to which the received text belongs; nlu is the specific action to be performed after receiving the text; Step 4: The cloud receives the semantic result of the self-developed model and determines whether it is rejected based on the value of the topic field in the result. If it is rejected, the text result transmitted by the third-party service is used. If it is not rejected, that is, when the topic value is dm.output, it means that the model has successfully parsed the semantics. At this time, the cloud will read the configuration file to determine whether the value of the skill field in the segment semantic result exists in the configuration file. If it exists, the semantic result of the self-developed model is used. If not, the text result of the third-party service is used. Step 5: If the semantic result of the self-developed model is used, the previous semantics are searched from the context container of step 1. If there is no previous semantics in the context container, the semantic result of the self-developed model is directly sent to the vehicle machine. If there is a previous semantics, according to the value of the nlu field in the semantic result returned by the self-developed model, it is determined whether the previous semantics has a contextual relationship with the current semantics. If there is a relationship, a semantic conversion is performed, that is, the skill value in the current semantic result is changed to the skill value in the previous semantics, and the modified semantic result is stored in the context container. If there is no association, the semantic results of the self-developed model will be directly sent to the vehicle computer, and the semantic results will be stored in the context container, and the text results of the third-party service will be discarded.

2. According to claim 1, a voice fusion self-developed NLU cloud arbitration method is characterized in that: Before step 4, after the cloud receives the text returned by the third-party service, if the self-developed model has not returned the semantic result, the cloud waits for the self-developed model to return the semantic result, and the cloud does not perform the next step of processing; If the self-developed model does not return within the set time, the text result of the third-party service will be directly used, and the context of the third-party service will not be recorded in the context container on the cloud.

3. According to claim 1, a voice fusion self-developed NLU cloud arbitration method is characterized in that: In step 3, the skill field represents the field to which the received text belongs. When the skill value is vehicle control, it means that the received text belongs to the vehicle control field. When the skill value is vehicle system control, it means that the received text belongs to the vehicle system control field.

4. According to claim 1, a voice fusion self-developed NLU cloud arbitration method is characterized in that: In step 6, based on the value of the nlu field in the semantic result returned by the self-developed model, it is judged whether the previous semantics has a contextual relationship with the current semantics. Specifically, it is judged whether the previous semantics contains the value of the nlu field this time. If it is contained, it means that the previous semantics and the current semantics have a contextual relationship.

5. According to claim 1, a voice fusion self-developed NLU cloud arbitration method is characterized in that: The training process of the self-developed model in step 3 is as follows: The BERT model has 6 layers of encoders. The required training data in the following formats are input into the BERT model for training until the accuracy of the model output results reaches 96%. The model training is considered completed, where the accuracy of the model output results is the number of correct output results / total number of input data*100%.

6. According to claim 5, a voice fusion self-developed NLU cloud arbitration method is characterized in that: The training data format is as follows: {"text":"(I|)(wear too much|wear too much|)(too|) hot; the temperature is (too|)(high|hot); (let|)(cool down|lower the temperature|lower the temperature|cooler); (turn on|switch|change to|) cooling mode; it's (too|) hot; I'm (sweating|smoking) so hot;","contain":"heat; temperature; air temperature; cool down; cooler; cooler; cooler; cooler; cool down","reject":"air conditioning; car windows;","scene":"cooling mode-open"}.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, a speech fusion self-developed NLU cloud arbitration method as described in any one of claims 1-6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a speech fusion self-developed NLU cloud arbitration method as described in any one of claims 1-6 is implemented.