Vehicle voice processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a vehicle voice processing method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the processing of text information transmitted by a vehicle terminal through an end-to-end model, executing a natural language processing task, obtaining a vehicle control instruction which can be executed by the vehicle terminal or a reply text which needs to be fed back to a user, and carrying out the processing of the vehicle voice. And sending the in-vehicle machine control instruction or the reply text to the in-vehicle machine end. The end-to-end model supports various types of natural language processing tasks and provides a uniform calling interface for the outside, so that compared with the traditional mode that a plurality of task execution modules need to be deployed in a dialogue system to execute different types of tasks, the scheme disclosed by the invention can greatly reduce the deployment and calling complexity of the dialogue system.

Description

Technical Field

[0001] The present application relates to the technical field of vehicle voice processing, and in particular to a vehicle voice processing method, device, electronic device and storage medium. Background Art

[0002] In order to facilitate users to operate vehicles, vehicles are often equipped with dialogue systems. Users can interact with the dialogue system to control vehicle equipment such as windows and air conditioners. Specifically, the user's input voice is first collected, and then the voice is processed by natural language to generate instructions that can be executed by the vehicle or text that needs to be fed back to the user. In traditional dialogue systems, multiple types of task execution modules are usually deployed to support different types of natural language processing tasks, and multiple rounds of conversations may be required between users and dialogue systems. The types of natural language processing tasks required to be executed in each round of conversation may be different, and different task execution modules need to be called for each round of conversation, resulting in high deployment complexity and call complexity of the dialogue system. For example, the dialogue system calls the Natural Language Understanding (NLU) module to perform NLU tasks on the collected voice to generate vehicle control instructions and then sends them to the vehicle end. After the vehicle end executes the instructions, the dialogue system needs to reply to the user. At this time, the Natural Language Generation (NLG) module needs to be called to reply. In this case, the dialogue system needs to deploy two modules, NLU and NLG, and needs to call the two modules according to the progress of the conversation. Overall, the complexity is high. Therefore, how to reduce the complexity of the dialogue system is an urgent problem to be solved. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure provides a vehicle speech processing method, device, electronic device and storage medium.

[0004] In a first aspect, the present disclosure provides a method for processing vehicle speech, comprising:

[0005] Receiving text information sent by the vehicle terminal; the text information is obtained by the vehicle terminal performing voice recognition on the collected user voice data;

[0006] Input the text information and the first vehicle machine device state information into an application program interface of an end-to-end model, and call the application program interface to control the end-to-end model to perform a natural language understanding task or a natural language generation task on the input text information to obtain a corresponding result; wherein, if the natural language understanding task is performed, the result is used to indicate a vehicle machine control instruction; if the natural language generation task is performed, the result is used to indicate a reply text;

[0007] The vehicle computer control instruction or the reply text is sent to the vehicle computer end, so that the vehicle computer end executes the vehicle computer control instruction or feeds back the reply text to the user.

[0008] In some embodiments, after receiving the text message sent by the vehicle terminal, the method further includes:

[0009] Acquire first vehicle machine device status information associated with the text information;

[0010] Inputting the first vehicle machine device status information into the application program interface of the end-to-end model;

[0011] The step of controlling the end-to-end model to perform a natural language understanding task or a natural language generation task on the text information by calling the application program interface to obtain a corresponding result includes:

[0012] The application program interface is called to control the end-to-end model to perform a natural language understanding task or a natural language generation task on the text information and the first vehicle device status information to obtain corresponding results.

[0013] In some embodiments, the step of controlling the end-to-end model to perform a natural language understanding task or a natural language generation task on the text information and the first vehicle device status information by calling the application program interface to obtain a corresponding result includes:

[0014] Converting the state information of the first vehicle machine device into a state vector;

[0015] The text information and the state vector are concatenated to obtain a concatenated vector, and the concatenated vector is input into the end-to-end model to perform a natural language understanding task or a natural language generation task to obtain a corresponding result.

[0016] In some embodiments, the result output by the end-to-end model includes prompt information; before sending the vehicle control instruction or the reply text to the vehicle end, the method further includes:

[0017] If it is determined according to the prompt information that the end-to-end model performs a natural language understanding task, then converting the structured data contained in the result into a vehicle control instruction executable by the vehicle end;

[0018] If it is determined according to the prompt information that the end-to-end model performs a natural language generation task, no conversion operation is performed.

[0019] In some embodiments, the end-to-end model includes a plurality of encoders connected in sequence and a plurality of decoders connected in sequence, and an output end of the encoder of the last layer is connected to an input end of the plurality of decoders; and the step of inputting the concatenated vector to the end-to-end model to perform a natural language understanding task or a natural language generation task to obtain a corresponding result includes:

[0020] Inputting the concatenated vector into the plurality of sequentially connected encoders for encoding to obtain an encoding vector output by the last layer of encoders;

[0021] Inputting the encoding vectors into the plurality of decoders connected in sequence respectively;

[0022] The multiple decoders connected in sequence each decode the encoding vector and the decoding vector output by the connected previous-level encoder and output a decoding vector, wherein the decoding vector output by the decoder of the last layer is the result output by the end-to-end model.

[0023] In some embodiments, after receiving the text message sent by the vehicle terminal, the method further includes:

[0024] Get historical session information;

[0025] The historical session information is input into the application program interface of the end-to-end model, and the application program interface is called to control the end-to-end model to perform a natural language understanding task or a natural language generation task on the input historical session information, the text information and the first vehicle device status information.

[0026] In some embodiments, after sending the vehicle computer control instruction or the reply text to the vehicle computer end, the method further includes:

[0027] receiving second vehicle machine device status information sent by the vehicle machine end; the second vehicle machine device status information is used to indicate whether the vehicle machine end successfully executes the result of the vehicle machine control instruction or successfully feeds back the reply text;

[0028] The text information and the second vehicle-machine device status information are input into the application program interface of the end-to-end model, and the application program interface is called to control the end-to-end model to perform a natural language understanding task or a natural language generation task on the input text information and the second vehicle-machine device status information to obtain corresponding results and feed them back to the vehicle-machine end.

[0029] In a second aspect, the present disclosure provides a vehicle speech processing device, comprising:

[0030] An information acquisition module, used for receiving text information sent by the vehicle terminal; the text information is obtained by the vehicle terminal performing voice recognition on the collected user voice data;

[0031] a processing module, configured to input the text information into an application program interface of an end-to-end model, and call the end-to-end model to perform a natural language understanding task or a natural language generation task on the input text information to obtain a corresponding result; wherein, if the natural language understanding task is performed, the result is used to indicate a vehicle control instruction; and if the natural language generation task is performed, the result is used to indicate a reply text;

[0032] The sending module is used to send the vehicle computer control instruction or the reply text to the vehicle computer end, so that the vehicle computer end executes the vehicle computer control instruction or feeds back the reply text to the user.

[0033] In a third aspect, the present disclosure provides an electronic device, including: a memory and a processor;

[0034] The memory is configured to store computer program instructions; the processor is configured to execute the computer program instructions, so that the electronic device implements the vehicle voice processing method as described in the first aspect and any one of the first aspects.

[0035] In a fourth aspect, the present disclosure provides a readable storage medium, comprising: computer program instructions; at least one processor of an electronic device executes the computer program instructions, so that the electronic device implements the vehicle voice processing method as described in the first aspect and any one of the first aspects.

[0036] In a fifth aspect, the present disclosure provides a computer program product, and a vehicle-mounted terminal runs the computer program product so that the electronic device implements the vehicle voice processing method as described in the first aspect and any one of the first aspects.

[0037] The disclosed embodiment provides a vehicle speech processing method, device, electronic device and storage medium, wherein the method uses an end-to-end model to process text information sent by the vehicle terminal and the first vehicle device status information, performs a natural language processing task, obtains a vehicle control instruction executable by the vehicle terminal or a reply text that needs to be fed back to the user, and then sends the vehicle control instruction or reply text to the vehicle terminal. Since the end-to-end model supports multiple types of natural language processing tasks and provides a unified calling interface to the outside, the deployment and calling complexity is greatly reduced compared with the traditional method of deploying multiple task execution modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 A flowchart of a method for processing vehicle speech provided by an embodiment of the present disclosure;

[0041] Figure 2 A schematic diagram of the structure of an end-to-end model provided in one embodiment of the present disclosure;

[0042] Figure 3 A flowchart of a method for processing vehicle speech provided by another embodiment of the present disclosure;

[0043] Figure 4 A schematic diagram of a framework for processing vehicle speech provided by an embodiment of the present disclosure;

[0044] Figure 5 A flowchart of a method for processing vehicle speech provided by another embodiment of the present disclosure;

[0045] Figure 6 A structural diagram of a vehicle speech processing device provided by an embodiment of the present disclosure;

[0046] Figure 7 A structural diagram of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0048] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation method separately. It should be noted that the brief description of the terms in this application is only for the convenience of understanding the implementation methods described below, and is not intended to limit the implementation methods of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0049] To solve this problem, the present disclosure provides a method for processing vehicle speech, which uses an end-to-end model to process text information sent by the vehicle computer, performs natural language processing tasks, obtains vehicle computer control instructions that can be executed by the vehicle computer or reply text that needs to be fed back to the user, and then sends the vehicle computer control instructions or reply text to the vehicle computer. Since the end-to-end model supports multiple types of natural language processing tasks and provides a unified calling interface to the outside world, the deployment and calling complexity is greatly reduced compared with the traditional method of deploying multiple task execution modules.

[0050] In addition, the present disclosure can also utilize multimodal information to improve the accuracy of the results obtained by the end-to-end model performing natural speech processing tasks, thereby helping to improve the voice interaction experience between the user and the vehicle-machine dialogue system, wherein the multimodal information includes: text information and vehicle-machine device status information.

[0051] The method provided by the present disclosure can be implemented by the vehicle voice processing device provided by the present disclosure, and the device can be implemented by any software and / or hardware. The vehicle-machine dialogue system is the dialogue system in the vehicle scenario described above, and can also be called a vehicle dialogue system. The dialogue systems described below all refer to the vehicle-machine dialogue system. Among them, the dialogue system may include a client (i.e., a vehicle-machine end) and a vehicle voice processing device, wherein the client is configured in a vehicle, wherein the vehicle may be, but is not limited to, any of the following types: smart cars, new energy vehicles, ordinary passenger cars, convertible passenger cars, high-end passenger cars, small passenger cars, convertibles, hatchback passenger cars, station wagons, multi-purpose passenger cars, short-head passenger cars, off-road passenger cars, special passenger cars, etc. The vehicle voice processing device can be deployed on the service end, such as any type of server such as a cloud server, an Internet of Things server, a service cluster, etc.

[0052] Next, the vehicle voice processing method provided by the present disclosure will be described in detail in combination with the accompanying drawings and application scenarios.

[0053] Figure 1 This is a flow chart of a method for processing vehicle speech provided by an embodiment of the present disclosure. Figure 1As shown, the method provided in this embodiment may include:

[0054] S101, receiving text information sent by a vehicle terminal; the text information is obtained by the vehicle terminal performing voice recognition on collected user voice data.

[0055] In the following embodiments, the method for processing vehicle voice executed by a server is taken as an example for explanation.

[0056] The user can wake up the client of the dialogue system through voice or by pressing a physical button on the vehicle or other means. The dialogue system then collects the voice input by the user through an audio collection device and converts the voice into text information using an STT (Speech to Text) model. The vehicle computer then transmits the text information to the server through a communication link. The present disclosure does not limit parameters such as the content and length of the text information.

[0057] S102. Input text information into the application program interface of the end-to-end model, and call the application program interface to control the end-to-end model to perform a natural language understanding task or a natural language generation task on the input text information to obtain a corresponding result; if the natural language understanding task is performed, the result is used to indicate the vehicle control instruction; if the natural language generation task is performed, the result is used to indicate the reply text.

[0058] The vehicle control command is a command used to control the status of the vehicle equipment, which may include but is not limited to air conditioning, windows, lights, engines, vehicle terminals, etc. The reply text is content used to give feedback to the user in the vehicle, which may be provided by voice, by screen display, or by a combination of the two.

[0059] The server pre-deploys a trained end-to-end model. The end-to-end model is a deep learning network built and trained for human-vehicle dialogue interaction scenarios. The end-to-end model supports the execution of NLG tasks and NLU tasks, and can automatically distinguish tasks and obtain results with accuracy that meets the requirements. The end-to-end model in the server provides a unified external interface for calling. When executing natural language processing tasks, it can avoid distinguishing task types and calling interfaces, which greatly reduces complexity. For example, see Figure 2As shown, the end-to-end model can adopt a traditional Transformer structure, including an encoder 201 and a decoder 202. The number of encoders 201 and decoders 202 can be multiple, and the number can be inconsistent. For example, a structure of 6 layers of encoders 201 and 6 layers of decoders 202, or a structure of 12 layers of encoders 201 and 12 layers of decoders 202 can be adopted. The present disclosure does not limit the specific structure of the end-to-end model. In the end-to-end model, multiple encoders 201 are connected in sequence, and the output end of the last layer of encoders 201 is respectively connected to the input ends of multiple decoders 202. It should be understood that Figure 2 The direction of the arrow in the figure indicates the direction of information transmission.

[0060] Please combine Figure 2 As shown, the server can input text information (which can also be understood as a text vector) into the end-to-end model by calling the application program interface of the end-to-end model, and control the end-to-end model to perform natural language understanding tasks. It should be noted that the vehicle-side can obtain a vector form of expression by using the speech-to-text (STT) model for speech recognition. The vehicle-side does not need to convert the vector form of expression into a form that can be understood by the user, so as to facilitate direct use by the server and avoid the server repeatedly converting the expression form of the text information, which is conducive to improving the processing efficiency of the server. The server inputs the text information into the first layer encoder 201, and the first layer encoder 201 encodes the text information. The first layer encoder 201 outputs the encoding vector to the second layer encoder 201, and so on, until the encoding vector output by the last layer encoder 201 is obtained, wherein the structure of each layer encoder 201 can be the same but use different weight parameters. When the encoder 201 encodes a specific position of the splicing vector, it not only pays attention to the elements of the current specific position, but also pays attention to the elements of other positions, so that the elements of the current specific position can be better encoded. The server inputs the encoding vector output by the last layer encoder 201 into each layer decoder 202 respectively and uses it in each layer decoder 202, which helps each layer decoder 202 focus on the appropriate position of the input vector. In the decoding stage, the last layer decoder 202 outputs an element at each time step, and continuously updates the time step until an end symbol is output, indicating that the entire decoder 202 has completed the output. The output of each step will be input into the first layer decoder 202 in the next time step. In the first time step, the initialization vector can be input into the first layer decoder 202 as another part of the input of the decoder 202.

[0061] The output of the end-to-end model is distinguished by prompts, and it is also distinguished whether the end-to-end model performs an NLG task or an NLU task. Specifically, the result output by the end-to-end model contains prompt information (which can also be understood as a prompt field). The prompt information can be but is not limited to being located at the starting position of the result output by the end-to-end model. The prompt information is specifically used to prompt the type of natural language understanding task. The server determines whether other conversion processing needs to be performed based on the prompt information. For example, if the server determines that the end-to-end model performs an NLU task based on the prompt information, it is necessary to further convert the structured data in the output result into a vehicle control instruction that can be executed by the vehicle side; if the server determines that the end-to-end model performs an NLG task based on the prompt information, there is no need to convert the result output by the model, and the structured data output by the model (such as a vector indicating the reply text) is directly sent to the vehicle side.

[0062] S103: Send the vehicle computer control instruction or the reply text to the vehicle computer end, so that the vehicle computer end executes the vehicle computer control instruction or feeds back the reply text to the user.

[0063] The server sends the vehicle control command or reply text to the vehicle terminal through the communication link. The vehicle terminal can execute the vehicle control command to adjust the status of the corresponding vehicle device, or use the text to speech (TTS) model to convert the reply text into speech and output it to the user to meet the purpose of the user's interaction with the dialogue system.

[0064] In this embodiment, since the end-to-end model supports multiple types of natural language processing tasks and provides a unified calling interface to the outside world, the deployment and calling complexity is greatly reduced compared with the traditional method of deploying multiple task execution modules.

[0065] exist Figure 1 On the basis of the illustrated embodiment, the vehicle-machine end will change the state of the vehicle-machine device by executing the vehicle-machine control instruction. The vehicle-machine end can promptly report the updated second vehicle-machine device state information. The server will input the second vehicle-machine device state information and the text information obtained in the previous round of conversation into the end-to-end model. The end-to-end model will re-execute the natural language understanding task and feed back the result to the user, thereby improving the conversation experience.

[0066] Figure 3 This is a flow chart of a method for processing vehicle speech provided by another embodiment of the present disclosure. Figure 3 As shown, the method of this embodiment includes:

[0067] S301, receiving text information sent by a vehicle terminal; the text information is obtained by the vehicle terminal performing voice recognition on collected user voice data.

[0068] In this embodiment, step S301 and Figure 1 Step S101 in the embodiment shown is similar, and can be referred to Figure 1 For the sake of brevity, the detailed description of the illustrated embodiment will not be repeated here.

[0069] S302: Acquire first vehicle device status information associated with the text information.

[0070] In some embodiments, the vehicle computer end can identify keywords in the converted text information, determine the associated vehicle computer device based on the keywords, collect the status information of the vehicle computer device to obtain the first vehicle computer device status information, and upload it to the server.

[0071] In other embodiments, the server may perform keyword recognition on the text information, and obtain the first vehicle device status information by querying the database based on the recognized keywords, wherein the database is deployed on the server and is mainly used to store the vehicle device status information; or, the server sends an information acquisition request to the vehicle terminal based on the recognized keywords, and the server obtains the first vehicle device status information according to the response information returned by the vehicle terminal.

[0072] For example, the text message is "Turn on the air conditioner", and the keyword is "air conditioner", which can determine that the current status of the air conditioner needs to be obtained; the vehicle computer can directly collect the air conditioner status and send it to the server, or the server can query the air conditioner status information from the database, or the server can also send an air conditioner status acquisition request to the vehicle computer. The vehicle computer responds to the received air conditioner status acquisition request and accesses the vehicle air conditioning system to obtain the air conditioning status information, which may include the current on / off status of the air conditioner. If the air conditioner is on, the current working mode, temperature and other information of the air conditioner can also be obtained, and then fed back to the server.

[0073] In some other embodiments, in addition to sending text messages, the vehicle computer end can also send keywords identified from the text messages to the server. The server no longer needs to perform keyword recognition on the text messages, and can use the additional keywords sent by the vehicle computer end as indication information to determine the vehicle computer device, and then query the database for the matching first vehicle computer device status information.

[0074] S303. Input the text information and the first vehicle-machine device status information into the application program interface of the end-to-end model, and call the application program interface to control the end-to-end model to perform a natural language understanding task or a natural language generation task on the input text information and the first vehicle-machine device status information to obtain corresponding results; if the natural language understanding task is performed, the result is used to indicate the vehicle-machine control instruction; if the natural language generation task is performed, the result is used to indicate the reply text.

[0075] For example, in combination Figure 2 As shown, the server first converts the state information of the first vehicle-machine device into a state vector. The server can pre-store the state vectors corresponding to various states of the vehicle-machine device, and then obtain the state vector corresponding to the state information of the first vehicle-machine device by querying; or, the state vector that meets the data format requirements of the end-to-end model can be obtained by encoding the state information of the first vehicle-machine device in real time. Then, the text information and the state vector are spliced ​​to obtain a spliced ​​vector. After that, the server inputs the spliced ​​vector to the first layer encoder 201, the first layer encoder 201 encodes the spliced ​​vector, and the first layer encoder 201 outputs the encoded vector to the second layer encoder 201, and so on, until the encoded vector output by the last layer encoder 201 is obtained, wherein the structure of each layer encoder 201 can be the same but use different weight parameters. When the encoder 201 encodes a specific position of the spliced ​​vector, it not only pays attention to the elements of the current specific position, but also pays attention to the elements of other positions, so that the elements of the current specific position can be better encoded. The server inputs the encoding vector output by the last layer encoder 201 into each layer decoder 202 respectively and uses it in each layer decoder 202, which helps each layer decoder 202 focus on the appropriate position of the input vector. In the decoding stage, the last layer decoder 202 outputs an element at each time step, and continuously updates the time step until an end symbol is output, indicating that the entire decoder 202 has completed the output. The output of each step will be input into the first layer decoder 202 in the next time step. In the first time step, the initialization vector can be input into the first layer decoder 202 as another part of the input of the decoder 202.

[0076] The output of the end-to-end model is distinguished by prompts, and it is also distinguished whether the end-to-end model performs an NLG task or an NLU task. Specifically, the result output by the end-to-end model contains prompt information (which can also be understood as a prompt field). The prompt information can be but is not limited to being located at the starting position of the result output by the end-to-end model. The prompt information is specifically used to prompt the type of natural language understanding task. The server determines whether other conversion processing needs to be performed based on the prompt information. For example, if the server determines that the end-to-end model performs an NLU task based on the prompt information, it is necessary to further convert the structured data in the output result into a vehicle control instruction that can be executed by the vehicle side; if the server determines that the end-to-end model performs an NLG task based on the prompt information, there is no need to convert the result output by the model, and the structured data output by the model (such as a vector indicating the reply text) is directly sent to the vehicle side.

[0077] S304: Send the vehicle computer control instruction or the reply text to the vehicle computer end, so that the vehicle computer end executes the vehicle computer control instruction or feeds back the reply text to the user.

[0078] The server sends the vehicle control command or reply text to the vehicle terminal through the communication link. The vehicle terminal can execute the vehicle control command to adjust the status of the corresponding vehicle device, or use the text to speech (TTS) model to convert the reply text into speech and output it to the user to meet the purpose of the user's interaction with the dialogue system.

[0079] In this embodiment, since the end-to-end model supports multiple types of natural language processing tasks and provides a unified calling interface to the outside world, the deployment and calling complexity is greatly reduced compared with the traditional way of deploying multiple task execution modules. In addition, the present disclosure uses multimodal information, namely text information and vehicle equipment status information, to improve the accuracy of natural language processing, which is beneficial to improving the user's voice interaction experience. In addition, the present disclosure uses multimodal information, namely text information and the first vehicle equipment status information, to improve the accuracy of the results obtained by the end-to-end model performing natural language processing, which is beneficial to improving the voice interaction experience between the user and the vehicle dialogue system.

[0080] exist Figure 3 On the basis of the illustrated embodiment, the vehicle-machine end will change the state of the vehicle-machine device by executing the vehicle-machine control instruction. The vehicle-machine end can promptly report the updated second vehicle-machine device state information. The server will input the second vehicle-machine device state information and the text information obtained in the previous round of conversation into the end-to-end model. The end-to-end model will re-execute the natural language understanding task and feed back the result to the user, thereby improving the conversation experience.

[0081] The method disclosed herein will be described in detail below in conjunction with a specific vehicle usage scenario.

[0082] See also Figure 4 As shown, in a specific embodiment, it is assumed that the user wakes up the dialogue system client deployed on the vehicle side and inputs the voice "open the front passenger window". The vehicle side receives the user's voice demand, calls the STT model to convert the voice into text, and the vehicle side then transmits the text to the cloud server through the link; the cloud server receives the text information, obtains the status information of the front passenger window for encoding, and inputs the encoded state vector and the text information into the end-to-end model together.

[0083] If the current passenger window is not open, the end-to-end model will generate structured data. The cloud server converts the structured data into a window control command that can be executed by the vehicle computer and sends it to the vehicle computer for execution. After the vehicle computer executes the command, the vehicle computer updates the window status stored in the cloud server. The window status also reflects the result of the command execution, whether it is a failure or success. Therefore, the state vector corresponding to the text "open the passenger window" and the updated window status is input into the end-to-end model again to generate the corresponding result. Assuming that the command execution fails, that is, the passenger window fails to open, the end-to-end model will generate a reply text "passenger window failed to open" and transmit it to the vehicle computer for feedback to the user; assuming that the command execution is successful, that is, the passenger window is successfully opened, the end-to-end model will generate a reply text "passenger window successfully opened" and transmit it to the vehicle computer for feedback to the user.

[0084] If the passenger window is currently open, the end-to-end model will generate a reply text "The window is already open", and the vehicle computer can provide feedback to the user via voice.

[0085] exist Figure 4 In the illustrated embodiment, the text corresponding to the user input voice, the vehicle device status information, and the input and output of the end-to-end model may be shown in Table 1 below:

[0086] Table 1

[0087]

[0088] Of course, [INFORM] and [REQUEST] in the output of the end-to-end model are prompt information, which distinguishes the natural language processing task type performed by the end-to-end model as NLU type or NLG type. Among them, the content of the reply text output by the end-to-end model can be set according to needs, and can be adjusted using the corresponding training data during the training phase.

[0089] Traditional dialogue systems have insufficient context understanding capabilities when performing natural language understanding tasks, which may cause the vehicle control instructions or reply texts generated by the dialogue system to not match the current conversation progress. For example, when traditional dialogue systems perform NLG tasks, they need to call the NLG module, and the input of the NLG module includes: the vehicle control instructions generated by the NLU module and the execution results of the instructions, without any other contextual information for reference, resulting in poor context understanding capabilities of traditional dialogue systems.

[0090] Figure 5 This is a flow chart of a method for processing vehicle speech provided by another embodiment of the present disclosure. Figure 5 As shown, the method of this embodiment includes:

[0091] S401, receiving text information sent by the vehicle terminal; the text information is obtained by the vehicle terminal performing voice recognition on collected user voice data.

[0092] S402: Acquire first vehicle device status information associated with the text information.

[0093] In this embodiment, steps S401 and S402 are Figure 3 Steps S301 and S302 of the embodiment shown are similar, and can be referred to Figure 3 For the sake of brevity, the detailed description of the illustrated embodiment will not be repeated here.

[0094] S403: Obtain historical session information.

[0095] The historical conversation information may include one or more of the text information corresponding to the previous round or rounds of voice interaction, the generated historical vehicle control instructions, or the historical reply text. Alternatively, the historical conversation information may also be information indicating the progress of the conversation. Specifically, which information is used as the historical conversation information associated with the text information can be determined based on the keywords in the text information or the content of the text information.

[0096] It should be noted that steps S402 and S403 may be executed in the order shown in this embodiment, or may be executed in parallel, or step S402 may be executed first and then step S403, which is not limited in the present disclosure.

[0097] S404. Input the historical conversation information, text information, and the first vehicle-machine device status information into the application program interface of the end-to-end model, and call the application program interface to control the end-to-end model to perform a natural language understanding task or a natural language generation task on the input historical conversation information, text information, and the first vehicle-machine device status information to obtain corresponding results; if the natural language understanding task is performed, the result is used to indicate the vehicle-machine control instruction; if the natural language generation task is performed, the result is used to indicate the reply text.

[0098] and Figure 3 The difference from the embodiment shown is that in this embodiment, the historical conversation information needs to be converted into a vector form and then concatenated with the vector corresponding to the text information and the state vector to obtain a concatenated vector, and then the concatenated vector is input into the end-to-end model for encoding and decoding. The encoding and decoding processes can be referred to in Figure 2 The relevant description of the illustrated embodiment will not be repeated here.

[0099] S405: Send the vehicle computer control instruction or the reply text to the vehicle computer end, so that the vehicle computer end executes the vehicle computer control instruction or feeds back the reply text to the user.

[0100] In this embodiment, step S405 and Figure 3 The step S304 of the embodiment shown is similar, and can be referred to Figure 3 For the sake of brevity, the detailed description of the illustrated embodiment will not be repeated here.

[0101] In this embodiment, by inputting historical conversation information into the end-to-end model, the model can use more contextual information as a reference when performing natural language processing tasks, thereby improving the model's contextual understanding ability and generating results that are more in line with the current conversation.

[0102] It should be noted that in this embodiment, historical conversation information is obtained, and the historical conversation information is concatenated with text information to obtain a concatenated vector, and then the concatenated vector is input into the end-to-end model to perform the natural language understanding task to obtain the corresponding result. Figure 5 The illustrated embodiment is similar and will not be described in detail here. This method can also improve the context understanding ability of the model to a certain extent, generate results that are more in line with the current conversation, and achieve the effect of improving the accuracy of the results obtained by natural language processing.

[0103] Figure 6 This is a structural diagram of a vehicle speech processing device provided by an embodiment of the present disclosure. Figure 6 As shown, the device 600 provided in this embodiment includes:

[0104] The information acquisition module 601 is used to receive text information sent by the vehicle terminal; the text information is obtained by the vehicle terminal performing voice recognition on the collected user voice data.

[0105] Processing module 602 is used to input the text information into the application program interface of the end-to-end model, and call the application program interface to control the end-to-end model to perform a natural language understanding task or a natural language generation task on the input text information to obtain a corresponding result; wherein, if the natural language understanding task is performed, the result is used to indicate the vehicle control instruction; if the natural language generation task is performed, the result is used to indicate the reply text.

[0106] The sending module 603 is used to send the vehicle control instruction or the reply text to the vehicle terminal, so that the vehicle terminal executes the vehicle control instruction or feeds back the reply text to the user.

[0107] In some embodiments, the information acquisition module 601 is further used to acquire first vehicle-mounted device status information associated with the text information. The processing module 602 is further used to input the first vehicle-mounted device status information into the application program interface of the end-to-end model. And, by calling the application program interface, the end-to-end model is controlled to perform a natural language understanding task or a natural language generation task on the text information and the first vehicle-mounted device status information to obtain a corresponding result.

[0108] In some embodiments, the processing module 602 is specifically used to convert the first vehicle device state information into a state vector; concatenate the text information and the state vector to obtain a concatenated vector, and input the concatenated vector into the end-to-end model to perform a natural language understanding task or a natural language generation task to obtain a corresponding result.

[0109] In some embodiments, the processing module 602 is also used to, before sending the vehicle control instruction or the reply text to the vehicle end, if it is determined according to the result output by the end-to-end model that the end-to-end model performs a natural language understanding task including prompt information, then convert the structured data contained in the result into a vehicle control instruction executable by the vehicle end; if it is determined according to the prompt information that the end-to-end model performs a natural language generation task, then no conversion operation is performed.

[0110] In some embodiments, the end-to-end model includes multiple encoders connected in sequence and multiple decoders connected in sequence, and the output end of the last layer of encoders is connected to the input ends of the multiple decoders; the processing module 602 is specifically used to input the splicing vector into the multiple encoders connected in sequence for encoding processing to obtain the encoding vector output by the last layer of encoder; the encoding vectors are respectively input into the multiple decoders connected in sequence; the multiple decoders connected in sequence each decode the encoding vector and the decoding vector output by the connected previous level encoder and output the decoding vector, wherein the decoding vector output by the last layer of decoder is the result output by the end-to-end model.

[0111] In some embodiments, the information acquisition module 601 is further used to acquire historical session information after receiving the text information sent by the vehicle terminal.

[0112] The processing module 602 is also used to input the historical session information into the application program interface of the end-to-end model, and call the application program interface to control the end-to-end model to perform a natural language understanding task or a natural language generation task on the input historical session information, the text information and the first vehicle device status information.

[0113] In some embodiments, after sending the vehicle control command or the reply text to the vehicle terminal, the information acquisition module 601 is further used to receive second vehicle device status information sent by the vehicle terminal. The second vehicle device status information is used to indicate whether the vehicle terminal successfully executes the result of the vehicle control command or successfully feeds back the reply text.

[0114] The processing module 602 is also used to input the text information and the second vehicle-machine device status information into the application program interface of the end-to-end model, call the application program interface to control the end-to-end model to perform a natural language understanding task or a natural language generation task on the input text information and the second vehicle-machine device status information to obtain corresponding results and feed them back to the vehicle-machine end.

[0115] The device provided in this embodiment can be used to implement the technical solution of any of the aforementioned method embodiments. Its implementation principle and technical effects are similar. Please refer to the detailed description of the aforementioned method embodiments. For the sake of brevity, they will not be repeated here.

[0116] Figure 7 This is a structural diagram of an electronic device provided by an embodiment of the present disclosure. Figure 7 As shown, the electronic device 700 provided in this embodiment includes: a memory 701 and a processor 702 .

[0117] The memory 701 may be an independent physical unit, and may be connected to the processor 702 via a bus 703. The memory 701 and the processor 702 may also be integrated together and implemented by hardware. The memory 701 is used to store program instructions, and the processor 702 calls the program instructions to execute the vehicle voice processing method provided by any of the above method embodiments.

[0118] Optionally, when part or all of the methods of the above embodiments are implemented by software, the electronic device 700 may also include only a processor 702. The memory 701 for storing programs is located outside the electronic device 700, and the processor 702 is connected to the memory through circuits / wires to read and execute the programs stored in the memory. The processor 702 may be a central processing unit (CPU), a network processor (NP) or a combination of a CPU and a NP. The processor 702 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0119] The memory 701 may include a volatile memory, such as a random-access memory (RAM); the memory may also include a non-volatile memory, such as a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory may also include a combination of the above types of memory.

[0120] Exemplarily, the present disclosure provides a chip system, which is applied to an electronic device including a memory and a sensor; the chip system includes: a processor; the processor is used to execute the vehicle voice processing method of the above embodiment.

[0121] Exemplarily, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor of an electronic device so that the vehicle voice processing method of the above embodiment is implemented when the computer program is executed by the electronic device.

[0122] Exemplarily, the present disclosure provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the vehicle voice processing method of the above embodiment.

[0123] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0124] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing vehicle speech, characterized in that: include: Receive text messages sent by the vehicle terminal; The text information is obtained by performing voice recognition on the collected user voice data by the vehicle terminal; Input the text information into an application program interface of an end-to-end model, and control the end-to-end model to perform a natural language understanding task or a natural language generation task on the text information by calling the application program interface to obtain a corresponding result; wherein, if the natural language understanding task is performed, the result is used to indicate a vehicle control instruction; if the natural language generation task is performed, the result is used to indicate a reply text; The vehicle computer control instruction or the reply text is sent to the vehicle computer end, so that the vehicle computer end executes the vehicle computer control instruction or feeds back the reply text to the user.

2. The method according to claim 1, characterized in that After receiving the text message sent by the vehicle terminal, the method further includes: Acquire first vehicle machine device status information associated with the text information; Inputting the first vehicle machine device status information into the application program interface of the end-to-end model; The step of controlling the end-to-end model to perform a natural language understanding task or a natural language generation task on the text information by calling the application program interface to obtain a corresponding result includes: The application program interface is called to control the end-to-end model to perform a natural language understanding task or a natural language generation task on the text information and the first vehicle device status information to obtain corresponding results.

3. The method according to claim 2, characterized in that The step of controlling the end-to-end model to perform a natural language understanding task or a natural language generation task on the text information and the first vehicle device status information by calling the application program interface to obtain a corresponding result includes: Converting the state information of the first vehicle machine device into a state vector; The text information and the state vector are concatenated to obtain a concatenated vector, and the concatenated vector is input into the end-to-end model to perform a natural language understanding task or a natural language generation task to obtain a corresponding result.

4. The method according to claim 1 or 2, characterized in that: The result output by the end-to-end model includes prompt information; before sending the vehicle control instruction or the reply text to the vehicle end, the method further includes: If it is determined according to the prompt information that the end-to-end model performs a natural language understanding task, then converting the structured data contained in the result into a vehicle control instruction executable by the vehicle end; If it is determined according to the prompt information that the end-to-end model performs a natural language generation task, no conversion operation is performed.

5. The method according to claim 3, characterized in that: The end-to-end model includes a plurality of encoders connected in sequence and a plurality of decoders connected in sequence, and the output end of the encoder of the last layer is connected to the input ends of the plurality of decoders; the step of inputting the concatenated vector into the end-to-end model to perform a natural language understanding task or a natural language generation task to obtain a corresponding result includes: Inputting the concatenated vector into the plurality of sequentially connected encoders for encoding to obtain an encoding vector output by the last layer of encoders; Inputting the encoding vectors into the plurality of decoders connected in sequence respectively; The multiple decoders connected in sequence each decode the encoding vector and the decoding vector output by the connected previous-level encoder and output a decoding vector, wherein the decoding vector output by the decoder of the last layer is the result output by the end-to-end model.

6. The method according to claim 1 or 2, characterized in that: After receiving the text message sent by the vehicle terminal, the method further includes: Get historical session information; The historical session information is input into the application program interface of the end-to-end model, and the application program interface is called to control the end-to-end model to perform a natural language understanding task or a natural language generation task on the input historical session information, the text information and the first vehicle device status information.

7. The method according to claim 1 or 2, characterized in that: After sending the vehicle computer control instruction or the reply text to the vehicle computer end, the method further includes: receiving second vehicle machine device status information sent by the vehicle machine end; the second vehicle machine device status information is used to indicate whether the vehicle machine end successfully executes the result of the vehicle machine control instruction or successfully feeds back the reply text; The text information and the second vehicle-machine device status information are input into the application program interface of the end-to-end model, and the application program interface is called to control the end-to-end model to perform a natural language understanding task or a natural language generation task on the input text information and the second vehicle-machine device status information to obtain corresponding results and feed them back to the vehicle-machine end.

8. A vehicle voice processing device, characterized in that: include: An information acquisition module is used to receive text information sent by the vehicle terminal; The text information is obtained by performing voice recognition on the collected user voice data by the vehicle terminal; a processing module, configured to input the text information into an application program interface of an end-to-end model, and call the application program interface to control the end-to-end model to perform a natural language understanding task or a natural language generation task on the input text information to obtain a corresponding result; wherein, if the natural language understanding task is performed, the result is used to indicate a vehicle control instruction; and if the natural language generation task is performed, the result is used to indicate a reply text; The sending module is used to send the vehicle computer control instruction or the reply text to the vehicle computer end, so that the vehicle computer end executes the vehicle computer control instruction or feeds back the reply text to the user.

9. An electronic device, characterized in that: include: Memory and processor; The memory is configured to store computer program instructions; The processor is configured to execute the computer program instructions so that the electronic device implements the vehicle speech processing method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: include: Computer program instructions; At least one processor of the electronic device executes the computer program instructions so that the electronic device implements the vehicle voice processing method as described in any one of claims 1 to 7.

Images