Intention recognition method, vehicle control method, electronic device and vehicle

Abstract

The invention discloses an intention recognition method, a vehicle control method, an electronic device and a vehicle, and the method comprises the steps: determining a target intention of a user based on a fuzzy intention instruction of the user and multi-mode perception data; visibly, when the user issues the fuzzy intention instruction, the real intention contained in the fuzzy instruction can be comprehensively judged through multi-aspect multi-mode sensing data input, and the intelligent level of the cabin and the user experience are improved.

Description

Technical Field

[0001] This application relates to the technical field of intelligent cockpits, and particularly to an intention recognition method, a vehicle control method, an electronic device, and a vehicle. Background Art

[0002] In the current intelligent voice technology of vehicle intelligent cockpits, the user issues an instruction to the intelligent voice, but does not clearly specify which electrical component needs to be controlled, but describes that the user himself / herself feels uncomfortable with a certain physical sensation. For such fuzzy intention instructions, it is usually difficult for the existing technology to accurately recognize them, or only one or more fixed instructions can be executed for them, which cannot fully meet the user's intention in the current scenario. For example, when the user's instruction is "feeling a bit cold", it can only help the user increase the air-conditioning temperature by 1 or 2 degrees based on the currently set temperature value, unable to recognize the user's true intention, resulting in a poor user experience.

[0003] Application Content

[0004] This application provides an intention recognition method, a vehicle control method, an electronic device, and a vehicle, which can, when the user issues a fuzzy intention instruction, comprehensively determine the true intention contained in the fuzzy instruction through the input of multi-modal perception data from multiple aspects, improving the intelligent level of the cockpit and the user experience.

[0005] To solve the above technical problems, in the first aspect of this application, an intention recognition method is disclosed, and the method includes:

[0006] Based on the user's fuzzy intention instruction and multi-modal perception data, determine the user's target intention.

[0007] As an optional implementation manner, in the first aspect of this application, the multi-modal perception data includes one or more of dialogue data, environmental data, user historical habit data, user status data, and vehicle status data.

[0008] As an optional implementation manner, in the first aspect of this application, the method includes:

[0009] The dialogue data includes one or more of the user's historical voice vehicle control information, historical N-round dialogue information, and current dialogue information;

[0010] The environmental data includes one or more of in-vehicle environmental information and out-of-vehicle environmental information;

[0011] The user historical habit data includes one or more of the user's historical vehicle control habit data and personal preference data;

[0012] The user data includes one or more of the user's clothing, gender, age, expression, mood, and body temperature;

[0013] The vehicle status data includes one or more of vehicle positioning information, vehicle speed information, and vehicle attitude information.

[0014] As an optional implementation, in the first aspect of this application, the fuzzy intent instruction is a fuzzy voice instruction, where determining the target intent of the user based on the user's fuzzy intent instruction and multi-modal perception data includes:

[0015] Determining the target intent of the user based on the fuzzy voice instruction, the multi-modal perception data, and a pre-trained intent analysis model.

[0016] The second aspect of this application discloses a vehicle control method, and the method includes:

[0017] Determining an action instruction set for the vehicle according to the target intent determined by the intent recognition method according to any one of claims 1-4 and vehicle configuration information, so as to control the vehicle to execute the action instruction set.

[0018] As an optional implementation, in the second aspect of this application, the vehicle configuration information includes the configured functions of vehicle electrical appliances and / or the configured status of the vehicle electrical appliances.

[0019] As an optional implementation, in the second aspect of this application, determining the action instruction set for the vehicle according to the target intent and vehicle configuration information includes:

[0020] Determining the action instruction set for the vehicle according to the target intent, the vehicle configuration information, user data, and a pre-trained decision analysis model.

[0021] As an optional implementation, in the second aspect of this application, the method further includes:

[0022] Displaying an action execution list and / or an action execution result to the user.

[0023] The third aspect of this application discloses an electronic device, including a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the method described in any one of the first aspect and the second aspect of this application is implemented.

[0024] The fourth aspect of this application discloses a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program runs, the method described in any one of the first aspect and the second aspect of this application is implemented.

[0025] The fifth aspect of the present application discloses a vehicle, which includes a fault diagnosis system implementing the third aspect of the present application, a controller implementing the fourth aspect of the present application, or a computer-readable storage medium implementing the fifth aspect of the present application.

[0026] The sixth aspect of the present application discloses a computer program product, including a computer program or instruction, and when the computer program or instruction is executed by a processor, it implements the steps of the method according to any one of the first and second aspects of the present application.

[0027] Compared with the prior art, the present application has the following beneficial effects:

[0028] In the present application, based on the user's fuzzy intention instruction and multi-modal perception data, the target intention of the user is determined. It can be seen that when the user issues a fuzzy intention instruction, the present application can comprehensively judge the true intention contained in the fuzzy instruction through the input of multi-modal perception data from multiple aspects, improving the intelligent level of the cockpit and the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0030] Figure 1 is a schematic flowchart of an intention recognition method disclosed according to an embodiment of the present application;

[0031] Figure 2 is a schematic flowchart of another intention recognition method disclosed according to an embodiment of the present application;

[0032] Figure 3 is a schematic flowchart of a vehicle control method disclosed according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] In order to enable those skilled in the art to better understand the solutions of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0034] In the description, claims and the above-mentioned drawings of this application, terms such as "first" and "second" are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or terminal that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or terminals.

[0035] Reference to "embodiment" herein means that a particular feature, structure or characteristic described in connection with the embodiment can be included in at least one embodiment of this application. The phrase appears in various places in the description and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0036] The first aspect of this application discloses an intention recognition method, which will be described in detail below.

[0037] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of an intention recognition method disclosed in an embodiment of this application. The intention recognition method may include the following operations:

[0038] 101. Determine the user's target intention based on the user's fuzzy intention instruction and multimodal perception data.

[0039] The user's fuzzy intention instructions in the embodiments of this application, such as: voice instructions such as "a bit cold", "feeling a bit cold", "the air outside is so bad", "feeling a bit tired", "it's stuffy in the car" collected through a microphone; or fuzzy action instructions such as "the user wiped the sweat on the forehead", "the user made a raising gesture near the car window glass" recognized through a camera. The present invention does not limit the specific instruction form. At the same time, combined with the input of multimodal perception data, comprehensively analyze the specific intention of the fuzzy intention instruction issued by the user in the current scenario to obtain the user's true target intention, improve the intelligent level of the cockpit and the user experience. At the same time, it can lower the threshold for users to use intelligent voice. Users do not need to issue very accurate execution instructions, but only need to express the problems that trouble them in the current scenario, and the user's intention can be accurately recognized.

[0040] In an optional embodiment, the multimodal perception data includes one or more of dialogue data, environmental data, user historical habit data, user status data, and vehicle status data.

[0041] In the embodiments of the present application, through the input of multi-modal perception data of one or more of dialogue data, environmental data, user historical habit data, user status data, and vehicle status data, the fuzzy intention instructions of the user can be analyzed in multiple dimensions and comprehensively to obtain the more accurate target intention of the user.

[0042] In an alternative embodiment, the method further includes:

[0043] The dialogue data includes one or more of the user's historical voice vehicle control information, historical N-round dialogue information, and current dialogue information;

[0044] The environmental data includes one or more of the in-vehicle environmental information and the out-of-vehicle environmental information;

[0045] The user historical habit data includes one or more of the user's historical vehicle control habit data and personal preference data;

[0046] The user data includes one or more of the user's clothing, gender, age, expression, mood, and body temperature;

[0047] The vehicle status data includes one or more of the vehicle positioning information, vehicle speed information, and vehicle attitude information.

[0048] In the embodiments of the present application, the dialogue data may include the user's historical voice control information, such as the voice control instruction information issued by the user to the vehicle within a historical event segment; it may also include the historical N-round dialogue information, such as the context dialogue information within N rounds between the user and other users in the cockpit; it may also include the current dialogue information, such as the user instruction issued by the user most recently. Exemplarily, for example, if the user mentions in the dialogue that they have a cold and then issues an instruction saying "a bit cold", at this time, it is recognized that the user's true intention is "significantly increase the temperature and improve the ventilation quality", and at this time, a corresponding vehicle control instruction is generated to increase the temperature by a large margin and turn on air purification or external circulation. At the same time, based on the above-obtained dialogue data, it can also be comprehensively judged and analyzed whether the intention instruction issued by the user is a fuzzy intention instruction.

[0049] In the embodiments of the present application, the environmental data may include the in-vehicle environmental information, such as the in-vehicle temperature, in-vehicle humidity, in-vehicle illuminance, in-vehicle noise index, in-vehicle air index, etc.; it may also include the out-of-vehicle environmental information, such as the road surface information (snow, mud, sand, road flatness, road gradient, etc.), road condition information (congested, unobstructed, etc.), out-of-vehicle temperature, out-of-vehicle humidity, out-of-vehicle illuminance, out-of-vehicle noise index, out-of-vehicle air index, etc.

[0050] In the embodiments of the present application, the user's historical habit data may include the user's historical vehicle control habit data. For example, User A tends to set the air conditioner temperature at 24 degrees and the first gear, while User B is more inclined to adjust the air conditioner temperature to 20 degrees and the second gear. It may also include personal preference data. For example, when adjusting the temperature, User A prefers it to be colder than most people.

[0051] In the embodiments of the present application, the user data may include one or more of the user's clothing, gender, age, expression, mood, and body temperature. Exemplarily, it can be identified and obtained through sensors in the vehicle or through cloud services. For example, the thickness of the clothing worn by the user, as well as the user's expression and mood, including tired, excited, calm, sad, happy, etc., can be identified using the in-vehicle portrait recognition system; the thermal sensation of the people in the vehicle can be detected using thermal imaging technology (infrared sensor).

[0052] In the embodiments of the present application, the vehicle state data may include vehicle positioning information, and the local weather information can be obtained through the positioning information; it may also include the vehicle speed information, such as the average speed in the current or past historical period; it may also include vehicle attitude information, such as the pitch angle and turning angle of the vehicle.

[0053] It can be seen that through the above more abundant multi-modal perception data, the reference factors for judging the user's intention are further enriched, and the recognition accuracy of the user's target intention is further improved.

[0054] In an alternative embodiment, please refer to Figure 2 , Figure 2 which is a schematic flowchart of another intention recognition method disclosed in the embodiments of the present application. When the fuzzy intention instruction is a fuzzy voice instruction, based on the fuzzy voice instruction, multi-modal perception data, and a pre-trained intention analysis model, the user's target intention is determined.

[0055] In the embodiments of the present application, the user's fuzzy voice instructions such as "a bit cold", "feeling a bit cold", "the air outside is so bad", "feeling a bit tired", "the car is a bit stuffy", etc. are collected through the microphone. Combined with the above-mentioned identified multi-modal perception data and input into the pre-trained intention analysis model, the user's target intention can be obtained. Relying on the powerful generalization ability and learning ability of the intention analysis model, the recognition speed and accuracy of the fuzzy intention instruction are improved.

[0056] The second aspect of the present application discloses a vehicle control method, which will be described in detail below.

[0057] Please refer to Figure 3 , Figure 3 which is a schematic flowchart of a vehicle control method disclosed in the embodiments of the present application. The vehicle control method may include the following operations:

[0058] Determine the action instruction set of the vehicle based on the target intention of the user and the configuration information of the vehicle determined by the method steps of the first aspect above, so as to control the vehicle to execute the action instruction set.

[0059] In the embodiments of the present application, the configuration information of the vehicle may specifically refer to which electrical component functions the vehicle supports for control, etc. The specific method can be implemented according to the vehicle's information acquisition protocol. For example, the vehicle actively reports the configuration list through a message, or reports according to the vehicle brand, or reports according to a certain vehicle model ID or vehicle identification number method, etc., and queries through the cloud. For example, whether the vehicle's windows and sunroof support electrical control, whether the air conditioner supports switching between internal and external circulation, etc. According to the target intention of the user and the configuration information of the vehicle, it can be comprehensively analyzed what instructions the vehicle can execute to match the target intention of the user. Exemplarily, for example, the user says "feeling a bit tired". At this time, if the seat on which the user is sitting is equipped with a seat massage function, the seat massage function can be activated according to this configuration to massage the user, and at the same time, the user data can be matched to massage in the most suitable or favorite massage method; if the seat on which the user is sitting does not have a seat massage function at this time, the front-back and backrest positions of the seat can be adjusted according to the space state inside the vehicle to match the most suitable sitting posture of the user for relaxation and rest.

[0060] It can be seen that the method described in the embodiments of the present application can, based on the true intention of the user and the configuration information of the current vehicle, determine the action instruction set most suitable for this true intention, so that when the same instruction is issued in different scenarios, it can help the user execute a series of functional actions most in line with the current scenario, further improving the intelligent level and user experience of the cockpit, and at the same time enabling the user to be more focused on driving and improving driving safety.

[0061] In an alternative embodiment, the vehicle configuration information includes the configuration functions of vehicle electrical appliances and / or the configuration states of vehicle electrical appliances.

[0062] In the embodiments of the present application, the vehicle configuration information may include the configuration functions of vehicle electrical appliances, such as but not limited to windows, sunroofs, air conditioners (temperature, air volume, AC, internal and external circulation, defrosting mode, ventilation mode, etc.), seat massage, seat heating, seat ventilation, windshield wipers, rearview mirror heating, etc., and the specific configuration information of each seat; it may also include the configuration states of vehicle electrical appliances, such as the opening degree of the windows and the angle of the seats. Exemplarily, for example, the user's instruction is "the air feels very bad". If the current windows are in the open state and the pm2.5 value of the outdoor air is poor, then help the user execute the operation of closing all windows and the sunroof, switching the air conditioner to internal circulation and air purification, and turning on the air conditioner to an appropriate temperature; if the current windows and sunroof are not opened and the outdoor air instruction is good, then open the windows for ventilation for a period of time, then close the windows and turn on the air conditioner to external circulation and adjust the temperature to a suitable temperature.

[0063] It can be seen that the method described in the embodiments of the present application can further determine an action instruction that better conforms to the current scenario based on the configuration function and / or configuration state of the vehicle to meet the real needs of users.

[0064] In an optional embodiment, the vehicle control method of the present application can determine an action instruction set of the vehicle according to the target intention, vehicle configuration information, and a pre-trained decision analysis model.

[0065] In the embodiments of the present application, the target intention and vehicle configuration information can be input into a pre-trained decision analysis model, and the action instruction set that the vehicle can execute can be directly obtained through model analysis. Relying on the powerful generalization ability and learning ability of the decision analysis model, the determination speed and accuracy of the action instruction are improved. In addition, the input of the decision analysis model can also include user data, that is, when generating the action instruction, one or more pieces of information such as the user's clothing, gender, age, expression, emotion, and body temperature can also be referred to to formulate a customized action instruction for the user.

[0066] It should be noted that the model algorithms involved in the present application can be trained based on limited labeled data. Since the initial amount of labeled data may not be large, the following methods can be used to optimize the model performance: 1) Data augmentation: Increase the diversity of data, such as slight transformations of facial features and action features; 2) Transfer learning: Use a feature extraction network pre-trained on a large dataset (such as VGG, ResNet, etc.) to extract facial and action features; 3) Cross-validation: Adopt k-fold cross-validation to better evaluate the performance of the model.

[0067] The model algorithm processes the above input features (such as fuzzy intention instructions, multi-modal perception data, vehicle configuration information, etc.). The model can be a multi-layer perceptron (MLP), or a model combining a convolutional neural network (CNN) or a recurrent neural network (RNN), depending on the characteristics of the input data. The process of model pre-training is: determine the requirements → input limited data to train the model → test the model with a test set → correct the execution results and adjust the model parameters → enter the next round of testing. During the process of putting the model algorithm into the production environment after training, more executions and feedback are continuously generated and the parameters are adjusted, so that the model becomes more and more mature, and an algorithm model that can accurately analyze the user's intention and generate an execution list is gradually formed.

[0068] In an optional embodiment, the vehicle control method of the present application can also display an action execution list and / or an action execution result to the user. That is, it responds to the user's ambiguous intent instruction, so that the user can intuitively see or feel which vehicle configurations are involved in the processing of the current ambiguous intent instruction and what the execution result is. In addition, if the user is not satisfied with the execution of this action, they can continue to issue a new instruction to modify or adjust it. Exemplarily, after receiving the execution result feedback from the system, the user can issue an instruction to feedback their personal satisfaction with the execution result to the system, such as "turn up the temperature more", "the seat is too hot", etc. After receiving the feedback, the system executes it for the user in real time and records it as the user's preference habit and feeds it back to the algorithm model for reference in the next intent judgment.

[0069] It can be seen that through this method, the ambiguous intent instruction can be responded to in an intuitive way, further improving the user experience.

[0070] The third aspect of the present application discloses an electronic device, including a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, it implements the method steps as described in the first aspect and the second aspect above, and has all the above beneficial effects, which will not be elaborated herein.

[0071] The fourth aspect of the present application discloses a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program runs, it implements the method steps as described in the first aspect and the second aspect above, and has all the above beneficial effects, which will not be elaborated herein.

[0072] The fifth aspect of the present application discloses a vehicle, including the electronic device as described in the third aspect above or the computer-readable storage medium as described in the fourth aspect above. The vehicle has all the above beneficial effects, which will not be elaborated herein.

[0073] The sixth aspect of the present application discloses a computer program product, including a computer program or instruction. When the computer program or instruction is executed by a processor, it implements the method steps as described in the first aspect and the second aspect above, and has all the above beneficial effects, which will not be elaborated herein.

[0074] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative effort.

[0075] Through the specific descriptions of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, and the storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disk memories, tape memories, or any other medium that can be used to carry or store data and is computer-readable.

[0076] It should be noted that the computer program codes required for the operations of each part of this specification can be written in any one or more programming languages, including object-oriented programming languages such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages such as C language, Visual Basic, Fortran 2003, Perl, COBOL 2002, PHP, ABAP, dynamic programming languages such as Python, Ruby, and Groovy, or other programming languages. The program code can run entirely on a computer (PC, embedded intelligent device, etc.), or run as an independent software package on a user's computer, or partially run on the user's computer and partially run on a remote computer, or run entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer in any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0077] Finally, it should be noted that: the intention recognition method, vehicle control method, electronic device, and vehicle disclosed in the embodiments of this application are only the preferred embodiments of this application, and are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for intention recognition, characterized in that, The method includes: Based on the user's fuzzy intent instruction and multi-modal perception data, determine the user's target intent.

2. The intention recognition method according to claim 1, characterized in that, The multi-modal perception data includes one or more of conversation data, environmental data, user historical habit data, user status data, and vehicle status data.

3. The intention recognition method according to claim 2, characterized in that, The method includes: The conversation data includes one or more of the user's historical voice vehicle control information, historical N-round conversation information, and current conversation information; The environmental data includes one or more of in-vehicle environmental information and out-of-vehicle environmental information; The user historical habit data includes one or more of the user's historical vehicle control habit data and personal preference data; The user data includes one or more of the user's clothing, gender, age, expression, emotion, and body temperature; The vehicle status data includes one or more of vehicle positioning information, vehicle speed information, and vehicle attitude information.

4. The intention recognition method according to any one of claims 1-3, characterized in that, The fuzzy intent instruction is a fuzzy voice instruction. Among them, based on the user's fuzzy intent instruction and multi-modal perception data, determining the user's target intent includes: Based on the fuzzy voice instruction, the multi-modal perception data, and a pre-trained intent analysis model, determine the user's target intent.

5. A vehicle control method, characterized in that, The method includes: Based on the target intent determined by the intent recognition method according to any one of claims 1-4 and vehicle configuration information, determine an action instruction set for the vehicle to control the vehicle to execute the action instruction set.

6. The vehicle control method according to claim 5, wherein The vehicle configuration information includes the configured functions of vehicle electrical appliances and / or the configured status of the vehicle electrical appliances.

7. The vehicle control method according to claim 5 or 6, characterized in that, Based on the target intent and vehicle configuration information, determining the action instruction set for the vehicle includes: Based on the target intent, the vehicle configuration information, user data, and a pre-trained decision analysis model, determine the action instruction set for the vehicle.

8. The vehicle control method according to claim 5, characterized in that, The method further includes: Display an action execution list and / or action execution result to the user.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the method according to any one of claims 1-8 is implemented.

10. A computer-readable storage medium, characterized in that, A computer program or instruction is stored in the computer-readable storage medium. When the computer program runs, the method according to any one of claims 1-8 is implemented.

11. A vehicle, characterized in that, It includes the electronic device according to claim 9 or the computer-readable storage medium according to claim 10.

12. A computer program product, characterized in that, It includes a computer program or instruction. When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images