Interaction method and device for automatic driving, electronic equipment and vehicle

Abstract

The invention relates to an automatic driving interaction method and device, electronic equipment and a vehicle, and relates to the technical field of automatic driving, and the method comprises the steps: obtaining interaction control information inputted by a user, and obtaining automatic driving intervention information by recognizing the interaction control information, the automatic driving intervention information represents at least local intervention on the current automatic driving behavior; generating an automatic driving intervention instruction corresponding to the automatic driving intervention information; judging whether the current driving environment information supports safe execution of the automatic driving intervention instruction or not; if it is judged that the driving environment information supports safe execution of the automatic driving intervention instruction, the automatic driving intervention instruction is executed under the condition that the automatic driving process is not terminated. Through the scheme disclosed by the invention, the user can safely and flexibly change the local behavior of automatic driving through a convenient and rapid natural interaction mode on the premise of not terminating the current automatic driving plan.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an interactive method, device, electronic equipment and vehicle for autonomous driving. Background Art

[0002] As cars become more intelligent, autonomous driving is gradually being used. At this stage, autonomous driving can be used legally on some roads. As autonomous driving technology matures, highly autonomous driving will also be used. Users will rely more on autonomous driving to complete driving tasks.

[0003] Currently, during the process of vehicle autonomous driving, if the user needs to change the state and behavior of autonomous driving, the user can only operate through operations such as controlling the steering wheel and braking. However, this will terminate the current autonomous driving process. If autonomous driving needs to be restored, a new autonomous driving plan needs to be reset, which affects the user's car experience. Summary of the invention

[0004] In view of this, the present disclosure provides an interactive method, device, electronic device and vehicle for autonomous driving, the main purpose of which is to solve the technical problem that it is currently difficult for users to use convenient and fast interactions to safely and flexibly change the local behavior of autonomous driving.

[0005] According to a first aspect of the present disclosure, there is provided an interactive method for autonomous driving, the method comprising:

[0006] Acquire interactive control information input by a user, and obtain autonomous driving intervention information by identifying the interactive control information, wherein the autonomous driving intervention information represents at least partial intervention in a current autonomous driving behavior;

[0007] generating an autonomous driving intervention instruction corresponding to the autonomous driving intervention information;

[0008] Determining whether the current driving environment information supports the safe execution of the autonomous driving intervention instruction;

[0009] If it is determined that the driving environment information supports the safe execution of the autonomous driving intervention instruction, the autonomous driving intervention instruction is executed without terminating the autonomous driving process.

[0010] According to a second aspect of the present disclosure, there is provided an interactive device for autonomous driving, the device comprising:

[0011] a receiving module, configured to obtain interactive control information input by a user, and obtain autonomous driving intervention information by identifying the interactive control information, wherein the autonomous driving intervention information represents at least partial intervention in a current autonomous driving behavior;

[0012] A generating module, configured to generate an automatic driving intervention instruction corresponding to the automatic driving intervention information;

[0013] A determination module, used to determine whether the current driving environment information supports the safe execution of the automatic driving intervention instruction;

[0014] An execution module is used to execute the autonomous driving intervention instruction without terminating the autonomous driving process if it is determined that the driving environment information supports the safe execution of the autonomous driving intervention instruction.

[0015] According to the third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of the aforementioned first aspect.

[0016] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method of the aforementioned first aspect.

[0017] According to the fifth aspect of the present disclosure, a vehicle is provided, comprising at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described in the first aspect above.

[0018] The disclosure provides an interactive method, device, electronic device and vehicle for autonomous driving. Compared with the prior art, the disclosure first obtains interactive control information input by the user, and obtains autonomous driving intervention information by identifying the interactive control information, wherein the autonomous driving intervention information represents at least partial intervention on the current autonomous driving behavior; then generates an autonomous driving intervention instruction corresponding to the autonomous driving intervention information; then determines whether the current driving environment information supports the safe execution of the autonomous driving intervention instruction; if it is determined that the driving environment information supports the safe execution of the autonomous driving intervention instruction, the autonomous driving intervention instruction is executed without terminating the autonomous driving process. By applying the scheme of the disclosure, the user can issue an intervention instruction for autonomous driving through convenient and fast interactive control. If the current driving environment information supports the safe execution of the autonomous driving intervention instruction, the autonomous driving intervention instruction can be executed to implement the intervention on the local behavior of the autonomous driving without terminating the autonomous driving process to change the local behavior of the autonomous driving, thereby realizing the safe and flexible change of the local behavior of the autonomous driving through a convenient and fast natural interactive method, which can improve the user's car experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 A schematic diagram of a flow chart of an interactive method for autonomous driving provided in an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of a flow chart of an interactive method for autonomous driving provided by another embodiment of the present disclosure;

[0023] Figure 3 An intervention scenario timing diagram for changing an autonomous driving path under a map path planning change provided in an embodiment of the present disclosure;

[0024] Figure 4 A timing diagram of an intervention scenario that does not change the autonomous driving path provided by an embodiment of the present disclosure;

[0025] Figure 5 A timing diagram of an intervention scenario for changing an autonomous driving path provided by an embodiment of the present disclosure;

[0026] Figure 6 A timing diagram of an intervention scenario for changing an autonomous driving path under multi-modal interactive parallel verification provided by an embodiment of the present disclosure;

[0027] Figure 7 A principle flow chart of an interactive method for autonomous driving provided in an embodiment of the present disclosure;

[0028] Figure 8 A schematic diagram of the structure of an interactive device for autonomous driving provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments may be combined with each other.

[0030] The following describes the interactive method, device, electronic device and vehicle for autonomous driving according to the embodiments of the present disclosure with reference to the accompanying drawings.

[0031] The present disclosure provides an interactive method, device, electronic device and vehicle for autonomous driving, which enable users to safely and flexibly change or intervene in autonomous driving behavior through a convenient and fast natural interaction method without terminating the current autonomous driving plan when intervening in the autonomous driving behavior.

[0032] like Figure 1 As shown, an embodiment of the present disclosure provides an interactive method for autonomous driving, including:

[0033] Step 101: Obtain interactive control information input by a user, and obtain automatic driving intervention information by identifying the interactive control information.

[0034] Among them, the interactive control information can be one or more of voice control information, touch information, limb control information, gesture control information, etc.; the autonomous driving intervention information represents at least partial intervention in the current autonomous driving behavior.

[0035] This embodiment can identify the automatic driving intervention information that the user needs to intervene in the automatic driving behavior by identifying the interactive control information, and the automatic driving intervention information can be information used to change the automatic driving behavior operation. For example, by identifying the user's interactive control information, if it is determined that the user expects to listen to music, watch videos, etc., it can be determined that the automatic driving intervention information is not identified, and if the user expects to change the driving route of the automatic driving, or accelerate, or accelerate, etc., it can be determined that the automatic driving intervention information is identified.

[0036] The execution subject of the embodiment of the present disclosure may be an interactive device or equipment for autonomous driving, which may be configured on the vehicle side or the domain controller side, etc.

[0037] Step 102: Generate an autonomous driving intervention instruction corresponding to the autonomous driving intervention information.

[0038] For the disclosed embodiments, different autonomous driving intervention information corresponds to different autonomous driving intervention instructions, which can be used to indicate the target driving path of the vehicle and can trigger autonomous driving driving operations such as acceleration, deceleration, merging, left turn, right turn, etc. By generating autonomous driving intervention instructions corresponding to the autonomous driving intervention information, the autonomous driving driving behavior can be partially intervened or changed.

[0039] Step 103: Determine whether the current driving environment information supports the safe execution of the automatic driving intervention instruction.

[0040] The driving environment information may include: weather information, and / or road information, and / or surrounding vehicle information, etc.

[0041] With respect to the disclosed embodiments, by determining whether the current driving environment information supports the safe execution of the autonomous driving intervention instruction, and executing the instruction only when it is determined that the driving environment information supports the safe execution of the autonomous driving intervention instruction, the safety of the vehicle driving process can be improved, and the user can safely and flexibly change or intervene in the autonomous driving behavior through a convenient and fast natural interaction method without terminating the current autonomous driving plan.

[0042] Step 104: If it is determined that the driving environment information supports the safe execution of the autonomous driving intervention instruction, the autonomous driving intervention instruction is executed without terminating the autonomous driving process.

[0043] By applying the technical solution of this embodiment, the user can issue intervention instructions for autonomous driving through convenient and fast interactive control. If the current driving environment information supports the safe execution of the autonomous driving intervention instruction, the autonomous driving intervention instruction can be executed to intervene in the local behavior of the autonomous driving to change the local behavior of the autonomous driving without terminating the autonomous driving process. This can further improve the user experience of the car by safely and flexibly changing the local behavior of the autonomous driving through convenient and fast natural interaction.

[0044] Further, as a refinement and extension of the above embodiment, based on the solution of this embodiment, an interactive system may be provided, which may include but not limited to a cockpit interaction engine module, a map navigation module and an automatic driving engine module, wherein the cockpit interaction engine module may include but not limited to a natural interaction engine module and a graphical user interface (GUI) interaction engine module. The natural interaction engine module includes one or more processing modules, such as a voice processing module, a text processing module, a gesture processing module, a multimodal fusion processing module, etc. The natural interaction engine module is used to receive the automatic driving intervention information input by the user in the car, and to identify the automatic driving intervention information, and convert it into an automatic driving intervention instruction for the user in the car to change the automatic driving behavior. The graphical user interface interaction engine module is used to process the touch information input by the user in the car, and convert the touch information into an automatic driving intervention instruction for the user in the car to change the automatic driving behavior. The graphical user interface interaction engine module may include but not limited to a screen touch screen interaction module or a physical button interaction module. The map navigation module is used to plan and manage the path for the automatic driving intervention instruction of the user in the car, and the path information will be synchronized to the automatic driving engine module, while receiving the navigation or path change information from the cockpit interaction engine module, and initiating corresponding processing. The autonomous driving engine module is used to process the driving behavior of the vehicle during the autonomous driving process. The autonomous driving engine module may include but is not limited to an environmental perception module, a safety decision-making module, a driving first execution module, a path management module, etc.

[0045] Based on the above interactive system, in order to fully illustrate the specific implementation process of the method of this embodiment, the cockpit interaction engine module is used as the execution body, and this embodiment provides the following Figure 2 The specific method shown includes:

[0046] Step 201: Acquire interactive control information input by a user, and obtain user intention information by identifying the interactive control information.

[0047] Step 202: Based on the user intention information, determine whether the autonomous driving behavior needs to be changed.

[0048] Step 203: If it is determined that the autonomous driving behavior needs to be changed, the intention description information corresponding to the user intention information is used as the autonomous driving intervention information.

[0049] For the disclosed embodiment, in order to prevent the erroneous reception of the autonomous driving intervention information of the user in the car and the generation of the erroneous autonomous driving intervention instruction, it is necessary to identify the interactive control information input by the user, determine whether the interactive control information needs to change the autonomous driving behavior, and if it is determined that the autonomous driving behavior needs to be changed, the intention description information corresponding to the user intention information is used as the autonomous driving intervention information. Through this optional method, the autonomous driving intervention information can be accurately identified.

[0050] For example, the user intention information can be obtained by identifying the interactive control information. If the user intention information is within the preset intention information corresponding to the change of the automatic driving, it is determined that the automatic driving behavior needs to be changed, and the intention description information corresponding to the user intention information is used as the automatic driving intervention information. Among them, the user intention information is used to determine whether the user needs to change the automatic driving behavior; the preset intention information may include but is not limited to the indication of acceleration, deceleration, merging to the right, reducing or increasing the distance with the vehicle in front, etc. without changing the current automatic driving path planning, and may also include the intention information of changing the current automatic driving path planning such as going straight at the next intersection, turning left at the next intersection, turning right at the next intersection, getting off the highway in advance at the next intersection, and changing the route to the nearest gas station.

[0051] Step 204: Generate an autonomous driving intervention instruction corresponding to the autonomous driving intervention information.

[0052] As an optional method, step 204 may specifically include: when it is determined by identifying the intention description information that the user needs to change the autonomous driving driving path and no new driving path is indicated, obtaining at least one new path plan to reach the destination from the map navigation; outputting a first prompt message, wherein the first prompt message is used to prompt the selection of one of the at least one new path plan to execute; and subsequently generating an autonomous driving intervention instruction corresponding to the target path plan in response to the target path plan selected by the user from the at least one new path plan.

[0053] As another optional method, step 204 may specifically include: if it is determined that the driving environment information supports the safe execution of the autonomous driving intervention instruction, the target path planning is synchronized to the autonomous driving engine, and the autonomous driving engine is used to execute the autonomous driving task corresponding to the target path planning.

[0054] For the embodiments of the present disclosure, when it is determined based on the intention description information that the user needs to change the driving path of the autonomous driving and no new driving path is indicated, at least one new path plan to reach the destination is obtained; first prompt information is output, and the first prompt information is used to prompt the selection of one of the at least one new path plan to execute; in response to the target path plan selected by the user from the at least one new path plan, an autonomous driving intervention instruction corresponding to the target path plan is generated.

[0055] For example, Figure 3 As shown, when the current vehicle is in the automatic driving (or assisted driving) state, the user (Actor) in the car wants to change the automatic driving route during driving, for example, the user wants to change the default driving route and take another road to reach the same destination. At this time, the user in the car inputs the command of "change another road" or "don't take this road" through voice. The cockpit interaction engine module recognizes the automatic driving intervention information through voice, and recognizes that the user in the car has an indication to switch the driving route and has not indicated a new driving route. The cockpit interaction engine module then asks the map navigation module interface and displays several new route plans to the destination. The user in the car selects one of them as the target route plan based on the feedback after the new route plan, and generates an automatic driving intervention instruction corresponding to the target route plan according to the target route plan.

[0056] After confirmation, the map navigation module synchronizes the new route planning to the autonomous driving engine module, which performs a safety check on the new route planning and executes the change to switch to the new route after passing the check. The autonomous driving module feeds back the successful execution result to the map navigation module and the cockpit interaction engine module, and the cockpit interaction engine module feeds back the execution result to the user in the car.

[0057] As another optional method, step 204 may specifically include: when it is determined by identifying the intention description information that the user needs to change the autonomous driving driving path and indicates a new driving path, generating an autonomous driving intervention instruction corresponding to the new driving path indicated by the user.

[0058] For the embodiment of the present disclosure, when it is determined based on the intention description information that the user needs to change the driving path of the autonomous driving and indicates a new driving path, an autonomous driving intervention instruction corresponding to the new driving path indicated by the user is generated.

[0059] In a specific application scenario, the current vehicle is in automatic driving (or assisted driving) state, and the user in the car (Actor) wants to change lanes because of the slow-moving vehicle in front blocking the lane. At this time, the user in the car inputs the intention description information of "merge right" through voice, and it is judged that the user needs to change the automatic driving path and indicates a new driving path, then an automatic driving intervention instruction corresponding to the new driving path indicated by the user is generated.

[0060] Step 205: determine whether the current driving environment information supports the safe execution of the autonomous driving intervention instruction. If it is determined that the driving environment information supports the safe execution of the autonomous driving intervention instruction, execute the autonomous driving intervention instruction to intervene in the local behavior of the autonomous driving without terminating the autonomous driving process.

[0061] For the disclosed embodiment, before determining whether the current driving environment information supports the safe execution of the autonomous driving intervention instruction, it is first necessary to determine whether the new driving path indicated by the user will change the current path planning. If it is determined that the new driving path indicated by the user will change the current path planning, the map path planning of the autonomous driving is obtained from the map navigation; and it is determined whether the execution of the autonomous driving intervention instruction will change the map path planning.

[0062] As a possible solution, when the new driving path indicated by the user will change the current path planning, if the target driving path indicated by the autonomous driving intervention instruction is consistent with the map path planning, it is determined that the autonomous driving intervention instruction does not change the current autonomous driving path planning; if the target driving path indicated by the autonomous driving intervention instruction is inconsistent with the map path planning, it is determined that the autonomous driving intervention instruction changes the current autonomous driving path planning; when it is determined that executing the autonomous driving intervention instruction will change the map path planning, a second prompt message is output, and the second prompt message is used to ask the user whether to confirm the execution of the autonomous driving intervention instruction and change the current path planning.

[0063] After receiving feedback information sent by the user confirming the execution of the autonomous driving intervention instruction and the change of the current path planning, it is first necessary to determine whether the instruction contents of the multiple generated autonomous driving intervention instructions are consistent. If the instruction contents of the multiple autonomous driving intervention instructions are inconsistent, a third prompt information is output, and the third prompt information is used to prompt the selection of one of the multiple autonomous driving intervention instructions for execution; in response to a target intervention instruction selected from multiple autonomous driving intervention instructions with inconsistent instruction contents, determine whether the current driving environment information supports the safe execution of the target intervention instruction; or, if the instruction contents of multiple autonomous driving intervention instructions are consistent, merge the multiple autonomous driving intervention instructions into one intervention instruction, and determine whether the current driving environment information supports the safe execution of the merged intervention instruction.

[0064] As a possible implementation method, it is determined whether the automatic driving intervention instruction satisfies the verification conditions, wherein the purpose of determining whether the verification conditions are satisfied is to ensure that the user in the vehicle really wants to execute the automatic driving intervention instruction and change the path planning, and the verification conditions include: when the automatic driving intervention instruction includes at least one of voice instructions, gesture instructions, text instructions and touch instructions, playing the voice inquiry content of whether to execute the automatic driving intervention instruction, and receiving a confirmation change reply to the voice inquiry content within a preset time; or, when the automatic driving intervention instruction includes at least two of the voice instructions, gesture instructions, text instructions and touch instructions, the indication of at least two automatic driving intervention instructions Figure 1 The different automatic driving intervention instructions correspond to different expressions of the indication diagram. If the verification conditions are met, it is determined that the automatic driving intervention instruction is accurately recognized, and the path planning corresponding to the current automatic driving in the map navigation module is adjusted to the target path planning corresponding to the automatic driving intervention instruction, and the automatic driving is executed according to the target path planning when the safety decision-making judgment environment information supports the safe execution of the automatic driving intervention instruction under the current path position; if the verification conditions are not met, it is determined that the automatic driving intervention instruction is not accurately recognized, and the subsequent operations are stopped.

[0065] Determine whether the current driving environment information supports the safe execution of the autonomous driving intervention command. Specifically, based on the current weather information, and / or road information, and / or surrounding vehicle information, determine whether the execution of the autonomous driving intervention command will cause driving risks; if the execution of the autonomous driving intervention command will not cause driving risks, then it is determined that the driving environment information supports the safe execution of the autonomous driving intervention command.

[0066] In a specific application scenario, the driving environment information around the vehicle can be used to determine whether the execution of the automatic driving intervention command will cause driving risks. The driving environment information may include the vehicle's current road surface information, weather information, surrounding building information, surrounding vehicle information, surrounding pedestrian information and other vehicle surrounding environment information. The vehicle's road surface information, weather information, surrounding building information, surrounding vehicle information, surrounding pedestrian information and other vehicle surrounding environment information (which may be image information) currently collected are further calculated for similarity with the map information in the map navigation module, and the current position of the vehicle is located according to the similarity calculation result. The automatic driving safety decision can be used to determine whether the driving environment information supports the safe execution of the automatic driving intervention command at the current path position. For example, the automatic driving intervention command is "merge right". If the automatic driving safety decision determines that there is a vehicle on the right side of the vehicle's current path position, the "merge right" command will not be executed; if the automatic driving safety decision determines that there is no vehicle, pedestrian or other building that hinders the "merge right" on the right side of the vehicle's current path position, the "merge right" command will be executed.

[0067] In specific application scenarios, such as Figure 4 As shown, the current vehicle is in the automatic driving (or assisted driving) state, and the user (Actor) in the car wants to change lanes because of the slow-moving car in front. At this time, the user in the car inputs the command of "merge right" through voice. The cockpit interaction engine module recognizes the command through the command, and determines that the automatic driving intervention command will not change the current automatic driving path planning, so the cockpit interaction engine module sends the command of "merge right" to the automatic driving engine module. The automatic driving engine determines through the safety decision module that the driving environment information does not hinder the execution of the command of "merge right" under the current path position, and then uses the first driving execution module to execute the command of "merge right". After the execution is completed, the receipt information and execution result are fed back to the cockpit interaction engine module, wherein the receipt information is used to confirm whether the first driving execution module executes the command of "merge right". The cockpit interaction engine module feeds back the execution completion status of "merge right" to the user in the car through the graphical interface or voice, and refreshes the path planning.

[0068] For the disclosed embodiment, when the automatic driving intervention instruction is identified to change the path planning of the current automatic driving, it can be determined whether the target path planning corresponding to the automatic driving intervention instruction is consistent with the path planning corresponding to the current automatic driving. If they are not consistent, in order to avoid the wrong recognition of the automatic driving intervention instruction, the cockpit interaction engine module initiates a secondary inquiry to the user in the vehicle to ensure that the user really wants to execute the automatic driving intervention instruction and change the path planning. If the user in the vehicle responds to confirm the execution of the automatic driving intervention instruction and change the path planning, the path planning corresponding to the current automatic driving in the map navigation module is adjusted to the target path planning corresponding to the automatic driving intervention instruction, and the cockpit interaction engine module sends the automatic driving intervention instruction to the automatic driving engine module. The safety decision module in the automatic driving engine module determines whether the driving environment information of the current vehicle hinders the execution of the automatic driving intervention instruction at the current path position. If it does not hinder, the automatic driving engine module calls the first driving execution module to execute the driving action corresponding to the automatic driving intervention instruction, and feedbacks the receipt of starting execution and execution completion to the cockpit interaction engine module. After the driving action corresponding to the autonomous driving intervention instruction is executed, the map navigation module locates that the current path has changed, starts new path planning, and synchronizes the new path information to the autonomous driving engine module and the cockpit interaction engine module. The cockpit interaction engine module synchronizes the new path change to the user in the car.

[0069] In specific application scenarios, such as Figure 5 As shown in the figure, when the current vehicle is in the state of automatic driving (or assisted driving), the user in the car (Actor) suddenly wants to change the itinerary. The user in the car inputs the command "turn right at the intersection ahead" through voice. The cockpit interaction engine module recognizes the command through the command, and uses the path planning decision to determine that the automatic driving intervention command will change the current automatic driving path planning. Therefore, the cockpit interaction engine module asks the map navigation module interface whether to change the path planning for turning right at the intersection ahead. After executing the path comparison, the map navigation module replies that the new command will change the path planning.

[0070] The cockpit interaction engine module initiates a secondary inquiry to the user in the car to ensure that the user really wants to execute the automatic driving intervention command and change the path planning. If the user in the car responds to confirm the execution of the automatic driving intervention command and change the path planning, the path planning corresponding to the current automatic driving in the map navigation module is adjusted to the target path planning corresponding to the automatic driving intervention command, and the cockpit interaction engine module sends the "turn right at the intersection ahead" command to the automatic driving engine module. When the safety decision module in the automatic driving engine module determines that the current vehicle's environmental information does not hinder the execution of the driving action corresponding to the automatic driving intervention command under the current path position, the first driving execution module is called to execute the "turn right at the intersection ahead" command, and feedback to the cockpit interaction engine module to start the execution and the receipt of the completion of the execution. When the right turn is completed, the map navigation module locates that the current path has changed, starts the new path planning, and synchronizes the new path information to the automatic driving engine and the cockpit interaction engine. The cockpit interaction engine module synchronizes the new path change to the user in the car.

[0071] In specific application scenarios, such as Figure 6 As shown, when the user (Actor) in the car sends two or more autonomous driving intervention instructions, when the current vehicle passes the navigation and is in the autonomous driving (or assisted driving) state, the user in the car suddenly wants to change the itinerary. The user in the car inputs the command "turn right at the intersection ahead" through voice, and points his finger to the right. The cockpit interaction engine module recognizes the command through the command, and the gesture processing module can recognize the gesture action of the user in the car pointing to the right through the camera in the car, and use the path planning decision to judge that the autonomous driving intervention command will change the current autonomous driving path planning.

[0072] Therefore, the cockpit interaction engine module asks the map navigation module interface whether to change the route planning for the right turn at the intersection ahead. After the map navigation module performs the route comparison, it returns the new instruction and changes the route planning. Since the user in the car has a clear voice command and issues a gesture command that is consistent with it, the multimodal fusion processing module of the cockpit interaction engine module determines that the common intention of the voice command and the gesture action is to turn right. After double verification of the voice command and the gesture command, there is no need for secondary confirmation with the user. Here, the multimodal verification of the voice command and the gesture command in parallel actually achieves the effect of secondary confirmation, thereby reducing the command misrecognition rate.

[0073] The path planning corresponding to the current automatic driving in the map navigation module is adjusted to the target path planning corresponding to the automatic driving intervention instruction, and the cockpit interaction engine module sends the instruction of "turn right at the intersection ahead" to the automatic driving engine module. When the safety decision module in the automatic driving engine module determines that the environmental information of the current vehicle does not hinder the execution of the driving action corresponding to the automatic driving intervention instruction at the current path position, the first driving execution module is called to execute the instruction of "turn right at the intersection ahead", and the receipt of starting execution and completion of execution is fed back to the cockpit interaction engine module. When the right turn is completed, the map navigation module locates that the current path has changed, starts the new path planning, and synchronizes the new path information to the automatic driving engine and the cockpit interaction engine. The cockpit interaction engine module synchronizes the new path change to the user in the car.

[0074] Among them, the multi-modal fusion recognition method can more accurately understand the intentions of the users in the car. By verifying the intentions of voice and gestures, the overall recognition accuracy can be improved, reducing the subsequent operations that require confirmation from the users in the car.

[0075] For the embodiments of the present disclosure, Figure 7 As shown, the in-vehicle user (Actor) inputs information 1 (autonomous driving intervention information) through the human-computer interaction interface. If information 1 is voice information, the corresponding voice processing module processes information 1; if information 1 is gesture information, the corresponding gesture processing module processes information 1; if information 1 is text information, the corresponding text processing module processes information 1; if information 1 is multiple intervention information, such as text information and voice information, the corresponding multimodal fusion processing module processes information 1; if information 1 is touch information, the corresponding graphical user interface (Graphical User Interface, GUI) interaction engine module processes information 1.

[0076] The cockpit interaction engine module identifies the autonomous driving intervention information, generates the autonomous driving intervention instruction corresponding to the autonomous driving intervention information, and sends information 2 to the map navigation module. Information 2 is used to confirm whether the autonomous driving intervention instruction changes the path planning of the map navigation module. If it is determined to be changed, the cockpit interaction engine module needs to initiate a secondary confirmation to the user in the car. After the user in the car confirms, information 3 is sent to the autonomous driving engine module. Information 3 includes the user in the car's intention to change the autonomous driving (or assisted driving), specific actions and corresponding parameter information.

[0077] If the autonomous driving intervention instruction does not change the path planning of the map navigation module, information 3 is sent to the autonomous driving engine module. Information 3 contains the intention, specific actions and corresponding parameter information of the user in the car not to change the path planning of the map navigation module. After receiving the instruction intention of information 3, the autonomous driving engine module combines the environmental information output by the environmental perception module and the current path position output by the path management module, and determines whether to execute the autonomous driving intervention instruction through the safety decision module. If the judgment result is execution, the driving execution module is called to operate the driving action corresponding to the autonomous driving intervention instruction, and the execution result is output as information 3.1. If the judgment result is not to execute, the judgment result and reason are output as information 3.1. After receiving information 3.1, the cockpit interaction engine module feeds back the execution result of the instruction to change the current driving behavior as information 4 to the user in the car.

[0078] In the disclosed embodiment, the execution subject may be an interactive device for autonomous driving, and the interactive device for autonomous driving may include but is not limited to a cockpit interaction engine module, a map navigation module and an autonomous driving engine module, wherein the cockpit interaction engine module may include but is not limited to a natural interaction engine module and a graphical user interface (GUI) interaction engine module. The natural interaction engine module includes one or more processing modules, such as a voice processing module, a text processing module, a gesture processing module, a multimodal fusion processing module, etc. The natural interaction engine module is used to receive the interactive control information input by the user in the car, and to perform information recognition on the interactive control information, and convert it into an autonomous driving intervention instruction for the user in the car to change the autonomous driving behavior. The graphical user interface interaction engine module is used to process the touch information input by the user in the car, and convert the touch information into an autonomous driving intervention instruction for the user in the car to change the autonomous driving behavior. The graphical user interface interaction engine module may include but is not limited to a screen touch screen interaction module or a physical button interaction module. The map navigation module is used to plan and manage the path for the autonomous driving intervention instruction of the user in the car, and the path information will be synchronized to the autonomous driving engine module, while receiving the navigation or path change information from the cockpit interaction engine module, and initiating corresponding processing. The autonomous driving engine module is used to process the driving behavior of the vehicle during the autonomous driving process. The autonomous driving engine module may include but is not limited to an environmental perception module, a safety decision-making module, a driving first execution module, a path management module, etc.

[0079] For the disclosed embodiments, the cockpit interaction engine module in the interactive device of the autonomous driving can receive interactive control information, perform information recognition on the interactive control information, convert it into an autonomous driving intervention instruction for the user in the vehicle to change the autonomous driving behavior, and send the autonomous driving intervention instruction to the autonomous driving engine module, so that the autonomous driving engine module determines whether the autonomous driving intervention instruction meets the execution conditions based on the safety decision of the autonomous driving; if it is determined that the autonomous driving intervention instruction meets the execution conditions, the autonomous driving intervention instruction is executed.

[0080] Based on the above Figure 1 , Figure 2 The specific implementation of the method shown in the embodiment provides an interactive device for autonomous driving, such as Figure 8 As shown, the device includes: an acquisition module 31, a generation module 32, a judgment module 33, and an execution module 34;

[0081] An acquisition module 31 is used to acquire interactive control information input by a user, and obtain autonomous driving intervention information by identifying the interactive control information, wherein the autonomous driving intervention information represents at least partial intervention in the current autonomous driving behavior;

[0082] A generating module 32, configured to generate an automatic driving intervention instruction corresponding to the automatic driving intervention information;

[0083] A determination module 33, used to determine whether the current driving environment information supports the safe execution of the automatic driving intervention instruction;

[0084] The execution module 34 is used to execute the autonomous driving intervention instruction without terminating the autonomous driving process if it is determined that the driving environment information supports the safe execution of the autonomous driving intervention instruction, output the execution result of the autonomous driving intervention instruction, and output the change path information when changing the current path planning.

[0085] In a specific application scenario, the acquisition module 31 can be used to obtain user intention information by identifying the interactive control information; based on the user intention information, determine whether it is necessary to change the autonomous driving behavior at present; if it is determined that the autonomous driving behavior needs to be changed at present, the intention description information corresponding to the user intention information is used as the autonomous driving intervention information.

[0086] In a specific application scenario, the generation module 32 can be used to obtain at least one new path plan to reach the destination from the map navigation when it is determined that the user needs to change the autonomous driving route by identifying the intention description information and no new driving route is indicated; output a first prompt message, wherein the first prompt message is used to prompt the selection of one of the at least one new path plan to execute; and generate an autonomous driving intervention instruction corresponding to the target path plan in response to the target path plan selected by the user from the at least one new path plan.

[0087] In a specific application scenario, the generation module 32 can be used to generate an autonomous driving intervention instruction corresponding to the new driving path indicated by the user when it is determined that the user needs to change the autonomous driving driving path and indicates a new driving path by identifying the intention description information.

[0088] In a specific application scenario, when it is determined that the new driving path indicated by the user will change the current path planning, the map path planning of the autonomous driving is obtained from the map navigation; it is determined whether the execution of the autonomous driving intervention instruction will change the map path planning; when it is determined that the execution of the autonomous driving intervention instruction will change the map path planning, a second prompt message is output, and the second prompt message is used to ask whether to confirm the execution of the autonomous driving intervention instruction and change the current path planning; the judgment module 33 can be used to judge whether the current driving environment information supports the safe execution of the autonomous driving intervention instruction in response to confirming the execution of the autonomous driving intervention instruction and changing the current path planning.

[0089] In a specific application scenario, it is determined whether the instruction contents of the multiple generated autonomous driving intervention instructions are consistent; when the instruction contents of the multiple autonomous driving intervention instructions are inconsistent, a third prompt information is output, and the third prompt information is used to prompt the selection of one of the multiple autonomous driving intervention instructions for execution; the interactive control information is multimodal interactive control information, and the autonomous driving intervention instructions are multiple autonomous driving intervention instructions generated by identifying the multimodal interactive control information; the judgment module 33 can be used to respond to a target intervention instruction selected from multiple autonomous driving intervention instructions with inconsistent instruction contents, and determine whether the current driving environment information supports the safe execution of the target intervention instruction; or, when the instruction contents of the multiple autonomous driving intervention instructions are consistent, the multiple autonomous driving intervention instructions are merged into one intervention instruction, and it is determined whether the current driving environment information supports the safe execution of the merged intervention instruction.

[0090] In a specific application scenario, the judgment module 33 can be used to judge whether executing the autonomous driving intervention instruction will cause driving risks based on current weather information, and / or road information, and / or surrounding vehicle information; if executing the autonomous driving intervention instruction will not cause driving risks, it is determined that the driving environment information supports the safe execution of the autonomous driving intervention instruction.

[0091] It should be noted that for other corresponding descriptions of the functional units involved in the interactive device for autonomous driving provided in this embodiment, please refer to Figure 1 and Figure 2 The corresponding description in will not be repeated here.

[0092] Based on the above Figure 1 and Figure 2 The method shown in the embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Figure 1 and Figure 2 The method shown.

[0093] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0094] Based on the above Figure 1 and Figure 2 The method shown, and Figure 8 In order to achieve the above-mentioned purpose, the embodiment of the present application further provides an electronic device which can be configured on the vehicle (such as an electric vehicle) side, and the device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 and Figure 2 The method shown.

[0095] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0096] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0097] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.

[0098] Based on the above electronic device, the embodiment of the present application further provides a vehicle, which may specifically include: the above electronic device. The vehicle may specifically be an electric car, etc.

[0099] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or by hardware. By applying the technical solution of this embodiment, the user can issue intervention instructions for autonomous driving through convenient and fast interactive control. If the current driving environment information supports the safe execution of the autonomous driving intervention instruction, the autonomous driving intervention instruction can be executed to implement intervention in the local behavior of the autonomous driving to change the local behavior of the autonomous driving without terminating the autonomous driving process. This can then be achieved through a convenient and fast natural interaction method to safely and flexibly change the local behavior of the autonomous driving, which can improve the user's car experience.

[0100] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0101] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.

Claims

1. An interactive method for autonomous driving, It is characterized in that The method comprises: Acquire interactive control information input by a user, and obtain autonomous driving intervention information by identifying the interactive control information, wherein the autonomous driving intervention information represents at least partial intervention in a current autonomous driving behavior; generating an autonomous driving intervention instruction corresponding to the autonomous driving intervention information; Determining whether the current driving environment information supports the safe execution of the autonomous driving intervention instruction; If it is determined that the driving environment information supports the safe execution of the autonomous driving intervention instruction, the autonomous driving intervention instruction is executed without terminating the autonomous driving process.

2. The method according to claim 1, It is characterized in that The obtaining the automatic driving intervention information by identifying the interactive control information includes: Obtaining user intention information by identifying the interactive control information; Based on the user intention information, determining whether the autonomous driving behavior needs to be changed; If it is determined that the autonomous driving behavior needs to be changed at present, the intention description information corresponding to the user intention information is used as the autonomous driving intervention information.

3. The method according to claim 2, It is characterized in that The generating the autonomous driving intervention instruction corresponding to the autonomous driving intervention information includes: When it is determined by identifying the intention description information that the user needs to change the driving path of the autonomous driving and no new driving path is indicated, obtaining at least one new path plan to reach the destination from the map navigation; Outputting first prompt information, where the first prompt information is used to prompt selecting one of the at least one new path planning to execute; In response to a target path plan selected by a user from the at least one new path plan, an automatic driving intervention instruction corresponding to the target path plan is generated.

4. The method according to claim 2, It is characterized in that The generating the autonomous driving intervention instruction corresponding to the autonomous driving intervention information includes: When it is determined by identifying the intention description information that the user needs to change the autonomous driving driving path and indicates a new driving path, an autonomous driving intervention instruction corresponding to the new driving path indicated by the user is generated.

5. The method according to claim 4, It is characterized in that Before determining whether the current driving environment information supports safe execution of the automatic driving intervention instruction, the method further includes: When it is determined that the new driving path indicated by the user will change the current path planning, obtaining the map path planning of the autonomous driving from the map navigation; Determining whether executing the autonomous driving intervention instruction will change the map path planning; If it is determined that executing the autonomous driving intervention instruction will change the map path planning, outputting a second prompt message, wherein the second prompt message is used to inquire whether to confirm executing the autonomous driving intervention instruction and changing the current path planning; The determining whether the current driving environment information supports the safe execution of the automatic driving intervention instruction includes: In response to confirming the execution of the autonomous driving intervention instruction and changing the current path planning, it is determined whether the current driving environment information supports the safe execution of the autonomous driving intervention instruction.

6. The method according to claim 1, It is characterized in that The interactive control information is multimodal interactive control information, and the autonomous driving intervention instruction is a plurality of autonomous driving intervention instructions generated by identifying the multimodal interactive control information; Before determining whether the current driving environment information supports safe execution of the automatic driving intervention instruction, the method further includes: Determine whether the command contents of multiple generated autonomous driving intervention commands are consistent; When the contents of the multiple autonomous driving intervention instructions are inconsistent, outputting third prompt information, wherein the third prompt information is used to prompt the user to select one of the multiple autonomous driving intervention instructions for execution; The determining whether the current driving environment information supports the safe execution of the automatic driving intervention instruction includes: In response to a target intervention instruction selected from a plurality of autonomous driving intervention instructions having inconsistent instruction contents, determining whether current driving environment information supports safe execution of the target intervention instruction; or, When the instruction contents of the multiple autonomous driving intervention instructions are consistent, the multiple autonomous driving intervention instructions are merged into one intervention instruction, and it is determined whether the current driving environment information supports the safe execution of the merged intervention instruction.

7. The method according to claim 1, It is characterized in that The determining whether the current driving environment information supports the safe execution of the automatic driving intervention instruction includes: Based on current weather information, and / or road information, and / or surrounding vehicle information, determining whether executing the autonomous driving intervention instruction will cause driving risks; If executing the autonomous driving intervention instruction does not cause driving risk, it is determined that the driving environment information supports the safe execution of the autonomous driving intervention instruction.

8. The method according to any one of claims 1 to 7, It is characterized in that After executing the automatic driving intervention instruction, the method further includes: Output the execution result of the autonomous driving intervention instruction, and output the changed path information when changing the current path planning.

9. An interactive device for autonomous driving, It is characterized in that include: a receiving module, configured to obtain interactive control information input by a user, and obtain autonomous driving intervention information by identifying the interactive control information, wherein the autonomous driving intervention information represents at least partial intervention in a current autonomous driving behavior; A generating module, configured to generate an automatic driving intervention instruction corresponding to the automatic driving intervention information; A determination module, used to determine whether the current driving environment information supports the safe execution of the automatic driving intervention instruction; An execution module is used to execute the autonomous driving intervention instruction without terminating the autonomous driving process if it is determined that the driving environment information supports the safe execution of the autonomous driving intervention instruction.

10. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

12. A vehicle, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

Images