Authentication system for voice-controlled automatic driving system

Abstract

The present invention relates to an authentication system for a voice-controlled automatic driving system of a vehicle, comprising: a plurality of perception sensors for collecting perception data indicating an environment around the vehicle; and one or more controllers in electronic communication with the one or more autonomous driving controllers and the plurality of perceptual sensors of the voice-controlled autonomous driving system. A voice-controlled automated driving system determines a trajectory of a vehicle based on voice commands generated by an occupant of the vehicle. The one or more controllers execute instructions to calculate a trustworthiness score for the occupant that quantifies the reliability of the one or more voice-based commands generated by the occupant. In response to determining that the overall credibility score of the occupant is less than the threshold access score, the one or more controllers adjust a level of access to the voice-controlled automated driving system by the occupant.

Description

Technical Field

[0001] The present disclosure relates to an authentication system for a voice-controlled automated driving system for a vehicle, wherein the authentication system confirms that an occupant of the vehicle who generates voice commands is an authorized user and determines an overall credibility score associated with the occupant. Background Art

[0002] Vehicles are increasingly incorporating additional user experience related features. For example, an occupant may utilize voice commands to operate various features and systems that are part of the vehicle. One example of utilizing voice commands to control various features and systems of a vehicle is a voice controlled autonomous driving system. The voice controlled autonomous driving system includes a microphone that receives voice commands spoken by a driver or occupant of the vehicle, wherein a trajectory is determined based on the voice commands. However, it should be understood that not all occupants in a vehicle may be trained or have the level of responsibility required to control the trajectory of the vehicle based on voice commands. As an example, one or more passengers of the vehicle may be children or teenagers who do not have the necessary knowledge or maturity required to control the vehicle via voice commands. In addition, it should be understood that some occupants may also attempt to test the limits of the autonomous driving system based on voice commands.

[0003] Therefore, while voice-controlled autonomous driving systems achieve their intended purpose, there is a need in the art for an improved method to authenticate and verify a user attempting to control a voice-controlled autonomous driving system. Summary of the invention

[0004] According to several aspects, a certification system for a voice-controlled autonomous driving system for a vehicle is disclosed. The certification system includes: a plurality of perception sensors for collecting perception data indicating the surrounding environment of the vehicle; and one or more controllers in electronic communication with the one or more autonomous driving controllers and the plurality of perception sensors of the voice-controlled autonomous driving system. The voice-controlled autonomous driving system determines a trajectory of the vehicle based on a voice command generated by an occupant of the vehicle. The one or more controllers execute instructions to receive an electrical signal representing a voice command generated by an occupant of the vehicle, wherein the voice command indicates a driving maneuver that changes the movement of the vehicle. The one or more controllers convert the electrical signal into a text-based command based on one or more speech recognition algorithms. The one or more controllers receive a pre-compiled driving maneuver indicated by the text-based command, a feasibility message indicating the feasibility of the pre-compiled driving maneuver, a legal message indicating the authorization status of the pre-compiled driving maneuver, and an execution message indicating the execution status of the pre-compiled driving maneuver from the one or more autonomous driving controllers. A confidence score for the occupant is calculated, by one or more controllers, based on pre-compiled driving maneuvers, feasibility of the pre-compiled driving maneuvers, authorization status of the pre-compiled driving maneuvers, execution status of the pre-compiled driving maneuvers, and perception data collected by a plurality of perception sensors, wherein an overall confidence score for the occupant quantifies the reliability of one or more voice-based commands generated by the occupant over time, and based on the confidence score, the one or more controllers compare the overall confidence score for the occupant to a threshold access score, and in response to determining that the overall confidence score for the occupant is less than the threshold access score, adjust a level of access for the occupant to the voice-controlled automated driving system.

[0005] In another aspect, one or more controllers execute instructions to classify a voice command represented by a text-based command into one of an emergency driving situation and a routine maneuver by identifying one or more emergency keywords present in the text-based command.

[0006] In yet another aspect, in response to classifying a voice command represented by a text-based command as an emergency driving situation, the one or more controllers send the text-based command to one or more autopilot controllers of a voice-controlled autopilot system without confirming that the occupant is an authorized user.

[0007] In one aspect, in response to determining that the voice command generated by the occupant is a routine driving maneuver, it is confirmed that the occupant generating the voice command is an authorized user of the voice-controlled automated driving system.

[0008] In another aspect, the one or more controllers are in electronic communication with an interior camera positioned to capture image data representative of a head and face of an occupant of the vehicle.

[0009] In yet another aspect, one or more controllers wirelessly communicate with one or more user databases storing facial recognition data representing facial features and voice recognition data representing voices of all authorized users of a voice-controlled autonomous driving system and a library of authorized digital images, each of which represents a digital image of an individual previously authenticated as an authorized user of the voice-controlled autonomous driving system.

[0010] In one aspect, one or more controllers confirm that an occupant generating a voice command is an authorized user of a voice-controlled automated driving system by comparing image data from an interior camera with facial recognition data stored in a user database, comparing an electrical signal representing the voice command with the voice recognition data stored in the user database, and in response to determining that there is no match between the image data representing the occupant's face and the facial recognition data of one of the authorized users of the voice-controlled automated driving system stored in the user database, generating a request for a user-specific code, which is a unique identifier associated with an authorized digital image of the occupant stored in one or more user databases.

[0011] In another aspect, one or more controllers execute instructions to compare an authorized digital image of an occupant associated with a user-specific code to a library of authorized digital images stored in one or more user databases; and in response to determining that the digital image of the occupant associated with the user-specific code does not match one of the authorized digital images stored in the one or more user databases, execute a procedure to register the authorized digital image of the occupant.

[0012] In yet another aspect, one or more controllers execute instructions to calculate a confidence score based on the following equation:

[0013] CreditabilityScore=w1S success -w2S stat -w3S infeasible -w4f user S consequence

[0014] Among them, w1, w2, w3, and w4 represent weight factors, S success represents the reward for each voice command generated by the occupant that results in the vehicle successfully performing the corresponding pre-compiled driving maneuver, and S success Based on the execution status, S stat indicating a statutory penalty for each voice command generated that causes the vehicle to not perform a corresponding pre-compiled driving maneuver because the pre-compiled driving maneuver is not authorized based on traffic rules associated with the particular location in which the vehicle is located, infeasiblerepresents the feasibility penalty for each voice command generated by the occupant that causes the vehicle to not perform a driving maneuver because the pre-compiled driving maneuver is not feasible, f user a value indicating an error associated with an occupant generating a voice command in which the precompiled driving maneuver was successfully executed but degraded the performance of the voice-controlled automated driving system, consequence Represents a consequence penalty for each voice command generated by the occupant that instructs the performance of the voice-controlled automated driving system to degrade.

[0015] In one aspect, one or more controllers execute instructions to determine a value f associated with an occupant indicating an error based on user :

[0016] f user =f(time,distance goal ,deviation params )

[0017] Among them, time represents the time interval, distance goal Indicates the change of target state, and deviation params Represents the degree of change between multiple behavior planner parameters and one or more updated behavior planner parameters.

[0018] In another aspect, the change in target state is determined by comparing a distance measured between an original target position indicating a planned target state of the vehicle initially determined by a behavior planner of the voice-controlled automated driving system and a modified target position indicating a modified target state of the vehicle achieved after performing a pre-compiled driving maneuver.

[0019] In yet another aspect, the authorization status indicates when the pre-compiled driving maneuver is authorized based on traffic regulations associated with a particular location where the vehicle is located.

[0020] In one aspect, the execution status indicates that the pre-compiled driving maneuver indicated by the voice command has been successfully completed by the vehicle.

[0021] In another aspect, adjusting an occupant's level of access to a voice-controlled automated driving system includes denying access to the voice-controlled automated driving system.

[0022] In yet another aspect, adjusting an occupant's level of access to a voice-controlled automated driving system includes assigning a lower level of access to the voice-controlled automated driving system, wherein the lower level of access only allows the occupant to control the voice-controlled automated driving system within a restricted area.

[0023] In one aspect, a method for calculating an overall credibility score of an occupant of a vehicle by an authentication system of a voice-controlled autonomous driving system is disclosed. The method includes: receiving, by one or more controllers, an electrical signal representing a voice command generated by an occupant of the vehicle, wherein the voice command indicates a driving maneuver that changes the motion of the vehicle, and wherein the one or more controllers are in electronic communication with one or more autonomous driving controllers and a plurality of perception sensors of the voice-controlled autonomous driving system, the voice-controlled autonomous driving system determining a trajectory of the vehicle based on the voice command generated by the occupant of the vehicle. The method includes: converting, by the one or more controllers, the electrical signal into a text-based command based on one or more speech recognition algorithms. The method includes: receiving, by the one or more controllers, a pre-compiled driving maneuver indicated by the text-based command, a feasibility message indicating the feasibility of the pre-compiled driving maneuver, a statutory message indicating the authorization status of the pre-compiled driving maneuver, and an execution message from the one or more autonomous driving controllers indicating the execution status of the pre-compiled driving maneuver. The method further includes: calculating, by one or more controllers, a credibility score for the occupant based on pre-compiled driving maneuvers, feasibility of the pre-compiled driving maneuvers, authorization status of the pre-compiled driving maneuvers, execution status of the pre-compiled driving maneuvers, and perception data collected by a plurality of perception sensors, wherein the credibility score for the occupant quantifies the reliability of one or more voice-based commands generated by the occupant over time, and based on the credibility score. The method includes: comparing, by the one or more controllers, the overall credibility score for the occupant to a threshold access score. Finally, in response to determining that the overall credibility score for the occupant is less than the threshold access score, the method includes adjusting the occupant's access level to the voice-controlled automated driving system.

[0024] In one aspect, a certification system for a voice-controlled autonomous driving system for a vehicle is disclosed. The certification system includes: a plurality of perception sensors for collecting perception data indicating the surrounding environment of the vehicle; and one or more controllers in electronic communication with the one or more autonomous driving controllers and the plurality of perception sensors of the voice-controlled autonomous driving system. The voice-controlled autonomous driving system determines a trajectory of the vehicle based on a voice command generated by an occupant of the vehicle, wherein the one or more controllers execute instructions to receive an electrical signal representing a voice command generated by an occupant of the vehicle, wherein the voice command indicates a driving maneuver that changes the movement of the vehicle. The one or more controllers convert the electrical signal into a text-based command based on one or more speech recognition algorithms. The one or more controllers receive a pre-compiled driving maneuver indicated by the text-based command, a feasibility message indicating the feasibility of the pre-compiled driving maneuver, a legal message indicating an authorization status of the pre-compiled driving maneuver, and an execution message indicating an execution status of the pre-compiled driving maneuver from the one or more autonomous driving controllers, wherein the authorization status indicates when the pre-compiled driving maneuver is authorized based on traffic rules associated with a specific location where the vehicle is located. A confidence score for the occupant is calculated by one or more controllers based on precompiled driving maneuvers, feasibility of the precompiled driving maneuvers, authorization status of the precompiled driving maneuvers, execution status of the precompiled driving maneuvers, and perception data collected by a plurality of perception sensors, wherein the confidence score for the occupant quantifies the reliability of one or more voice-based commands generated by the occupant over time, and based on the confidence score, the one or more controllers compare the overall confidence score for the occupant to a threshold access score, and in response to determining that the confidence score for the occupant is less than the threshold access score, the one or more controllers adjust the occupant's access level to the voice-controlled automated driving system, wherein the authorization status indicates when the precompiled driving maneuver is authorized based on traffic rules associated with a particular location where the vehicle is located.

[0025] In another aspect, one or more controllers execute instructions to classify a voice command represented by a text-based command into one of an emergency driving situation and a routine maneuver by identifying one or more emergency keywords present in the text-based command.

[0026] In yet another aspect, in response to classifying a voice command represented by a text-based command as an emergency driving situation, the one or more controllers send the text-based command to one or more autopilot controllers of a voice-controlled autopilot system without confirming that the occupant is an authorized user.

[0027] In one aspect, in response to determining that the voice command generated by the occupant is a routine driving maneuver, one or more controllers confirm that the occupant generating the voice command is an authorized user of the voice-controlled automated driving system.

[0028] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0030] Figure 1 is a schematic diagram of a vehicle including a disclosed authentication system including one or more controllers in electronic communication with a plurality of perception sensors and a voice-controlled autonomous driving system according to an exemplary embodiment;

[0031] Figure 2 According to an exemplary embodiment, Figure 1 the software architecture of one or more controllers shown;

[0032] Figure 3 A process flow chart illustrating a method for confirming that an occupant generating a voice command is an authorized user of a voice-controlled automated driving system according to an exemplary embodiment is shown;

[0033] Figure 4 is a diagram showing a change in a target state of a behavior planner of a voice-controlled autonomous driving system according to an exemplary embodiment; and

[0034] Figure 5 is a process flow diagram illustrating a method for scoring voice commands generated by an occupant of a vehicle according to an exemplary embodiment. DETAILED DESCRIPTION

[0035] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses.

[0036] refer to Figure 1 , a schematic diagram of an exemplary authentication system 10 for a voice-controlled autonomous driving system 12 for a vehicle 14 is shown. In an embodiment, the voice-controlled autonomous driving system 12 is provided with at least Level 2 autonomous driving control, such as an advanced driver assistance system (ADAS). It should be understood that the vehicle 14 can be any type of vehicle, such as but not limited to a car, a truck, a sport utility vehicle, a van, or an RV. The authentication system 10 includes one or more controllers 20 in electronic communication with a plurality of perception sensors 22, a microphone 24, a human-machine interface (HMI) 26, and an interior camera 28. The plurality of perception sensors 22 are configured to collect perception data indicative of the environment surrounding the vehicle 14. In the example Figure 1In the non-limiting embodiment shown, the plurality of perception sensors 22 include one or more cameras 30 , an inertial measurement unit (IMU) 32 , a global positioning system (GPS) 34 , radar 36 , and lidar 38 , however, it should be understood that additional sensors may also be used.

[0037] The one or more controllers 20 also communicate wirelessly with a backend server 40 via a communication network 42, wherein the backend server 40 is remote from the vehicle 14 as part of a backend office 44. The backend server 40 represents one or more computers, each of which includes at least one processor, memory, associated input and output buses, and wireless communication capabilities. The backend server 40 is in electronic communication with one or more user databases 46, wherein the one or more user databases 46 store facial recognition data representing facial features and voice recognition data representing voice for all authorized users of the voice-controlled automated driving system 12. The one or more user databases 46 also store a library of authorized digital images. Each authorized digital image represents a digital image of an individual who has been previously authenticated as an authorized user of the voice-controlled automated driving system 12. The one or more user databases 46 also store an overall credibility score associated with an occupant 50 of the vehicle 14, wherein the overall credibility score is described below.

[0038] The voice-controlled automated driving system 12 determines a trajectory 48 of the vehicle 14 based on voice commands generated by an occupant 50 of the vehicle 14. The voice commands generated by the occupant 50 of the vehicle 14 indicate driving maneuvers that change the motion of the vehicle 14 while the vehicle 14 is traveling. In other words, the voice commands generated by the occupant 50 change the current trajectory of the vehicle 14. Some examples of voice commands include, but are not limited to, commands to take the next exit along a highway, to make a lane change, to pass another vehicle, and to change the destination location of a route plan.

[0039] As described below, the authentication system 10 confirms that the occupant 50 who generated the voice command that determined the trajectory 48 of the vehicle 14 is an authorized user of the voice-controlled automated driving system 12. It should be understood that the voice-controlled automated driving system 12 only accepts voice commands generated by authorized users unless the voice command indicates an emergency driving situation. In one embodiment, an emergency driving situation is a maneuver performed in response to a sudden, unexpected situation, such as another vehicle immediately crossing the path of the host vehicle (i.e., vehicle 14). In another embodiment, an emergency driving situation is receiving an instruction to immediately stop the vehicle 14, such as when the occupant 50 indicates that he or she is experiencing motion sickness.

[0040] Once the occupant 50 is confirmed as an authorized user, the one or more controllers 20 of the authentication system 10 score the voice commands generated by the occupant 50. As described below, the authentication system 10 adjusts the occupant's 50 access level to the voice-controlled automated driving system 12 in response to determining that the overall credibility score associated with the occupant 50 is below a threshold access score. For example, the occupant 50 may generate a voice command that causes the vehicle 14 to perform an unauthorized, unfeasible driving maneuver, or cause a degradation in the performance of the voice-controlled automated driving system 12. Some examples of degradation in the performance of the voice-controlled automated driving system 12 include, but are not limited to, driver takeover of the vehicle 14, active emergency braking or steering, not reaching a destination, significantly delaying the estimated time to reach a destination, causing a traffic incident (such as a collision), or performing a driving maneuver that reduces the comfort of the occupant.

[0041] The microphone 24 represents a device that converts sound waves into electrical signals, where the electrical signals are received by the one or more controllers 20. Specifically, the microphone 24 converts voice commands generated by the occupant 50 of the vehicle 14 into electrical signals representing the voice commands. The HMI 26 represents a user interface, where the occupant 50 of the vehicle 14 receives messages from the voice-controlled automated driving system 12 through the HMI 26 and provides user input thereto. An example of the HMI 26 is a touch screen that receives user input and displays images on a display to the occupant 50. The interior camera 28 is positioned to capture image data representing the head and face of the occupant 50 of the vehicle 14.

[0042] Figure 2 Shows Figure 1 The software architecture of one or more controllers 20 shown in FIG. 1 , wherein one or more controllers 20 are used as a voice-controlled autonomous driving system 12 ( Figure 1 ) is in electronic communication with one or more autonomous driving controllers 52 that are part of the authentication system 10. The one or more controllers 20 of the authentication system 10 include a speech recognition module 60, an authorization module 62, a scoring module 64, and an access module 66. In one embodiment, the speech recognition module 60 of the one or more controllers 20 receives an electrical signal representing a voice command generated by the occupant 50 from the microphone 24 and converts the electrical signal into a text-based command based on one or more speech recognition algorithms, such as perceptual linear prediction (PLP) features, Viterbi search, and speech recognition methods using deep neural networks and discriminative training.

[0043] The speech recognition module 60 then classifies the speech command represented by the text-based command as an emergency driving situation or a routine maneuver by identifying one or more emergency keywords present within the text-based command. The emergency keywords indicate that an occupant of the vehicle 14 is attempting to perform an emergency driving situation. Some examples of the one or more emergency keywords include the words stop, help, emergency, or phrases indicating that the occupant 50 is attempting to stop the vehicle 14 immediately, such as "I feel sick and am about to vomit, pull over." In response to determining that the one or more emergency keywords are present in the text-based command, the speech recognition module 60 classifies the speech command as an emergency driving situation.

[0044] As described above, the voice-controlled automated driving system 12 only accepts voice commands generated by authorized users unless the voice command indicates an emergency driving situation. In response to the voice recognition module 60 classifying the voice command represented by the text-based command as an emergency driving situation, the voice recognition module 60 sends the text-based command to the one or more automated driving controllers 52 of the voice-controlled automated driving system 12 without confirming that the occupant 50 is an authorized user. Otherwise, the authorization module 62 of the one or more controllers 20 confirms that the occupant 50 who generated the voice command is an authorized user of the voice-controlled automated driving system 12.

[0045] The authorization module 62 of the one or more controllers 20 receives image data representing the face of an occupant 50 of the vehicle 14 from the interior camera 28 and receives a voice command represented by a text-based command generated by the occupant 50 from the voice recognition module 60. The authorization module 62 of the one or more controllers 20 also receives facial recognition data representing facial features and voice recognition data representing voices of all authorized users of the voice-controlled automated driving system 12, as well as a library of authorized digital images from the one or more user databases 46, over the communication network 42. In response to the voice recognition module 60 determining that the voice command generated by the occupant 50 is a regular driving maneuver, the authorization module 62 confirms that the occupant 50 who generated the voice command is an authorized user of the voice-controlled automated driving system 12 by comparing the image data from the interior camera 28 with the facial recognition data stored in the user database 46 and comparing the electrical signal representing the voice command from the microphone 24 with the voice recognition data stored in the user database 46, wherein Figure 3 A process for identifying an occupant 50 is shown.

[0046] Reference now Figure 3 , shows an exemplary process flow diagram of a method 300 for confirming that the occupant 50 generating the voice command is an authorized user of the voice-controlled automated driving system 12. Figures 1 to 3, the method 300 may begin at block 302. In block 302, the authorization module 62 of the one or more controllers 20 receives image data from the interior camera 28. The method 300 may then proceed to decision block 304.

[0047] In decision block 304, the authorization module 62 of the one or more controllers 20 executes one or more facial recognition algorithms that compare the image data from the interior camera 28 representing the face of the occupant 50 of the vehicle 14 with the facial recognition data stored in the user database 46. In response to determining a match between the image data representing the face of the occupant 50 of the vehicle 14 and the facial recognition data of one of the authorized users of the voice-controlled automated driving system 12 stored in the user database 46, the method 300 may proceed to block 306. Otherwise, the method proceeds to block 314.

[0048] In block 306 , the authorization module 62 of the one or more controllers 20 receives an electrical signal representing a voice command generated by the occupant 50 from the microphone 24 . The method 300 may then proceed to decision block 308 .

[0049] In decision block 308, the authorization module 62 of the one or more controllers 20 executes one or more speech recognition algorithms that compare the electrical signal representing the voice command generated by the occupant 50 from the microphone 24 with the speech recognition data representing the voice of all authorized users who voice control the automated driving system 12 stored in the one or more user databases 46. In response to determining a match between the voice command generated by the occupant 50 from the microphone 24 and the speech recognition data representing the voice of all authorized users who voice control the automated driving system 12, the method 300 may proceed to decision block 310. Otherwise, the method 300 may terminate.

[0050] In decision block 310, the authorization module 62 of the one or more controllers 20 confirms that the occupant 50 is still within the interior compartment of the vehicle 14. The authorization module 62 of the one or more controllers 20 confirms that the occupant 50 is still within the interior compartment of the vehicle 14 based on various methods, such as determining whether a door of the vehicle 14 is open through one or more infrared sensors located within the interior compartment of the vehicle 14, or monitoring the occupant 50 through the interior camera 28. In response to determining that the occupant 50 has left the vehicle 14, the method terminates. Otherwise, the method 300 proceeds to block 312.

[0051] In block 312, the authorization module 62 of the one or more controllers 20 confirms that the occupant 50 who generated the voice command is an authorized user of the voice-controlled automated driving system 12. The authorization module 62 also sends a notification to the one or more automated driving controllers 52 of the voice-controlled automated driving system 12 indicating that the occupant 50 is also an authorized user. The method 300 may then terminate.

[0052] Returning to decision box 304, in response to determining that there is no match between the image data representing the face of the occupant 50 of the vehicle 14 and the facial recognition data of one of the authorized users of the voice-controlled automated driving system 12 stored in the user database 46, the method 300 can proceed to box 314. In box 314, the authorization module 62 of the one or more controllers 20 instructs the HMI 26 to generate a notification to the occupant 50, wherein the notification prompts the occupant 50 to provide a user input indicating a user-specific code to the HMI 26. The user-specific code is a unique identifier associated with the authorized digital image of the occupant 50 stored in the one or more user databases 46. In one embodiment, the user-specific code represents biometric data, such as a fingerprint or retinal scan. Prior to executing the method 300, the occupant 50 needs to register the authorized digital image in the one or more user databases 46. In one embodiment, the user-specific code is a digital password sent by the backend server 40 to the smartphone of the occupant 50. Then, the method 300 can proceed to decision box 316.

[0053] In decision block 316, the authorization module 62 of the one or more controllers 20 compares the image data representing the face of the occupant 50 to a library of authorized digital images stored in the one or more user databases 46, wherein each authorized digital image represents a digital image of an individual previously authenticated as an authorized user of the voice-controlled automated driving system 12. In response to determining that the image data representing the face of the occupant 50 associated with the user-specific code matches one of the authorized digital images stored in the one or more user databases 46, the method 300 may proceed to decision block 318. In block 318, the authorization module 62 checks whether voice data representing the voice of the occupant 50 exists in the memory of the one or more controllers 20. If no recording exists, the method 300 proceeds to block 320, and the authorization module 62 instructs the HMI 26 to generate a prompt requesting the occupant 50 to create a recording. Otherwise, the method returns to block 306.

[0054] Referring to decision block 316, in response to determining that the image data representing the face of the occupant 50 does not match one of the authorized digital images stored in the one or more user databases 46, the method 300 may proceed to block 322. In block 320, the authorization module 62 executes a procedure to register the authorized digital image of the occupant 50 within the library stored in the one or more user databases 46. The procedure includes requiring the occupant 50 to distinguish himself or herself from other occupants in the vehicle 14 by performing an action such as raising his or her hand. The method 300 may then proceed to block 322, where the authorized digital image of the occupant 50 is added to the library stored in the one or more user databases 46. The method 300 may then return to decision block 318.

[0055] refer to Figure 2 In response to confirming that the occupant 50 who generated the voice command is an authorized user of the voice-controlled automated driving system 12, the authorization module 62 of the one or more controllers 20 sends a confirmation message and a text-based command representing the voice command to the one or more controllers 20 of the voice-controlled automated driving system 12, wherein the confirmation message indicates that the occupant 50 is an authorized user. As described below, the scoring module 64 of the one or more controllers 20 receives the pre-compiled driving maneuver indicated by the text-based command representing the voice command, a feasibility message indicating the feasibility of the pre-compiled driving maneuver, a legal message indicating the authorization status of the pre-compiled driving maneuver, and an execution message indicating the execution status of the pre-compiled driving maneuver from the one or more automated driving controllers 52 of the voice-controlled automated driving system 12. As described below, in response to determining that the pre-compiled driving maneuver was successfully executed, the scoring module 64 scores the pre-compiled driving maneuver based on the text-based command received from the plurality of perception sensors 22 ( Figure 1 ) to compute consequence penalties indicating potential performance degradation caused by executing pre-compiled driving maneuvers.

[0056] The scoring module 64 calculates a credibility score for the occupant 50 based on input from the one or more autonomous driving controllers 52 of the voice-controlled autonomous driving system 12 and the perception data collected by the plurality of perception sensors 22. The credibility score for the occupant 50 is combined with previous credibility scores associated with the occupant 50 stored on the one or more user databases 46. The overall credibility score quantifies the reliability of the occupant 50 over time. For example, if the occupant 50 continuously generates voice-based commands that instruct the vehicle 14 to accelerate to a speed exceeding the current speed limit, or performs driving maneuvers that require driver intervention while autonomous driving, the overall credibility score for the occupant 50 may decrease.

[0057] In response to receiving the confirmation message by the authorization module 62 of the one or more controllers 20, the one or more autopilot controllers 52 of the voice-controlled autopilot system 12 classify the text-based command representing the voice command as one of a plurality of pre-compiled driving maneuvers based on one or more machine learning algorithms. In one embodiment, the one or more autopilot controllers 52 classify the text-based command based on a neural network trained to classify the text-based command. The pre-compiled driving maneuvers represent a list of potential driving maneuvers that the vehicle 14 may perform. Some examples of potential driving maneuvers include, but are not limited to, commands to take the next exit, make a lane change (left lane change or right lane change), and overtake another vehicle (through the left lane or the right lane). It should be understood that in some cases, the text-based command represents a driving maneuver that is impossible to perform or is not feasible. For example, if the voice command instructs the vehicle 14 to fly over a vehicle traveling on the road, this would be an infeasible driving maneuver.

[0058] If the one or more autonomous driving controllers 52 cannot classify the text-based command representing the voice command as one of the plurality of pre-compiled driving maneuvers because the driving maneuver is not feasible, the one or more autonomous driving controllers 52 send a feasibility message to the scoring module 64 of the one or more controllers 20 indicating that the pre-compiled driving maneuver cannot be accurately classified as one of the pre-compiled driving maneuvers and is not feasible. In response to classifying the text-based command representing the voice command as one of the plurality of pre-compiled driving maneuvers, the one or more autonomous driving controllers 52 send a feasibility message to the scoring module 64 of the one or more controllers 20 indicating that the pre-compiled driving maneuver is feasible.

[0059] In addition to the pre-compiled driving maneuver, the one or more autopilot controllers 52 of the voice-controlled autopilot system 12 also sends a statutory message to the scoring module 64 of the certification system 10, wherein the statutory message indicates an authorization status of the pre-compiled driving maneuver. The authorization status indicates when the pre-compiled driving maneuver is authorized based on traffic rules associated with the specific location where the vehicle 14 is located. The one or more autopilot controllers 52 of the voice-controlled autopilot system 12 store in a memory a set of traffic rules associated with the specific location where the vehicle 14 is currently located. The set of traffic rules indicates regulatory rules, such as a speed limit associated with the road that the vehicle 14 is currently traveling on, and whether passing is allowed within the current driving lane.

[0060] The one or more autonomous driving controllers 52 of the voice-controlled autonomous driving system 12 compare the precompiled driving maneuver to the traffic rules associated with the specific location of the vehicle 14. In response to determining that the precompiled driving maneuver is not authorized based on the traffic rules associated with the specific location where the vehicle 14 is located, the one or more autonomous driving controllers 52 generate a legal message indicating that the precompiled driving maneuver is not authorized. Similarly, in response to determining that the precompiled driving maneuver is authorized based on the traffic rules associated with the specific location where the vehicle 14 is located, the one or more autonomous driving controllers 52 generate a legal message indicating that the precompiled driving maneuver is authorized.

[0061] The scoring module 64 of the one or more controllers 20 also receives an execution message indicating an execution status of the pre-compiled driving maneuver from the one or more autonomous driving controllers 52 of the voice-controlled autonomous driving system 12. The execution status indicates that the vehicle 14 has successfully completed the pre-compiled driving maneuver indicated by the voice command. The scoring module 64 of the one or more controllers 20 calculates a credibility score of the occupant 50 based on the pre-compiled driving maneuver indicated by the text-based command representing the voice command, the feasibility message indicating the feasibility of the pre-compiled driving maneuver, the legal message indicating the authorization status of the pre-compiled driving maneuver, and the execution message indicating the execution status of the pre-compiled driving maneuver from the one or more autonomous driving controllers 52 of the voice-controlled autonomous driving system 12, and the perception data from the one or more perception sensors 22.

[0062] In one embodiment, the credibility score is calculated based on Equation 1 as follows:

[0063] CreditabilityScore=w1S success -w2S stat -w3S infeasible -w4f user S consequence

[0064] Equation 1

[0065] Among them, w1, w2, w3 and w4 represent weight factors with values ​​ranging from 0 to 1. success represents a reward for each voice command generated by the occupant 50 that results in successful execution of a corresponding pre-compiled driving maneuver by the vehicle 14 and is determined based on the execution state indicated by the execution message. stat represents the legal penalty for each voice command generated by the passenger 50 that causes the vehicle 14 to not perform a corresponding pre-compiled driving maneuver because the pre-compiled driving maneuver is not authorized based on the traffic rules associated with the particular location where the vehicle 14 is located. Legal Penalty S statis determined based on the authorization status of the pre-compiled driving maneuver indicated by the statutory message. infeasible represents a feasibility penalty for each voice command generated by the occupant 50 that is not among the plurality of pre-compiled driving maneuvers and is not feasible. infeasible is determined based on the feasibility message. user represents a value indicating an error associated with the occupant 50 generating the voice command, in which case the pre-compiled driving maneuver was successfully executed but degraded the performance of the voice-controlled automated driving system 12 , and is determined based on Equation 2 described below.

[0066] S consequence represents a consequence penalty for each voice command generated by the occupant 50, which indicates the potential performance degradation caused by executing the pre-compiled driving maneuver and is generated by the scoring module 64 based on the perceptual sensors 22 ( Figure 1 ) collected by the vehicle 14. As described above, some examples of potential performance degradation caused by executing the pre-compiled driving maneuvers include, but are not limited to, driver takeover of the vehicle 14, active emergency braking or steering, not reaching the destination, significantly delaying the estimated time to reach the destination, causing a traffic incident (such as a collision), or executing a driving maneuver that reduces occupant comfort.

[0067] The calculation of the value f indicating an error associated with the occupant 50 generating the voice command in the event that the pre-compiled driving maneuver is successfully executed but the performance of the voice-controlled automated driving system 12 is degraded will now be described. user The value f user is determined based on feedback messages from one or more autonomous driving controllers 52 of the voice-controlled autonomous driving system 12. Specifically, the value f user is calculated based on a time interval measured from a first point in time when a pre-compiled driving maneuver is successfully executed to a second point in time when a performance degradation of the voice-controlled automated driving system 12 is detected, a change in a target state of a behavior planner of the voice-controlled automated driving system 12 created by a voice command generated by the occupant 50, and a degree of change between default values ​​of a plurality of behavior planner parameters and one or more updated behavior planner parameters, wherein the performance degradation, the default values ​​of the plurality of behavior planner parameters, and the one or more updated behavior planner parameters are received from one or more automated driving controllers 52. As the time interval value increases, the value f indicating an error associated with the occupant 50 generating the voice command increases. user It should be appreciated that as the degree of change between the plurality of behavior planner parameters and the one or more updated behavior planner parameters increases, the value f associated with the occupant 50 generating the voice command indicating an error user Also increased.

[0068] In one embodiment, the error-indicating value f associated with the occupant 50 is user is determined based on Equation 2, which is:

[0069] f user =f(time,distance goal ,deviation params ) Equation 2

[0070] Among them, time represents the time interval, distance goal Indicates the change of target state, and deviation params Represents the degree of change between multiple behavior planner parameters and one or more updated behavior planner parameters.

[0071] Figure 4 is a diagram illustrating a change in a target state of a behavior planner of a voice controlled automated driving system 12, wherein the change in target state is determined by comparing a distance D measured between an original target position 70 and a modified target position 72. The original target position 70 indicates a planned target state of the vehicle 14 initially determined by the behavior planner of the voice controlled automated driving system 12, and the modified target position 72 indicates a modified target state of the vehicle 14 achieved after executing a pre-compiled driving maneuver. It should be appreciated that the value f associated with the occupant 50 generating the voice command indicating an error user As the distance D measured between the original target position 70 and the modified target position 72 increases.

[0072] refer to Figure 2 , the access module 66 of the one or more controllers 20 receives the overall credibility score from the scoring module 64. The access module 66 of the one or more controllers 20 compares the overall credibility score of the occupant 50 to a threshold access score. The threshold access score indicates a minimum reliability of the occupant required to operate the voice controlled automated driving system 12. In response to determining that the overall credibility score of the occupant 50 is less than the threshold access score, the access module 66 of the one or more controllers 20 adjusts the access level of the occupant 50 to the voice controlled automated driving system 12.

[0073] In one embodiment, adjusting the occupant 50's level of access to the voice-controlled automated driving system 12 includes denying access to the voice-controlled automated driving system 12. For example, the access module 66 of the one or more controllers 20 may instruct the HMI 26 to generate a notification 78 to the occupant 50, wherein the notification 78 informs the occupant 50 that access to the voice-controlled automated driving system 12 is denied. In one embodiment, the notification 78 may inform the occupant 50 to take remedial action (e.g., take a specialized automated navigation course) to regain access to the voice-controlled automated driving system 12.

[0074] In another embodiment, adjusting the level of access of the occupant 50 to the voice-controlled automated driving system 12 includes assigning a lower level of access to the voice-controlled automated driving system 12, wherein the lower level of access only allows the occupant 50 to control the voice-controlled automated driving system 12 within a restricted area. The restricted area includes, but is not limited to, a restricted radius from the residence of the occupant 50 and roads with a speed limit below a threshold speed. For example, a lower level of access may allow the occupant 50 to control the voice-controlled automated driving system 12 on roads with a speed limit below highway speed (approximately 100 kilometers per hour or more). The access module 66 of the one or more controllers 20 may instruct the HMI 26 to generate a notification 78 to the occupant 50, wherein the notification 78 informs the occupant 50 that access to the voice-controlled automated driving system 12 has been denied or limited.

[0075] In response to determining that the overall credibility score of the occupant 50 is greater than the threshold access score, the access module 66 of the one or more controllers 20 compares the overall credibility score of the occupant 50 to a threshold warning score. The threshold warning score is greater than the threshold access score, indicating that the occupant's current credibility score is reduced and within a predefined margin of the threshold access score required to operate the voice-controlled automated driving system 12. In response to determining that the overall credibility score of the occupant 50 is less than the threshold warning score, the access module 66 of the one or more controllers 20 instructs the HMI 26 to generate a notification 78 to the occupant 50, wherein the notification 78 warns the occupant 50 that the current credibility score is reduced and may eventually be rejected.

[0076] Figure 5 is an exemplary process flow diagram illustrating a method 500 for scoring a voice command generated by an occupant 50. Referring generally to Figure 1 , Figure 2 and Figure 5 , the method 500 may start at block 502 . In block 502 , the authorization module 62 determines that the occupant 50 is an authorized user of the voice-controlled automated driving system 12 . Then, the method 500 may proceed to block 504 .

[0077] In block 504, the scoring module 64 of the one or more controllers 20 receives the pre-compiled driving maneuver indicated by the text-based command representing the voice command, a feasibility message indicating the feasibility of the pre-compiled driving maneuver, a statutory message indicating the authorization status of the pre-compiled driving maneuver, and an execution message indicating the execution status of the pre-compiled driving maneuver from the one or more autonomous driving controllers 52 of the voice-controlled autonomous driving system 12. The scoring module 64 of the one or more controllers 20 also receives perception data from the plurality of perception sensors 22. The method 500 may then proceed to decision block 506.

[0078] In decision block 506 , the scoring module 64 of the one or more controllers 20 determines whether the pre-compiled driving maneuver is feasible and legal based on the feasibility indicated by the feasibility message and the authorization status indicated by the legal message. In response to determining that the pre-compiled driving maneuver was not successfully executed, the method 500 may proceed to block 508 .

[0079] In block 508, the scoring module 64 of the one or more controllers 20 calculates a credibility score for the occupant 50. Specifically, the statutory penalty S for the credibility score is calculated. stat and feasibility penalty S infeasible The method 500 may then proceed to block 510 .

[0080] In block 510, the overall credibility score of the occupant 50 is updated based on the current credibility score determined by the scoring module 64 for the voice command being processed. It should be appreciated that in an embodiment, the overall credibility score of the occupant 50 is stored in one or more user databases 46 and downloaded and updated via the communication network 42. The method 500 may then proceed to decision block 512.

[0081] In decision block 512 , the access module 66 of the one or more controllers 20 compares the overall credibility score of the occupant 50 to the threshold access score. In response to determining that the overall credibility score of the occupant 50 is less than the threshold access score, the method 500 may then proceed to block 514 .

[0082] At block 514, the access module 66 of the one or more controllers 20 adjusts the access level of the occupant 50 to the voice-controlled automated driving system 12. In one embodiment, the access module 66 of the one or more controllers 20 may instruct the HMI 26 to generate a notification 78 informing the occupant 50 that access to the voice-controlled automated driving system 12 is denied. The method 500 may then terminate.

[0083] Referring to decision block 512 , in response to determining that the overall credibility score of the occupant 50 is greater than the threshold access score, the method proceeds to decision block 516 .

[0084] In decision block 516, the access module 66 of the one or more controllers 20 compares the overall credibility score of the occupant 50 to the threshold warning score. In response to determining that the overall credibility score is greater than the threshold warning score, the method terminates. In response to determining that the overall credibility score of the occupant 50 is less than the threshold warning score, the method proceeds to block 518.

[0085] In block 518 , the access module 66 of the one or more controllers 20 instructs the HMI 26 to generate a notification 78 to the occupant 50 , wherein the notification 78 warns the occupant 50 that the overall credibility score is decreasing and may ultimately be rejected. The method 500 may then terminate.

[0086] Returning to block 506, in response to determining that the pre-compiled driving maneuver is feasible and legal, the method 500 may proceed to block 520. In block 520, the scoring module 64 of the one or more controllers 20 determines a reward S based on the execution status indicated by the execution message. success Method 500 may then proceed to decision block 522 .

[0087] In decision block 522, the scoring module 64 determines whether the vehicle 14 successfully executed the pre-compiled driving maneuver based on the execution message (e.g., if the execution status is equal to 1, the pre-compiled driving maneuver was successfully executed, and if the execution status is equal to 0, the pre-compiled driving maneuver was not successfully executed). If the pre-compiled driving maneuver was not successfully executed, the method 500 returns to block 510, otherwise the method proceeds to decision block 524.

[0088] In decision block 524, the scoring module 64 of the one or more controllers 20 determines whether the voice command generated by the occupant 50 causes a performance degradation of the voice-controlled automated driving system 12. In response to determining that there is no performance degradation, the method 500 may terminate. Otherwise, the method 500 may proceed to block 526.

[0089] In block 526, the scoring module 64 of the one or more controllers 20 determines a consequence penalty S indicating a potential performance degradation caused by executing the pre-compiled driving maneuver. consequence and a value f associated with the occupant 50 generating the voice command indicating an error user Method 500 may then return to block 510 .

[0090] Referring generally to the accompanying drawings, the disclosed authentication system provides various technical effects and benefits. Specifically, the disclosed authentication system provides a method for verifying that an occupant is an authorized user of a voice-controlled automated driving system. If the occupant is an authorized user, the authentication system also provides a method for calculating a credibility score associated with the occupant. If the credibility score indicates that the occupant is not trained or does not have the level of responsibility required to control the trajectory of a vehicle based on voice commands, the disclosed authorization system can revoke or limit access to the voice-controlled automated driving system.

[0091] A controller may refer to an electronic circuit, a combinational logic circuit, a field programmable gate array (FPGA), a processor (shared processor, dedicated processor, or group processor) that executes code, or a combination of some or all of the above, such as in a system on a chip. In addition, the controller may be microprocessor-based, such as a computer having at least one processor, memory (RAM and / or ROM), and associated input and output buses. The processor may operate under the control of an operating system residing in the memory. The operating system may manage computer resources so that computer program code embodied as one or more computer software applications (such as applications residing in the memory) may have instructions executed by the processor. In an alternative embodiment, the processor may execute the application directly, in which case the operating system may be omitted.

[0092] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to fall within the scope of the present disclosure. These variations should not be regarded as departing from the spirit and scope of the present disclosure.

Claims

1. A certification system for a voice-controlled automatic driving system of a vehicle, the certification system comprising: a plurality of perception sensors for collecting perception data indicative of the vehicle's surroundings; one or more controllers in electronic communication with one or more autopilot controllers of the voice-controlled autopilot system and the plurality of perception sensors, wherein the voice-controlled autopilot system determines a trajectory of the vehicle based on voice commands generated by an occupant of the vehicle, wherein the one or more controllers execute instructions to: receiving an electrical signal representing the voice command generated by an occupant of the vehicle, wherein the voice command indicates a driving maneuver that changes the motion of the vehicle; converting the electrical signals into text-based commands based on one or more speech recognition algorithms; receiving a precompiled driving maneuver indicated by the text-based command, a feasibility message indicating feasibility of the precompiled driving maneuver, a statutory message indicating an authorization status of the precompiled driving maneuver, and an execution message from the one or more autonomous driving controllers indicating an execution status of the precompiled driving maneuver; calculating a confidence score for the occupant based on the precompiled driving maneuver, the feasibility of the precompiled driving maneuver, the authorization status of the precompiled driving maneuver, the execution status of the precompiled driving maneuver, and the perception data collected by the plurality of perception sensors, wherein an overall confidence score for the occupant quantifies the reliability of one or more speech-based commands generated by the occupant over time and is based on the confidence score; comparing the overall credibility score of the occupant to a threshold access score; and In response to determining that the overall confidence score of the occupant is less than the threshold access score, adjusting the occupant's level of access to the voice-controlled automated driving system.

2. The authentication system according to claim 1, wherein: The one or more controllers execute instructions to: The voice command represented by the text-based command is classified into one of: an emergency driving situation and a routine maneuver by identifying one or more emergency keywords present in the text-based command.

3. The authentication system according to claim 2, wherein: The one or more controllers execute instructions to: In response to classifying the voice command represented by the text-based command as an emergency driving situation, the text-based command is sent to the one or more autopilot controllers of the voice-controlled autopilot system without confirming that the occupant is an authorized user.

4. The authentication system according to claim 2, wherein: The one or more controllers execute instructions to: In response to determining that the voice command generated by the occupant is a routine driving maneuver, confirming that the occupant who generated the voice command is an authorized user of the voice-controlled automated driving system.

5. The authentication system according to claim 4, wherein: The one or more controllers are in electronic communication with an interior camera positioned to capture image data representative of a head and face of an occupant of the vehicle.

6. The authentication system according to claim 5, wherein: The one or more controllers wirelessly communicate with one or more user databases and a library of authorized digital images, wherein the one or more user databases store facial recognition data representing facial features and voice recognition data representing voices of all authorized users of the voice-controlled autonomous driving system, and each of the authorized digital images represents a digital image of an individual who has been previously authenticated as an authorized user of the voice-controlled autonomous driving system.

7. The authentication system according to claim 6, wherein: The one or more controllers confirm that the occupant generating the voice command is an authorized user of the voice-controlled automated driving system by: comparing the image data from the internal camera with the facial recognition data stored in the user database, and comparing the electrical signal representing the voice command with the voice recognition data stored in the user database; as well as In response to determining that there is no match between the image data representing the face of the occupant and the facial recognition data of one of the authorized users of the voice-controlled automated driving system stored in the user database, a request for a user-specific code is generated, the user-specific code being a unique identifier associated with an authorized digital image of the occupant stored in the one or more user databases.

8. The authentication system according to claim 7, wherein: The one or more controllers execute instructions to: comparing the authorized digital image of the occupant associated with the user-specific code to the library of authorized digital images stored in the one or more user databases; as well as In response to determining that the digital image of the occupant associated with the user-specific code does not match one of the authorized digital images stored in the one or more user databases, a procedure is performed to register the authorized digital image of the occupant.

9. The authentication system according to claim 1, wherein: The one or more controllers execute instructions to: The confidence score is calculated based on the following equation: CreditabilityScore=w1S success -w2S stat -w3S infeasible -w4f user S consequence Among them, w1, w2, w3 and w4 represent weight factors, S success represents a reward for each voice command generated by the occupant that results in successful execution of the corresponding pre-compiled driving maneuver by the vehicle, and S success Based on the execution status, S stat represents a statutory penalty for each voice command generated that causes the vehicle to not perform the corresponding pre-compiled driving maneuver because the pre-compiled driving maneuver is not authorized based on traffic rules associated with the specific location where the vehicle is located, S infeasible represents a feasibility penalty for each voice command generated by the occupant that causes the vehicle to not perform the driving maneuver because the precompiled driving maneuver is not feasible, user representing a value indicating an error associated with the occupant generating the voice command, in which case the pre-compiled driving maneuver was successfully executed but degraded the performance of the voice-controlled automated driving system, S consequence Represents a consequence penalty for each voice command generated by the occupant that indicates a degradation in performance of the voice-controlled automated driving system.

Images