Authentication system for a voice-controlled autonomous driving system

Abstract

An authentication system for a voice-controlled autonomous driving system comprises a variety of perceptual sensors for collecting perceptual data that defines the environment around the vehicle, and one or more controllers electronically connected to one or more autonomous driving controllers of the voice-controlled autonomous driving system and the variety of perceptual sensors. The voice-controlled autonomous driving system determines the vehicle's trajectory based on a voice command generated by a vehicle occupant. The one or more controllers execute instructions to calculate an occupant trust score, which quantifies the reliability of one or more voice-based commands generated by the occupant.In response to the determination that the overall occupant trustworthiness score is lower than the threshold access score, one or more controllers adjust the occupant's access level to the voice-activated autonomous driving system.

Description

introduction

[0001] The present disclosure relates to an authentication system for a voice-activated autonomous driving system of a vehicle, wherein the authentication system confirms that an occupant of the vehicle generating voice commands is an authorized user and determines an overall credibility score associated with the occupant.

[0002] Vehicles increasingly incorporate additional features related to the user experience. For example, an occupant can use voice commands to operate various functions and systems that are part of a vehicle. An example of a system that uses voice commands to control various features and systems of a vehicle is a voice-activated autonomous driving system. A voice-activated autonomous driving system includes a microphone that receives a voice command spoken by a driver or occupant of the vehicle, with the trajectory being determined based on the voice command. However, it should be noted that not all occupants of a vehicle may be trained or have the level of responsibility required to control the vehicle's trajectory based on voice commands.For example, one or more passengers in a vehicle may be a child or adolescent who lacks the necessary knowledge or maturity to operate the vehicle using voice commands. Furthermore, it should also be noted that some passengers may attempt to test the limits of an autonomous driving system based on voice commands.

[0003] Although voice-activated autonomous driving systems serve their intended purpose, there is a need for an improved approach to authenticate and verify users attempting to control the voice-activated autonomous driving system. Summary

[0004] According to several aspects, an authentication system for a voice-activated autonomous driving system of a vehicle is disclosed. The authentication system includes a plurality of perception sensors for collecting perception data indicative of an environment surrounding the vehicle, and one or more controllers in electronic communication with one or more autonomous driving controllers of the voice-activated autonomous driving system and the plurality of perception sensors. The voice-activated autonomous driving system determines a trajectory of the vehicle based on a voice command generated by an occupant of the vehicle. The one or more controllers execute instructions to receive an electrical signal representative of the voice command generated by a vehicle occupant, the voice command indicating a driving maneuver that alters the motion of the vehicle.The one or more controllers convert the electrical signal into a text-based command based on one or more speech recognition algorithms. The one or more controllers receive from the one or more autonomous driving controllers a pre-compiled driving maneuver specified by the text-based command, a feasibility message indicating a feasibility of the pre-compiled driving maneuver, a regulatory message indicating an authorization status of the pre-compiled driving maneuver, and an execution message indicating an execution status of the pre-compiled driving maneuver. The one or more controllers calculate a score.an occupant trustworthiness score based on the pre-compiled driving maneuver, the feasibility of the pre-compiled driving maneuver, the authorization status of the pre-compiled driving maneuver, the execution status of the pre-compiled driving maneuver, and the perception data collected by the plurality of perception sensors, wherein an overall occupant trustworthiness score quantifies a reliability of one or more voice-based commands generated by the occupant over time and is based on the trustworthiness score. The one or more controllers compare the overall occupant trustworthiness score to a threshold score for access ora threshold access score, and in response to determining that the overall trustworthiness score of the occupant is less than the threshold access score, set an access level of the voice-activated autonomous driving system for the occupant.

[0005] In another aspect, the one or more controllers execute instructions to classify the voice command represented by the text-based command as an emergency driving situation or a regular maneuver by identifying one or more emergency keywords appearing in the text-based command.

[0006] In yet another aspect, in response to classifying the voice command represented by the text-based command as an emergency driving situation, the one or more controllers transmit the text-based command to the one or more autonomous driving controllers of the voice-activated autonomous driving system without confirming that the occupant is an authorized user.

[0007] In one aspect, in response to determining that the voice command generated by the occupant is a regular driving maneuver, confirming that the occupant generating the voice command is an authorized user of the voice-activated autonomous driving system.

[0008] In another aspect, the one or more controllers are in electronic communication with an interior camera positioned to capture image data representative of a head and face of the vehicle occupant.

[0009] In yet another aspect, the one or more controllers are in wireless communication with one or more user databases that store facial recognition data representative of facial features and voice recognition data representative of a voice of all authorized users of the voice-activated autonomous driving system, as well as a library of authorized digital images, each representing a digital image of a person previously authenticated as an authorized user of the voice-activated autonomous driving system.

[0010] In one aspect, the one or more controllers confirm that the occupant generating the voice command is an authorized user of the voice-activated autonomous driving system by comparing the image data from the interior camera with the facial recognition data stored in the user databases, comparing the electrical signal representative of the voice command with the voice recognition data stored in the user databases, and in response to determining that there is no match between the image data representative of the occupant's face and the facial recognition data of one of the authorized users of the voice-activated autonomous driving system stored in the user databases, generating a request for a user-specific code that is a unique identifier associated with an authorized digital image of the occupant,which is stored in the one or more user databases.

[0011] In another aspect, the one or more controllers execute instructions to compare the authorized digital image of the occupant associated with the user-specific code with the library of authorized digital images stored in the one or more user databases, and in response to determining that the digital image of the occupant associated with the user-specific code does not match one of the authorized digital images stored in the one or more user databases, execute a procedure to register an authorized digital image of the occupant. In yet another aspect, the one or more controllers execute instructions to calculate the credibility score based on the following equation: Creditability Score=w1Ssuccess−w2Sstat−w3Sinfeasible−w4fuserSconsequence where w1, w2, w3, and w4 represent weighting factors, S success represents a reward for each voice command generated by the occupant that results in the corresponding pre-compiled driving maneuver being successfully executed by the vehicle, and is based on the execution status, S stat represents a statutory penalty for each voice command generated by the occupant that results in the corresponding pre-compiled driving maneuver not being executed by the vehicle because the pre-compiled driving maneuver is not permitted or authorized based on traffic regulations associated with a specific location where the vehicle is located, S infeasible represents a feasibility penalty for each voice command generated by the occupant that results in the driving maneuver not being executed by the vehicle because the pre-compiled driving maneuver is not feasible, f userrepresents a value indicating an error associated with the occupant generating the voice command in the event that the pre-compiled driving maneuver is successfully executed but the performance of the voice-controlled autonomous driving system deteriorates, and S consequence represents a consequence penalty for each voice command generated by the occupant that indicates a performance degradation of the voice-activated autonomous driving system.

[0012] In one aspect, the one or more controllers execute instructions to determine the value f user which indicates the fault associated with the occupant based on: fuser=f(time,distancegoal,deviationparams) where time represents a time interval, distance goal represents a change in a target state and deviation paramsrepresents a degree of change between a plurality of behavior planning parameters and one or more updated behavior planning parameters.

[0013] In another aspect, the change in the target state is determined by comparing a distance measured between an original target location indicating a planned target state of the vehicle originally determined by a behavior planner of the voice-controlled autonomous driving system and a modified target location indicating a modified target state of the vehicle reached after executing the pre-compiled driving maneuver.

[0014] In yet another aspect, the authorization status indicates when the pre-compiled driving maneuver is authorized based on traffic rules associated with a particular location where the vehicle is located.

[0015] In one aspect, the execution status indicates that the pre-compiled driving maneuver specified by the voice command was successfully completed by the vehicle.

[0016] In another aspect, setting the access level of the voice-activated autonomous driving system for the occupant includes denying access to the voice-activated autonomous driving system.

[0017] In yet another aspect, setting the access level of the voice-activated autonomous driving system for the occupant includes assigning a lower level of access to the voice-activated autonomous driving system, wherein the lower access level only allows the occupant to control the voice-activated autonomous driving system within restricted areas.

[0018] In one aspect, a method for calculating an overall trustworthiness score for an occupant of a vehicle by an authentication system for a voice-activated autonomous driving system is disclosed. The method includes receiving, by one or more controllers, an electrical signal representative of a voice command generated by an occupant of the vehicle, wherein the voice command indicates a driving maneuver that alters the motion of the vehicle, and wherein the one or more controllers are in electronic communication with one or more autonomous driving controllers of the voice-activated autonomous driving system and a plurality of perception sensors, and the voice-activated autonomous driving system determines a trajectory of the vehicle based on a voice command generated by an occupant of the vehicle.The method comprises converting, by means of the one or more controllers, the electrical signal into a text-based command based on one or more speech recognition algorithms. The method comprises receiving, by means of the one or more controllers, a pre-compiled driving maneuver indicated by the text-based command, a feasibility message indicating a feasibility of the pre-compiled driving maneuver, a legally required message indicating an authorization status of the pre-compiled driving maneuver, and an execution message indicating an execution status of the pre-compiled driving maneuver from the one or more autonomous driving controllers. The method also comprises calculating, by means of the one or more controllers, a score.an occupant trustworthiness score based on the pre-compiled maneuver, the feasibility of the pre-compiled maneuver, the authorization status of the pre-compiled maneuver, the execution status of the pre-compiled maneuver, and perception data collected from the plurality of perception sensors, wherein an occupant trustworthiness score quantifies a reliability of one or more voice-based commands generated by the occupant over time and is based on the trustworthiness score. The method includes comparing, by the one or more controllers, the overall occupant trustworthiness score to a threshold access score.Finally, in response to determining that the overall trustworthiness score of the occupant is less than the threshold access score, the method includes setting an access level of the voice-activated autonomous driving system for the occupant.

[0019] In one aspect, an authentication system for a voice-activated autonomous driving system of a vehicle is disclosed. The authentication system includes a plurality of perception sensors for collecting perception data indicative of an environment surrounding the vehicle, and one or more controllers in electronic communication with one or more autonomous driving controllers of the voice-activated autonomous driving system and the plurality of perception sensors. The voice-activated autonomous driving system determines a trajectory of the vehicle based on a voice command generated by an occupant of the vehicle, wherein the one or more controllers execute instructions to receive an electrical signal representative of the voice command generated by an occupant of the vehicle, the voice command indicating a driving maneuver that alters the motion of the vehicle.The one or more controllers convert the electrical signal into a text-based command based on one or more speech recognition algorithms. The one or more controllers receive a pre-compiled driving maneuver specified by the text-based command, a feasibility message indicating a feasibility of the pre-compiled driving maneuver, a regulatory message indicating an authorization status of the pre-compiled driving maneuver, and an execution message indicating an execution status of the pre-compiled driving maneuver from the one or more autonomous driving controllers, wherein the authorization status indicates when the pre-compiled driving maneuver is authorized based on traffic laws associated with a particular location where the vehicle is located.The one or more controllers calculate an occupant trustworthiness score based on the pre-compiled driving maneuver, the feasibility of the pre-compiled driving maneuver, the authorization status of the pre-compiled driving maneuver, the execution status of the pre-compiled driving maneuver, and the perception data collected from the plurality of perception sensors, wherein an occupant trustworthiness score quantifies a reliability of one or more voice-based commands generated by the occupant over time and is based on the trustworthiness score. The one or more controllers compare the occupant trustworthiness score to a threshold score for access ora threshold access score, and in response to determining that the occupant trustworthiness score is less than the threshold access score, the one or more controllers set an access level of the voice-activated autonomous driving system for the occupant, wherein the authorization status indicates when the pre-compiled driving maneuver is authorized based on traffic rules associated with a particular location where the vehicle is located.

[0020] In another aspect, the one or more controllers execute instructions to classify the voice command represented by the text-based command as an emergency driving situation or a regular maneuver by identifying one or more emergency keywords appearing in the text-based command.

[0021] In yet another aspect, in response to classifying the voice command represented by the text-based command as an emergency driving situation, the one or more controllers transmit the text-based command to the one or more autonomous driving controllers of the voice-activated autonomous driving system without confirming that the occupant is an authorized user.

[0022] In one aspect, in response to determining that the voice command generated by the occupant is a regular driving maneuver, the one or more controllers confirm that the occupant generating the voice command is an authorized user of the voice-activated autonomous driving system.

[0023] Further areas of applicability will become apparent from the description provided herein. It should be understood that the description and specific examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Brief description of the drawings

[0024] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way. Fig. 1 is a schematic diagram of a vehicle incorporating the disclosed authentication system including one or more controllers in electronic communication with a plurality of perception sensors and a voice-activated autonomous driving system, according to an exemplary embodiment; Fig. 2 illustrates the software architecture of the one or more Fig. 1 illustrated controller according to an exemplary embodiment; Fig. 3 illustrates a process flow diagram illustrating a method for confirming that an occupant generating a voice command is an authorized user of the voice-activated autonomous driving system, according to an example embodiment; Fig. 4 is a diagram illustrating a change in a target state of a behavior planner of the voice-controlled autonomous driving system according to an exemplary embodiment; and Fig. 5 is a process flow diagram illustrating a method for evaluating voice commands generated by the vehicle occupant according to an exemplary embodiment. Detailed description

[0025] The following description is merely exemplary in nature and is not intended to limit the disclosure, application, or uses.

[0026] Referring to Fig. 1 illustrates a schematic diagram of an exemplary authentication system 10 for a voice-activated autonomous driving system 12 of a vehicle 14. In embodiments, the voice-activated autonomous driving system 12 provides at least one Level 2 autonomous driving controller, such as an advanced driver assistance system (ADAS). It should be noted that the vehicle 14 may be any type of vehicle, such as, but not limited to, a sedan, truck, SUV, van, or RV. The authentication system 10 includes one or more controllers 20 in electronic communication with a plurality of perception sensors 22, a microphone 24, a human-machine interface (HMI) 26, and an interior camera 28. The plurality of perception sensors 22 are configured to collect perception data indicative of an environment surrounding the vehicle 14.In the non-limiting embodiment as set forth in . Fig. 1, the plurality of perception sensors 22 includes one or more cameras 30, an inertial measurement unit (IMU) 32, a global positioning system (GPS) 34, radar 36, and LiDAR 38; however, it is understood that additional sensors may be used.

[0027] The one or more controllers 20 are also in wireless communication with a backend server 40 via a communications network 42, wherein the backend server 40 is located remotely from the vehicle 14 as part of a backend office 44. The backend server 40 represents one or more computers, each including at least a processor, memory, associated input and output buses, and wireless communication capabilities. The backend server 40 is in electronic communication with one or more user databases 46, wherein the one or more user databases 46 store facial recognition data representative of facial features and voice recognition data representative of the voice of all authorized users of the voice-activated autonomous driving system 12. The one or more user databases 46 also store a library of authorized digital images.The authorized digital images each represent a digital image of a person previously authenticated as an authorized user of the voice-activated autonomous driving system 12. The one or more user databases 46 also store an overall trustworthiness score associated with an occupant 50 of the vehicle 14, the overall trustworthiness score being described below.

[0028] The voice-activated autonomous driving system 12 determines a trajectory 48 of the vehicle 14 based on a voice command generated by an occupant 50 of the vehicle 14. The voice command generated by the occupant 50 of the vehicle 14 signals or indicates a driving maneuver that changes the motion of the vehicle 14 while the vehicle 14 is currently traveling. In other words, the voice command generated by the occupant 50 changes the current trajectory of the vehicle 14. Some examples of the voice command include, but are not limited to, a command to take the next exit on a highway, change lanes, overtake another vehicle, and change the destination of a route plan.

[0029] As explained below, the authentication system 10 confirms that the occupant 50 generating the voice commands that determine the trajectory 48 of the vehicle 14 is an authorized user of the voice-activated autonomous driving system 12. It should be understood that the voice-activated autonomous driving system 12 only accepts voice commands generated by authorized users unless the voice command designates or signals an emergency driving situation. In one embodiment, an emergency driving situation is a maneuver performed in response to a sudden, unexpected situation, such as when another vehicle is about to intersect the path of the host vehicle (i.e., the vehicle 14). In another embodiment, the emergency driving situation is when instructions are received to immediately stop the vehicle 14, such as when the occupant 50 indicates that he or she is experiencing motion sickness.

[0030] Once the occupant 50 is confirmed as an authorized user, the one or more controllers 20 of the authentication system 10 evaluate the voice commands generated by the occupant 50. As explained further below, the authentication system 10 sets an access level of the voice-enabled autonomous driving system 12 for the occupant 50 in response to determining that an overall trustworthiness score associated with the occupant 50 falls below a threshold access score. For example, the occupant 50 may generate voice commands that cause the vehicle 14 to perform driving maneuvers that are unauthorized, infeasible, or that result in a degraded performance of the voice-enabled autonomous driving system 12.Some examples of performance degradation of the voice-activated autonomous driving system 12 include, but are not limited to, takeover by a driver of the vehicle 14, active emergency braking or steering, failure to reach a destination, a significant delay in the estimated time of arrival at the destination, causing a traffic incident such as a collision, or performing a driving maneuver that compromises the comfort of the occupants.

[0031] The microphone 24 represents a device that converts sound waves into electrical signals, where the electrical signals are received by the one or more controllers 20. Specifically, the microphone 24 converts a voice command generated by an occupant 50 of the vehicle 14 into an electrical signal representative of the voice command. The HMI 26 represents a user interface through which the occupant 50 of the vehicle 14 receives a message from and provides user input to the voice-activated autonomous driving system 12. An example of the HMI 26 is a touchscreen that receives user input and displays images on a display to the occupant 50. The interior camera 28 is positioned to capture image data representative of the head and face of the occupant 50 of the vehicle 14.

[0032] Fig. 2 illustrates the software architecture of the one or more Fig. 1, wherein the one or more controllers 20 are in electronic communication with one or more autonomous driving controllers 52 that are part of the voice-activated autonomous driving system 12 ( Fig. 1). The one or more controllers 20 of the authentication system 10 include a speech recognition module 60, an authorization module 62, a scoring module 64, and an access module 66. In one embodiment, the speech recognition module 60 of the one or more controllers 20 receives the electrical signal representative of the voice command generated by the occupant 50 from the microphone 24 and converts the electrical signal into a text-based command based on one or more speech recognition algorithms such as perceptual linear prediction (PLP) features, Viterbi search, and speech recognition approaches using deep neural networks and discrimination training.

[0033] The speech recognition module 60 then classifies the voice command represented by the text-based command as either an emergency driving situation or a normal driving maneuver by identifying one or more emergency keywords present in the text-based command. The emergency keywords indicate that an occupant of the vehicle 14 is attempting to perform an emergency driving situation. Some examples of one or more emergency keywords include the words "stop," "help," "emergency," or phrases indicating that the occupant 50 is attempting to stop the vehicle 14 immediately, such as "I feel sick and sick, stop." In response to determining that one or more emergency keywords are present in the text-based command, the speech recognition module 60 classifies the voice command as an emergency driving situation.

[0034] As mentioned above, the voice-activated autonomous driving system 12 only accepts voice commands generated by authorized users unless the voice command signals or indicates an emergency driving situation. In response to the voice recognition module 60 classifying the voice command represented by the text-based command as an emergency driving situation, the voice recognition module 60 transmits the text-based command to the one or more autonomous driving controllers 52 of the voice-activated autonomous driving system 12 without confirming that the occupant 50 is an authorized user. Otherwise, the authorization module 62 of the one or more controllers 20 confirms that the occupant 50 generating the voice command is an authorized user of the voice-activated autonomous driving system 12.

[0035] The authorization module 62 of the one or more controllers 20 receives the image data from the interior camera 28 representative of the face of the occupant 50 of the vehicle 14 and the voice command represented by the text-based command generated by the occupant 50 from the voice recognition module 60. The authorization module 62 of the one or more controllers 20 also receives the facial recognition data representative of the facial features and the voice recognition data representative of the voice of all the authorized users of the voice-activated autonomous driving system 12, as well as the library of authorized digital images from the one or more user databases 46 via the communication network 42.In response to the speech recognition module 60 determining that the voice command generated by the occupant 50 is a regular driving maneuver, the authorization module 62 confirms that the occupant 50 generating the voice command is an authorized user of the voice-activated autonomous driving system 12 by comparing the image data from the interior camera 28 with the facial recognition data stored in the user databases 46 and the electrical signal representative of the voice command from the microphone 24 with the speech recognition data stored in the user databases 46, a method for confirming the occupant 50 being described in . Fig. 3 is illustrated.

[0036] Referring now to Fig. 3, an exemplary process flow diagram is presented illustrating a method 300 for confirming that the occupant 50 generating the voice command is an authorized user of the voice-activated autonomous driving system 12. With general reference to Fig. 1-3, the method 300 may begin with block 302. In block 302, the authorization module 62 of the one or more controllers 20 receives the image data from the interior camera 28. The method 300 may then proceed to a decision block 304.

[0037] At decision block 304, the authorization module 62 of the one or more controllers 20 executes one or more facial recognition algorithms that compare the image data from the interior camera 28 representative of the face of the occupant 50 of the vehicle 14 with the facial recognition data stored in the user databases 46. In response to determining a match between the image data representative of the face of the occupant 50 of the vehicle 14 and the facial recognition data of one of the authorized users of the voice-activated autonomous driving system 12 stored in the user databases 46, the method 300 may proceed to block 306. Otherwise, the method proceeds to block 314.

[0038] In block 306, the authorization module 62 of the one or more controllers 20 receives the electrical signal representative of the voice command generated by the occupant 50 from the microphone 24. The method 300 may then proceed to decision block 308.

[0039] At decision block 308, the authorization module 62 of the one or more controllers 20 executes one or more speech recognition algorithms that compare the electrical signal from the microphone 24 representative of the voice command generated by the occupant 50 with the speech recognition data representative of the voice of all authorized users of the voice-activated autonomous driving system 12 stored in the one or more user databases 46. In response to determining a match between the voice command generated by the occupant 50 from the microphone 24 and the speech recognition data representative of the voice of all authorized users of the voice-activated autonomous driving system 12, the method 300 may proceed to a decision block 310. Otherwise, the method 300 may abort or terminate.

[0040] At decision block 310, the authorization module 62 of the one or more controllers 20 confirms that the occupant 50 is still within an interior of the vehicle 14. The authorization module 62 of the one or more controllers 20 confirms that the occupant 50 is still within the interior of the vehicle 14 based on a variety of approaches, such as determining whether a door of the vehicle 14 has been opened, using one or more infrared sensors located within the interior of the vehicle 14, or monitoring the occupant 50 via the interior camera 28. In response to determining that the occupant 50 has exited the vehicle 14, the method ends. Otherwise, the method 300 proceeds to block 312.

[0041] At block 312, the authorization module 62 of the one or more controllers 20 confirms that the occupant 50 generating the voice command is an authorized user of the voice-activated autonomous driving system 12. The authorization module 62 also transmits a notification to the one or more autonomous driving controllers 52 of the voice-activated autonomous driving system 12 indicating that the occupant 50 is also an authorized user. The method 300 may then end.

[0042] Returning to decision block 304, in response to determining that there is no match between the image data representative of the face of the occupant 50 of the vehicle 14 and the facial recognition data of one of the authorized users of the voice-activated autonomous driving system 12 stored in the user databases 46, the method 300 may proceed to block 314. At block 314, the authorization module 62 of the one or more controllers 20 instructs the HMI 26 to generate a notification to the occupant 50, the notification prompting the occupant 50 to provide user input to the HMI 26 indicating a user-specific code. The user-specific code is a unique identifier associated with an authorized digital image of the occupant 50 stored in one or more user databases 46.In one embodiment, the user-specific code represents biometric data such as fingerprints or a retinal scan. The occupant 50 must register the authorized digital image within the one or more user databases 46 before executing the method 300. In one embodiment, the user-specific code is a numeric passcode transmitted from the backend server 40 to a smartphone of the occupant 50. The method 300 may then proceed to a decision block 316.

[0043] At decision block 316, the authorization module 62 of the one or more controllers 20 compares the image data representative of the face of the occupant 50 with the library of authorized digital images stored in the one or more user databases 46, where the authorized digital images each represent a digital image of a person previously authenticated as an authorized user of the voice-activated autonomous driving system 12. In response to determining that the image data representative of the face of the occupant 50 associated with the user-specific code matches one of the authorized digital images stored in the one or more user databases 46, the method 300 may proceed to a decision block 318.In block 318, the authorization module 62 checks a memory of the one or more controllers 20 for voice data representative of the voice of the occupant 50. If no recording exists, the method 300 proceeds to block 320, and the authorization module 62 instructs the HMI 26 to generate a prompt requesting the occupant 50 to create a recording. Otherwise, the method returns to block 306.

[0044] Referring to decision block 316, in response to determining that the image data representative of the face of the occupant 50 does not match one of the authorized digital images stored in the one or more user databases 46, the method 300 may proceed to block 322. At block 320, the authorization module 62 executes a procedure to register an authorized digital image of the occupant 50 within the library stored in the one or more user databases 46. The procedure includes asking the occupant 50 to distinguish themselves from the other occupants of the vehicle 14 by performing actions such as raising their hand. The method 300 may then proceed to block 322 where an authorized digital image of the occupant 50 is added to the library stored in the one or more user databases 46.The method 300 may then return to a decision block 318.

[0045] Referring to Fig. 2, in response to confirming that the occupant 50 generating the voice command is an authorized user of the voice-activated autonomous driving system 12, the authorization module 62 of the one or more controllers 20 transmits a confirmation message and the text-based command representing the voice command to the one or more controllers 20 of the voice-activated autonomous driving system 12, wherein the confirmation message indicates that the occupant 50 is an authorized user.As explained below, the scoring module 64 of the one or more controllers 20 receives a pre-compiled driving maneuver indicated by the text-based command representing the voice command, a feasibility message indicating feasibility of the pre-compiled driving maneuver, a regulatory message indicating authorization status of the pre-compiled driving maneuver, and an execution message indicating execution status of the pre-compiled driving maneuver from the one or more autonomous driving controllers 52 of the voice-enabled autonomous driving system 12. As explained below, in response to determining that the pre-compiled driving maneuver is successfully executed, the scoring module 64 calculates a consequence penalty (COP).: consequence penalty) indicating a potential performance degradation caused by executing the pre-compiled driving maneuver based on the perception data acquired by the plurality of perception sensors 22 (. Fig. 1).

[0046] The scoring module 64 calculates the credibility score of the occupant 50 based on the inputs from the one or more autonomous driving controllers 52 of the voice-activated autonomous driving system 12 and the perception data collected by the plurality of perception sensors 22. The credibility score of the occupant 50 is combined with the previous trustworthiness score associated with the occupant 50 stored in the one or more user databases 46. The overall trustworthiness score quantifies the reliability of the occupant 50 over time.For example, if the occupant 50 continually gives voice-based commands instructing the vehicle 14 to increase its speed beyond the current speed limit or to perform a driving maneuver during autonomous driving that requires driver intervention, then the overall trustworthiness rating of the occupant 50 would decrease or decrease.

[0047] The one or more autonomous driving controllers 52 of the voice-activated autonomous driving system 12 classify the text-based command representing the voice command into one of a plurality of pre-compiled driving maneuvers based on one or more machine learning algorithms in response to the authorization module 62 of the one or more controllers 20 receiving the confirmation message. In one embodiment, the one or more autonomous driving controllers 52 classify the text-based command based on a neural network trained to classify the text-based command. The pre-compiled driving maneuvers represent a list of potential driving maneuvers that the vehicle 14 can perform.Some examples of potential maneuvers include, but are not limited to, a command to take the next exit, change lanes (either left or right), and overtake another vehicle (either in the left or right lane). It should be understood that in some cases, the text-based command represents a maneuver that cannot be performed or is not feasible. For example, if the voice command instructs vehicle 14 to "fly over" traffic, this would be an infeasible maneuver.

[0048] In the event that the one or more autonomous driving controllers 52 are unable to classify the text-based command representing the voice command into one of the plurality of pre-compiled driving maneuvers because the driving maneuver is not feasible, the one or more autonomous driving controllers 52 transmit a feasibility message to the scoring module 64 of the one or more controllers 20 indicating that the pre-compiled driving maneuver cannot be accurately classified into one of the pre-compiled driving maneuvers and is not feasible.In response to the text-based command representing the voice command being classified into one of the plurality of pre-compiled driving maneuvers, the one or more autonomous driving controllers 52 transmit a feasibility message to the scoring module 64 of the one or more controllers 20 indicating that the pre-compiled driving maneuver is feasible.

[0049] In addition to the pre-compiled maneuver, the one or more autonomous driving controllers 52 of the voice-activated autonomous driving system 12 also transmit a legally required message to the scoring module 64 of the authentication system 10, wherein the legally required message indicates an authorization status of the pre-compiled maneuver. The authorization status indicates when the pre-compiled maneuver is authorized based on the traffic rules associated with the specific location where the vehicle 14 is located. The one or more autonomous driving controllers 52 of the voice-activated autonomous driving system 12 stores a set of traffic rules associated with a specific location where the vehicle 14 is currently located in memory.The set of traffic rules specifies regulatory requirements such as, for example, a speed limit for the road on which the vehicle 14 is currently traveling and whether overtaking is permitted within a current lane.

[0050] The one or more autonomous driving controllers 52 of the voice-activated autonomous driving system 12 compare the pre-compiled driving maneuver to the traffic rules associated with the specific location of the vehicle 14. In response to determining that the pre-compiled driving maneuver is unauthorized based on the traffic rules associated with the specific location of the vehicle 14, the one or more autonomous driving controllers 52 generate a legally required message indicating that the pre-compiled driving maneuver is unauthorized. Similarly, in response to determining that the pre-compiled driving maneuver is permitted based on the traffic rules associated with the specific location of the vehicle 14, the one or more autonomous driving controllers 52 generate a legally required message indicating that the pre-compiled driving maneuver is unauthorized.authorized, a legally required message indicating that the precompiled maneuver is authorized.

[0051] The scoring module 64 of the one or more controllers 20 also receives the execution message indicating an execution status of the pre-compiled driving maneuver from the one or more autonomous driving controllers 52 of the voice-activated autonomous driving system 12. The execution status indicates that the pre-compiled driving maneuver specified by the voice command was successfully completed by the vehicle 14.The scoring module 64 of the one or more controllers 20 calculates the trustworthiness score of the occupant 50 based on the pre-compiled driving maneuver indicated by the text-based command representing the voice command, the feasibility message indicating the feasibility of the pre-compiled driving maneuver, the regulatory message indicating the authorization status of the pre-compiled driving maneuver, and the execution message indicating the execution status of the pre-compiled driving maneuver from the one or more autonomous driving controllers 52 of the voice-activated autonomous driving system 12, and the perception data from the one or more perception sensors 22.

[0052] In one embodiment, the credibility score is calculated based on Equation 1, which is as follows: Creditability Score=w1Ssuccess−w2Sstat−w3Sinfeasible−w4fuserSconsequence where w1, w2, w3, and w4 represent weighting factors that have a value in the range from 0 to 1. S success represents a reward for each voice command generated by the occupant 50 that results in the corresponding precompiled driving maneuver being successfully executed by the vehicle 14, and is determined based on the execution status indicated by the execution message. S stat represents a statutory penalty for each voice command generated by the occupant 50 that results in the corresponding pre-compiled maneuver not being executed by the vehicle 14 because, based on the traffic regulations associated with the specific location where the vehicle 14 is located, the pre-compiled maneuver is not authorized. The statutory penalty S statis determined based on the authorization status of the pre-compiled driving maneuver, which is indicated by the legally required message. S infeasible represents a feasibility penalty for each voice command generated by the occupant 50 that is not part of the multitude of pre-compiled driving maneuvers and is not feasible. The feasibility penalty S infeasible will be determined based on the feasibility report. f user represents a value indicative of an error associated with the occupant 50 generating the voice command in the event that the pre-compiled driving maneuver is successfully executed but the performance of the voice-activated autonomous driving system 12 deteriorates, and is determined based on Equation 2 explained below.

[0053] S consequencerepresents the consequence penalty for each voice command generated by the occupant 50, indicating the potential performance degradation caused by executing the pre-compiled driving maneuver, and is calculated by the scoring module 64 based on the perception data collected by the perception sensors 22 ( Fig. 1). As noted above, some examples of potential performance degradation caused by executing the pre-compiled driving maneuver include, but are not limited to, taking over by a driver of the vehicle 14, active emergency braking or steering, failure to reach a destination, a significant delay in the estimated time of arrival at the destination, causing a traffic incident such as a collision, or executing a driving maneuver that impairs occupant comfort.

[0054] Calculating the value f user, which indicates the error associated with the occupant 50 generating the voice command if the precompiled driving maneuver is successfully executed, but the performance of the voice-controlled autonomous driving system 12 deteriorates, will now be described. The value f user is determined based on feedback messages from the one or more autonomous driving controllers 52 of the voice-activated autonomous driving system 12. Specifically, the value f usercalculated based on a time interval measured between a first point in time at which the pre-compiled driving maneuver is successfully executed and a second point in time at which a performance degradation of the voice-controlled autonomous driving system 12 is detected, a change in a target state of a behavior planner of the voice-controlled autonomous driving system 12 generated by the voice command generated by the occupant 50, and a degree of change between a default value for a plurality of behavior planning parameters and one or more updated behavior planning parameters, wherein the performance degradation, the default values ​​of the plurality of behavior planning parameters, and the one or more updated behavior planning parameters are received from the one or more autonomous driving controllers 52. As the value of the time interval increases, the value f user, which indicates the error associated with the occupant 50 generating the voice command. It is understood that with the degree of change between the plurality of behavior planning parameters and the one or more updated behavior planning parameters, the value f user also increases, which indicates the error associated with the occupant 50 generating the voice command.

[0055] In one embodiment, the value f user , which indicates the error associated with the occupant 50, is determined based on equation 2, which is as follows: fuser=f(time,distancegoal,deviationparams) where time represents the time interval, distance goal represents the change in the target state and deviation params represents the degree of change between the plurality of behavior planning parameters and the one or more updated behavior planning parameters.

[0056] Fig. 4 is a diagram illustrating the change in the target state of the behavior planner of the voice-controlled autonomous driving system 12, wherein the change in the target state is determined by comparing a distance D measured between an original target location 70 and a changed target location 72. The original target location 70 indicates a planned target state of the vehicle 14 that was originally determined by the behavior planner of the voice-controlled autonomous driving system 12, and the changed target location 72 indicates a changed target state of the vehicle 14 that is reached after executing the pre-compiled driving maneuver. It should be understood that the value f user , which indicates the error associated with the occupant 50 generating the voice command, increases with increasing distance D between the original target location 70 and a modified target location 72.

[0057] Referring to Fig. 2, the access module 66 of the one or more controllers 20 receives the overall trustworthiness score from the scoring module 64. The access module 66 of the one or more controllers 20 compares the overall trustworthiness score of the occupant 50 to a threshold access score. The threshold access score indicates a minimum occupant reliability required to operate the voice-activated autonomous driving system 12. In response to determining that the overall trustworthiness score of the occupant 50 is less than the threshold access score, the access module 66 of the one or more controllers 20 sets an access level of the voice-activated autonomous driving system 12 for the occupant 50.

[0058] In one embodiment, adjusting the access level of the voice-activated autonomous driving system 12 for the occupant 50 includes denying access to the voice-activated autonomous driving system 12. For example, the access module 66 of the one or more controllers 20 may instruct the HMI 26 to generate a notification 78 to the occupant 50, wherein the notification 78 informs the occupant 50 that access to the voice-activated autonomous driving system 12 is denied. In one embodiment, the notification 78 may instruct the occupant 50 to take remedial action to regain access to the voice-activated autonomous driving system 12, such as, for example, participating in specialized autonomous navigation courses.

[0059] In another embodiment, setting the access level of the voice-activated autonomous driving system 12 for the occupant 50 includes assigning a lower level of access to the voice-activated autonomous driving system 12, wherein the lower access level only allows the occupant 50 to control the voice-activated autonomous driving system 12 within restricted areas. The restricted areas include, but are not limited to, a restricted radius around the residence of the occupant 50 and roads with a speed limit below a threshold speed. For example, the lower access level may allow the occupant 50 to control the voice-activated autonomous driving system 12 on roads with a speed limit below a highway speed (which is approximately 100 kilometers per hour or more).The access module 66 of the one or more controllers 20 may instruct the HMI 26 to generate a notification 78 to the occupant 50, wherein the notification 78 informs the occupant 50 that access to the voice-activated autonomous driving system 12 has been denied or restricted.

[0060] In response to determining that the overall trustworthiness score of the occupant 50 is greater than the threshold access score, the access module 66 of the one or more controllers 20 compares the overall trustworthiness score of the occupant 50 to a threshold warning score. The threshold warning score is greater than the threshold access score and indicates that a current occupant trustworthiness score is reduced and within a predefined range of the threshold access score required to operate the voice-activated autonomous driving system 12.In response to determining that the overall trustworthiness score of the occupant 50 is less than the threshold warning score, the access module 66 of the one or more controllers 20 instructs the HMI 26 to generate a notification 78 to the occupant 50, wherein the notification 78 warns the occupant 50 that the current trustworthiness score is reduced and may be denied.

[0061] Fig. 5 is an exemplary process flow diagram illustrating a method 500 for scoring the voice commands generated by the occupant 50. With general reference to Fig. 1, Fig. 2 and Fig. 5, the method 500 may begin in block 502. In block 502, the authorization module 62 determines that the occupant 50 is an authorized user of the voice-activated autonomous driving system 12. The method 500 may then proceed to block 504.

[0062] In block 504, the scoring module 64 of the one or more controllers 20 receives the pre-compiled driving maneuver indicated by the text-based command representing the voice command, the feasibility message indicating the feasibility of the pre-compiled driving maneuver, the regulatory message indicating the authorization status of the pre-compiled driving maneuver, and the execution message indicating the execution status of the pre-compiled driving maneuver from the one or more autonomous driving controllers 52 of the voice-enabled autonomous driving system 12. The scoring module 64 of the one or more controllers 20 also receives the perception data from the plurality of perception sensors 22. The method 500 may then proceed to a decision block 506.

[0063] At decision block 506, the evaluation module 64 of the one or more controllers 20 determines whether the pre-compiled maneuver was feasible and permissible based on the feasibility indicated by the feasibility message and the authorization status indicated by the legally required message. In response to determining that the pre-compiled maneuver was not successfully executed, the method 500 may proceed to block 508.

[0064] In block 508, the scoring module 64 of the one or more controllers 20 calculates the score or rating of the trustworthiness of the occupant 50. Specifically, the legally prescribed penalty S stat and the feasibility penalty S infeasible the trustworthiness score is calculated. The method 500 may then proceed to block 510.

[0065] At block 510, the overall trustworthiness score of the occupant 50 is updated based on the current trustworthiness score determined by the scoring module 64 for the voice command being processed. It should be understood that in embodiments, the overall trustworthiness score of the occupant 50 is stored in the one or more user databases 46 and downloaded and updated via the communications network 42. The method 500 may then proceed to a decision block 512.

[0066] At decision block 512, the access module 66 of the one or more controllers 20 compares the overall trustworthiness score of the occupant 50 with the threshold access score. In response to determining that the overall trustworthiness score of the occupant 50 is less than the threshold access score, the method 500 may then proceed to block 514.

[0067] In block 514, the access module 66 of the one or more controllers 20 sets the access level of the voice-activated autonomous driving system 12 for the occupant 50. In one embodiment, the access module 66 of the one or more controllers 20 may instruct the HMI 26 to generate a notification 78 informing the occupant 50 that access to the voice-activated autonomous driving system 12 is denied. The method 500 may then end.

[0068] Referring to decision block 512, in response to determining that the overall trustworthiness score of the occupant 50 is greater than the threshold access value, the method proceeds to decision block 516.

[0069] In decision block 516, the access module 66 of the one or more controllers 20 compares the overall trustworthiness score of the occupant 50 with the threshold warning score. In response to determining that the overall trustworthiness score is greater than the threshold warning score, the method ends. In response to determining that the overall trustworthiness score of the occupant 50 is less than the threshold warning score, the method proceeds to block 518.

[0070] At block 518, the access module 66 of the one or more controllers 20 instructs the HMI 26 to generate a notification 78 to the occupant 50, wherein the notification 78 warns the occupant 50 that the overall trustworthiness score has been reduced and may be denied. The method 500 may then end.

[0071] Returning to block 506, in response to determining that the pre-compiled driving maneuver was feasible and legally permitted or required, the method 500 may proceed to block 520. In block 520, the scoring module 64 of the one or more controllers 20 determines the reward S success based on the execution status indicated by the execution message. The method 500 may then proceed to decision block 522.

[0072] In decision block 522, the evaluation module 64 determines, based on the execution message, whether the precompiled maneuver was successfully executed by the vehicle 14 (e.g., if the execution status is 1, the precompiled maneuver was successfully executed; if the execution status is 0, the precompiled maneuver was not successfully executed). If the precompiled maneuver was not successfully executed, the method 500 returns to block 510; otherwise, the method proceeds to a decision block 524.

[0073] At decision block 524, the evaluation module 64 of the one or more controllers 20 determines whether the voice command generated by the occupant 50 resulted in a performance degradation of the voice-activated autonomous driving system 12. If it is determined that no performance degradation occurred, the method 500 may end. Otherwise, the method 500 may proceed to block 526.

[0074] In block 526, the evaluation module 64 of the one or more controllers 20 determines the consequence penalty S consequence , which indicates the potential performance degradation caused by executing the precompiled maneuver, and the value f user indicating the error associated with the occupant 50 generating the voice command. The method 500 may then return to block 510.

[0075] Referring generally to the figures, the disclosed authentication system offers various technical effects and advantages. Specifically, the disclosed authentication system provides an approach to verify whether the occupant is an authorized user of the voice-activated autonomous driving system. In the event that the occupant is an authorized user, the authentication system also provides an approach to determine a trustworthiness score associated with the occupant. In the event that the trustworthiness score indicates that the occupant is not trained or does not have the level of responsibility required to control the vehicle's trajectory based on voice commands, the disclosed authorization system can either revoke or restrict access to the voice-activated autonomous driving system.

[0076] The controllers may refer to or be part of an electronic circuit, a combinational logic circuit, a field-programmable gate array (FPGA), a processor (shared, dedicated, or clustered) executing code, or a combination of some or all of the above elements. Furthermore, the controllers may be based on a microprocessor, such as a computer having at least one processor, memory (RAM and / or ROM), and associated input and output buses. The processor may operate under the control of an operating system residing in memory. The operating system may manage computer resources such that computer program code embodied as one or more computer software applications, such as an application residing in memory, may include instructions executed by the processor.In an alternative embodiment, the processor may execute the application directly, in which case the operating system may be omitted.

[0077] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to be within the scope of the present disclosure. Such variations should not be considered a departure from the spirit and scope of the present disclosure. legend

[0078] In the drawings, N stands for No and Y stands for Yes.

Claims

[1] Authentication system for a voice-controlled system for autonomous driving of a vehicle, the authentication system comprising: a plurality of perception sensors to collect perception data indicative of an environment around the vehicle; one or more controllers in electronic communication with one or more several autonomous driving controllers of the voice-controlled autonomous driving system and the multitude of perception sensors, wherein the voice-activated autonomous driving system determines a trajectory of the vehicle based on a voice command generated by an occupant of the vehicle, wherein the one or more controllers execute instructions to: receive an electrical signal representative of the voice command generated by a vehicle occupant, the voice command indicating a driving maneuver that changes the movement of the vehicle; convert the electrical signal into a text-based command based on one or more speech recognition algorithms; receive from the one or more autonomous driving controllers a precompiled driving maneuver specified by the text-based command, a feasibility message indicating a feasibility of the precompiled driving maneuver, a regulatory message indicating an authorization status of the precompiled driving maneuver, and an execution message indicating an execution status of the precompiled driving maneuver; calculate an occupant trustworthiness score based on the pre-compiled driving maneuver, the feasibility of the pre-compiled driving maneuver, the authorization status of the pre-compiled driving maneuver, the execution status of the pre-compiled driving maneuver, and the perception data collected by the plurality of perception sensors, wherein an overall occupant trustworthiness score quantifies a reliability of one or more voice-based commands generated by the occupant over time and is based on the trustworthiness score; compare the occupant's overall trustworthiness score with a threshold access score; and in response to determining that the overall trustworthiness score of the occupant is less than the threshold access score, set an access level of the voice-activated autonomous driving system for the occupant. [2] The authentication system of claim 1, wherein the one or more controllers execute instructions to: to classify the voice command represented by the text-based command as an emergency driving situation or a regular maneuver by identifying one or more emergency keywords occurring in the text-based command. [3] The authentication system of claim 2, wherein the one or more controllers execute instructions to: in response to classifying the voice command represented by the text-based command as an emergency driving situation, transmit the text-based command to the one or more autonomous driving controllers of the voice-activated autonomous driving system without confirming that the occupant is an authorized user. [4] The authentication system of claim 2, wherein the one or more controllers execute instructions to: in response to determining that the voice command generated by the occupant is a regular driving maneuver, confirm that the occupant generating the voice command is an authorized user of the voice-activated autonomous driving system. [5] The authentication system of claim 4, wherein the one or more controllers are in electronic communication with an interior camera positioned to capture image data representative of a head and a face of the vehicle occupant. [6] The authentication system of claim 5, wherein the one or more controllers are in wireless communication with one or more user databases that store facial recognition data representative of facial features and voice recognition data representative of a voice of all authorized users of the voice-activated autonomous driving system, and a library of authorized digital images, each representing a digital image of a person previously authenticated as an authorized user of the voice-activated autonomous driving system. [7] The authentication system of claim 6, wherein the one or more controllers confirm that the occupant generating the voice command is an authorized user of the voice-activated autonomous driving system by: Comparing the image data from the interior camera with the facial recognition data stored in the user databases, the electrical signal representative of the voice command with the voice recognition data stored in the user databases; and in response to determining that there is no match between the image data representative of the occupant's face and the facial recognition data of one of the authorized users of the voice-activated autonomous driving system stored in the user databases, generate a request for a user-specific code that is a unique identifier associated with an authorized digital image of the occupant stored in the one or more user databases. [8] The authentication system of claim 7, wherein the one or more controllers execute instructions to: compare the authorized digital image of the occupant associated with the user-specific code with the library of authorized digital images stored in the one or more user databases; and in response to determining that the digital image of the occupant associated with the user-specific code does not match one of the authorized digital images stored in the one or more user databases, execute a procedure to register an authorized digital image of the occupant. [9] The authentication system of claim 1, wherein the one or more controllers execute instructions to: to calculate the trustworthiness score based on the following equation: Creditability Score =w1Ssuccess−w2Sstat−w3Sinfeasible−w4fuserSconsequence where w1, w2, w3, and w4 represent weighting factors, Ssuccess represents a reward for each voice command generated by the occupant, which results in the corresponding pre-compiled driving maneuver being successfully executed by the vehicle and is based on the execution status, S stat represents a statutory penalty for each voice command generated by the occupant that results in the corresponding pre-compiled driving maneuver not being executed by the vehicle because, based on traffic rules associated with a specific location, where the vehicle is located, the pre-compiled driving maneuver is not authorized, S infeasible represents a feasibility penalty for each voice command generated by the occupant that results in that the driving maneuver is not carried out by the vehicle because the pre-compiled driving maneuver is not feasible, f userrepresents a value indicating an error associated with the occupant generating the voice command in the event that the pre-compiled driving maneuver is successfully executed but the performance of the voice-controlled autonomous driving system deteriorates, and S consequence represents a consequence penalty for each voice command generated by the inmate, which indicates a performance degradation of the voice-controlled autonomous driving system. [10] The authentication system of claim 9, wherein the one or more controllers execute instructions to: the value f user which indicates the fault associated with the occupant based on: fuser=f(time,distancegoal,deviationparams) where time represents a time interval, distance goal represents a change in a target state and deviation paramsa degree of change between a variety of behavior planning parameters and one or more updated behavior planning parameters. [11] The authentication system of claim 10, wherein the change in the target state is determined by comparing a distance measured between an original target location indicating a planned target state of the vehicle originally determined by a behavior planner of the voice-controlled autonomous driving system and a modified target location indicating a modified target state of the vehicle reached after executing the pre-compiled driving maneuver. [12] The authentication system of claim 1, wherein the authorization status indicates when the pre-compiled driving maneuver is authorized based on traffic rules associated with a particular location where the vehicle is located. [13] The authentication system of claim 1, wherein the execution status indicates that the pre-compiled driving maneuver specified by the voice command was successfully completed by the vehicle. [14] The authentication system of claim 1, wherein setting the access level of the voice-activated autonomous driving system for the occupant includes denying access to the voice-activated autonomous driving system. [15] The authentication system of claim 1, wherein setting the access level of the voice-activated autonomous driving system for the occupant includes assigning a lower level of access to the voice-activated autonomous driving system, the lower access level only allowing the occupant to control the voice-activated autonomous driving system within restricted areas.

Images