Functional safety in autonomous driving

Abstract

To meet requirements for improvement of a vehicle having autonomous driving capability.SOLUTION: Autonomous driving of a vehicle in which computerized perception by the vehicle, including of its environment and of itself (e.g., egomotion), is used to autonomously drive the vehicle and, additionally, can also be used to provide feedback to enhance performance, safety, and / or other attributes of autonomous driving of the vehicle (e.g., when certain conditions affecting the vehicle are determined to exist by detecting patterns in or otherwise analyzing what is perceived by the vehicle), by adjusting autonomous driving of the vehicle, conveying messages regarding the vehicle, and / or performing other actions concerning the vehicle.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims priority based on U.S. Provisional Patent Application No. 62 / 903,845, filed on September 22, 2019, which is incorporated herein by reference.

[0002] This disclosure relates to vehicles (e.g., automobiles, trucks, buses, and other road vehicles) having autonomous driving (also known as self - driving) capabilities.

Background Art

[0003] Autonomous - driving (i.e., self - driving) vehicles that can travel without human control (e.g., by autonomously steering, accelerating, and / or decelerating) have become increasingly popular.

[0004] For example, automobiles, trucks, and other road vehicles can feature various levels of driving automation from partial driving automation using one or more advanced driver assistance systems (ADAS) to full driving automation (e.g., any one of levels 2 - 5 of the SAE J3016 driving automation levels).

[0005] To determine where and how to safely move these vehicles and accordingly control actuators (e.g., power train, steering system, etc.) to move the vehicles, computerized perception of the environment and the vehicle itself (e.g., ego - motion) by these vehicles based on various sensors (e.g., cameras, lidar (light detection and ranging) devices, radar devices, GPS or other position sensors, inertial measurement units (IMU), etc.) is used.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] [Non-Patent Document 1] Internet <https: / / waymo.com / tech / > [Non-Patent Document 2] Internet <https: / / waymo.com / safetyreport / > [Non-Patent Document 3] Pendleton et al., "Perception, Planning, Control, and Coordination for Autonomous Vehicles", MDPI, February 17, 2017 [Summary of the Invention] [Problems to be Solved by the Invention]

[0008] Although the computer-controlled perception by these vehicles has made great progress, in some cases it has not yet been fully utilized, and for this reason, the performance, safety, and / or other attributes of the autonomous driving of these vehicles may be sub-optimal.

[0009] For these and other reasons, it is necessary to focus on improving vehicles with autonomous driving capabilities. [Means for Solving the Problems]

[0010] According to various aspects, the present disclosure relates to autonomous driving of a vehicle and providing feedback to enhance the performance, safety, and / or other attributes of the autonomous driving of the vehicle, such as by using computer-controlled perception of the environment by the vehicle and the vehicle itself (e.g., its ego motion), and further (e.g., when it is determined that there are certain conditions affecting the vehicle by detecting or otherwise analyzing patterns of what is perceived by the vehicle) adjusting the autonomous driving of the vehicle, communicating a message regarding the vehicle, and / or performing other operations regarding the vehicle.

[0011] For example, according to one aspect, the present disclosure relates to a system for autonomous driving of a vehicle or various levels of driving assistance. The system includes an interface configured to receive data from sensors of the vehicle, including at least a camera and a lidar sensor. The system also includes a processing entity including at least one processor, the processing entity providing perception information regarding the perception of the environment of the vehicle and the state of the vehicle, including a 3D model of the environment of the vehicle and information regarding the position of the vehicle, generating a control signal for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle, and in addition to generating a control signal for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle, processing the perception information to determine whether there are certain predetermined conditions affecting the vehicle, and if so, configured to perform an operation regarding the vehicle based on the predetermined conditions.

[0012] According to another aspect, the present disclosure relates to a system for autonomous driving of a vehicle or various levels of driving assistance. The system includes an interface configured to receive data from sensors of the vehicle, including, among other things, a camera and a lidar sensor. The system also includes a processing entity including at least one processor, and the processing entity provides perception information regarding the environment and state of the vehicle, including a 3D model of the vehicle's environment and information regarding the position of the vehicle, based on the data from the sensors, generates a control signal for autonomously driving the vehicle based on the 3D model of the vehicle's environment and information regarding the position of the vehicle, and is configured to process the perception information to detect a pattern indicating a predetermined condition that affects the vehicle within the perception information.

[0013] According to another aspect, the present disclosure relates to a non-transitory computer-readable medium including instructions executable by a processing device for autonomous driving of a vehicle or various levels of driving assistance, and the instructions, when executed by the processing device, receive data from sensors of the vehicle, including, among other things, a camera and a lidar sensor, provide perception information regarding the environment and state of the vehicle, including a 3D model of the vehicle's environment and information regarding the position of the vehicle, based on the data from the sensors, generate a control signal for autonomously driving the vehicle based on the 3D model of the vehicle's environment and information regarding the position of the vehicle, and in addition to generating a control signal for autonomously driving the vehicle based on the 3D model of the vehicle's environment and information regarding the position of the vehicle, process the perception information to determine whether a predetermined condition that affects the vehicle exists, and if so, cause the processing device to perform an operation regarding the vehicle based on the predetermined condition.

[0014] According to another aspect, the present disclosure relates to a non-transitory computer-readable medium including instructions executable by a processing device for autonomous driving of a vehicle or various levels of driving assistance. When executed by the processing device, the instructions receive data from sensors of the vehicle including, among other things, a camera and a lidar sensor, provide perception information regarding perception of the vehicle's environment and the vehicle's state including a 3D model of the vehicle's environment and information regarding the vehicle's position based on the data from the sensors, generate a control signal for autonomously driving the vehicle based on the 3D model of the vehicle's environment and the information regarding the vehicle's position, and cause the processing device to process the perception information to detect a pattern indicating a predetermined condition that affects the vehicle within the perception information.

[0015] According to another aspect, the present disclosure relates to a method for autonomous driving of a vehicle or various levels of driving assistance. The method includes receiving data from sensors of the vehicle including, among other things, a camera and a lidar sensor, providing perception information regarding perception of the vehicle's environment and the vehicle's state including a 3D model of the vehicle's environment and information regarding the vehicle's position based on the data from the sensors, generating a control signal for autonomously driving the vehicle based on the 3D model of the vehicle's environment and the information regarding the vehicle's position, and in addition to generating a control signal for autonomously driving the vehicle based on the 3D model of the vehicle's environment and the information regarding the vehicle's position, processing the perception information to determine whether a predetermined condition that affects the vehicle exists, and if so, performing an operation regarding the vehicle based on the predetermined condition.

[0016] According to another aspect, the present disclosure relates to a method for autonomous driving of a vehicle or various levels of driving assistance. The method includes, among other things, receiving data from sensors of the vehicle including at least a camera and a lidar sensor, providing perception information regarding the environment of the vehicle and the state of the vehicle, including a 3D model of the environment of the vehicle and information regarding the position of the vehicle, based on the data from the sensors, generating a control signal for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle, and processing the perception information to detect a pattern indicating a predetermined condition that affects the vehicle within the perception information.

[0017] Upon reviewing the description of the embodiments together with the accompanying drawings, these and other aspects of the present disclosure will become apparent to those skilled in the art.

[0018] Hereinafter, a detailed description of the embodiments will be shown as an example with reference to the accompanying drawings.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0020] It should be clearly understood that the description and the drawings are only for exemplifying several embodiments and for assisting understanding. These are not intended to be limiting and must not be construed as limiting.

[0021] Figures 1 to 5 show an embodiment of a vehicle 10 capable of autonomous driving (i.e., automatic driving) in an environment 11 of the vehicle 10. In this embodiment, the vehicle 10 is a road vehicle, and its environment 11 includes a road 19. The vehicle 10 is designed to legally carry people and / or goods on the road 19, which is part of the public road infrastructure (e.g., public roads, arterial roads, etc.). In this example, the vehicle 10 is a motor vehicle (e.g., a passenger car).

[0022] The vehicle 10 is capable of autonomous driving in that it can travel without direct human control, for example, by automatically steering, accelerating, and / or decelerating (e.g., braking) to move towards a destination for at least a part of its use. The vehicle 10 is capable of autonomous driving in some embodiments, but can also be controlled or managed by a human driver depending on the situation. Therefore, the vehicle 10 can feature various levels of driving automation from partial driving automation using one or more advanced driver assistance systems (ADAS) to full driving automation (e.g., any one of levels 2 to 5 of the SAE J3016 driving automation levels).

[0023] As will be further described below, in this embodiment, the vehicle 10 uses computer - controlled perceptions such as the environment 11 by the vehicle 10 and the vehicle 10 itself (e.g., its ego - motion) to autonomously drive the vehicle 10, and further (e.g., when it is determined that there are some conditions that affect the vehicle 10 by detecting or otherwise analyzing the patterns of what is perceived by the vehicle 10), adjusts the autonomous driving of the vehicle 10, conveys a message regarding the vehicle 10, and / or performs other operations regarding the vehicle 10, etc., to provide feedback to enhance the performance, safety, and / or other attributes of the autonomous driving of the vehicle 10.

[0024] In this embodiment, the vehicle 10 includes a frame 12, a powertrain 14, a steering system 16, a suspension 18, wheels 20, a cabin 22, and a control system 15 configured to autonomously (i.e., without human control) operate the vehicle 10 for at least some of its uses.

[0025] The powertrain 14 is configured to generate power for the vehicle 10, including motive power for the wheels 20 to propel the vehicle 10 on a road 19. For this purpose, the powertrain 14 includes a power source (e.g., a prime mover) that includes one or more motors. For example, in some embodiments, the power source of the powertrain 14 can include an internal combustion engine, an electric motor (e.g., battery - driven), or a combination of different types of motors (e.g., an internal combustion engine and an electric motor). The powertrain 14 can transmit power from the power source to one or more of the wheels 20 in any suitable manner (e.g., through a transmission, a differential, a shaft engaged (i.e., directly connected) to the motor, and a given one of the wheels 20).

[0026] The steering system 16 is configured to steer the vehicle 10 on the road 19. In this embodiment, the steering system 16 is configured to bend the front wheels among the wheels 20 to change the orientation of the front wheels with respect to the frame 12 of the vehicle 10 in order to move the vehicle 10 in a desired direction.

[0027] The suspension 18 is connected between the frame 12 and the wheels 20 to enable relative movement between the frame 12 and the wheels 20 when the vehicle 10 moves on the road 19. For example, the suspension 18 can enhance the operation of the vehicle 10 on the road 19 by absorbing shocks and assisting in maintaining static friction between the wheels 20 and the road 19. The suspension 18 can include one or two or more springs, dampers, and / or other elastic devices.

[0028] The cabin 22 is configured to be occupied by one or more passengers of the vehicle 10. In this embodiment, the cabin 22 is configured to interact with one or more passengers of the vehicle, and includes an input unit including one or more input devices (for example, a series of buttons, levers, dials, etc., a touch screen, a microphone, etc.) that enable the passengers of the vehicle 10 to input commands and / or other information into the vehicle 10, and an output unit including one or more output devices (for example, a display, a speaker, etc.) that provide information to the passengers of the vehicle 10. The output unit of the user interface 70 can include an instrument panel (for example, a dashboard) that provides indicators related to the operation of the vehicle 10 (for example, a speedometer indicator, a tachometer indicator, etc.).

[0029] When the vehicle 10 is moving along a route on the road 19 towards a destination, the control system 15 is configured to operate the vehicle 10 autonomously (i.e., without human control), including steering, accelerating, and / or decelerating (e.g., braking) the autonomous vehicle 10. Specifically, the control system 15 controls the vehicle 10 to move on the road 19 towards its destination based on, among other things, computer-controlled perception of the environment 11 of the vehicle 10 and the vehicle 10 itself (e.g., ego motion), and includes a controller 80 and a detection device 82 that perform operations (such as steering, accelerating, and decelerating) to control the vehicle 10.

[0030] The vehicle 10 can be driven by its control system 15, but can also be controlled by a human driver, such as a passenger in the cabin 22, depending on the situation. For example, in some embodiments, the control system 15 can be selectively operable to operate the vehicle 10 autonomously (i.e., without human control) or under human control (i.e., by a human driver) in various environments (e.g., the vehicle 10 can operate in either an autonomous operation mode or an operation mode under human control). For example, in this embodiment, the user interface 70 in the cabin 22 can include an acceleration device (e.g., an accelerator pedal), a braking device (e.g., a brake pedal), and a steering device (e.g., a steering wheel) that can be operated by a human driver in the cabin 22 to control the vehicle 10 on the road 19.

[0031] The controller 80 is a processing device configured to process information received from the detection device 82 and optionally other sources to perform control operations of the vehicle 10, including steering, accelerating, and / or decelerating the vehicle 10 towards a destination on the road 19. In this embodiment, the controller 80 includes an interface 166, a processing entity 168, and a memory 170, which are implemented by suitable hardware and software.

[0032] Interface 166 enables the controller 80 to receive input signals and transmit output signals between the controller 80 and other components that are the connection destinations (i.e., direct or indirect connection destinations) of other components such as a communication interface 68 configured to communicate (i.e., vehicle-to-vehicle communication) with a detection device 82, a power train 14, a steering system 16, a suspension 18, and optionally a user interface 70, a communication network (e.g., the Internet and / or a cellular or other wireless network for other communications), and / or one or more other vehicles near the vehicle 10. The controller 80 can communicate with other components of the vehicle 10 via a vehicle bus 58 (e.g., a controller area network (CAN) bus or other suitable vehicle bus).

[0033] The processing entity 168 includes one or more processors that execute processing operations implementing the functionality of the controller 80. The processor of the processing entity 168 can be a general-purpose processor that executes program code stored in the memory 170. Alternatively, the processor of the processing entity 168 can also be a special-purpose processor that includes one or more pre-programmed hardware or firmware elements (e.g., an application-specific integrated circuit (ASIC), an electrically erasable programmable read-only memory (EEPROM), etc.), or other related elements.

[0034] Memory 170 includes one or more storage elements that store program code executed by processing entity 168 and / or data used during the operation of processing entity 168 (e.g., maps, vehicle parameters, etc.). The storage elements of memory 170 can be a semiconductor medium (including, for example, solid-state memory), a magnetic storage medium, an optical storage medium, and / or any other suitable type of memory. The storage elements of memory 170 can include, for example, read-only memory (ROM) elements and / or random access memory (RAM) elements.

[0035] In some embodiments, controller 80 can be associated with (e.g., include and / or interact with) one or more other control units of vehicle 10. For example, in some embodiments, controller 80 can include a powertrain control unit of powertrain 14, such as an engine control unit (ECU), a transmission control unit (TCU), etc., and / or interact with such powertrain control units.

[0036] Detection device 82 includes a sensor 90 configured to detect aspects of environment 11 that include objects 32 (e.g., people, animals, other vehicles, inanimate objects, traffic management devices such as traffic lights and traffic signs, other obstacles, lanes, free-drivable areas, and / or any other tangible static or dynamic object) within environment 11 of vehicle 10 and aspects of the state of vehicle 10 that include the position (e.g., location, orientation, and / or movement) of vehicle 10, and generate data indicative of these aspects. These data are provided to controller 80, and the controller can process this data to determine the actions that vehicle 10 should autonomously perform to continue moving towards its destination.

[0037] Sensor 90 can include any suitable detection device. For example, in some embodiments, sensor 90 - one or more passive sensors such as camera 92, a sound sensor, a light sensor, - One or more active sensors such as a lidar (light detection and ranging) sensor 94 (e.g., a solid-state lidar device without rotating mechanical parts such as a microelectromechanical systems (MEMS) lidar, flash lidar, optical phased array lidar, or frequency-modulated continuous wave (FMCW) lidar, or a mechanical lidar including a rotating assembly), a radar sensor 96, an ultrasonic sensor, - A position sensor 98 (e.g., based on GPS), - A vehicle speed sensor 97, - An inertial measurement unit (IMU) 95 including an accelerometer, gyroscope, etc., and / or - Any other detection device, can be included.

[0038] Vehicle 10 can be implemented in any suitable manner. For example, in some embodiments, vehicle 10 including a control system 15 can be implemented using the techniques described in https: / / waymo.com / tech / and https: / / waymo.com / safetyreport / , U.S. Patent No. 8,818,608, or U.S. Patent Application Publication No. 2014 / 0303827, or any other suitable autonomous driving technology (e.g., one or more advanced driver assistance systems (ADAS)), all of which are incorporated herein by reference.

[0039] Continuing to refer to FIG. 5, in this embodiment, controller 80 includes a plurality of modules including a perception module 50 and a driving module 54 that autonomously drive (e.g., accelerate, decelerate, steer, etc.) vehicle 10 towards a destination on road 19 and otherwise control it. In various embodiments, these modules can be implemented in any suitable manner (e.g., the methods described in Pendleton et al., "Perception, Planning, Control, and Coordination for Autonomous Vehicles," MDPI, February 17, 2017, or any known method, all of which are incorporated herein by reference).

[0040] Based on the data from the sensor 90, the perception module 50 is configured to provide in real time information 210 regarding the perception of the environment 11 of the vehicle 10 and the state of the vehicle 10. This information 210, referred to as "perception information", conveys knowledge of the environment 11 of the vehicle 10 and the state of the vehicle (e.g., position, ego motion, etc.), and is used by the driving module 54 to autonomously drive the vehicle 10.

[0041] Specifically, in this embodiment, the perception module 50 is configured to generate a 3D model of the environment 11 of the vehicle 10 based on the data from the sensor 90. This 3D model, referred to as the "3D environment model", includes information providing a representation of the environment 11 including the objects 32 within the environment 11 of the vehicle 10. The 3D environment model can include characteristics of the objects 32 such as the class (i.e., type) of these objects 32, shape, distance to the vehicle 10, speed, position relative to some reference points, etc. The perception module 50 can use any suitable known technique such as frame-based processing, segmentation, deep learning, or other machine learning algorithms using a deep neural network or other artificial neural network to detect and potentially classify various objects 32 within the scene of the environment 11 of the vehicle 10.

[0042] In some embodiments, as shown in FIG. 6, the perception module 50 can include a sensor data fusion module 55 configured to fuse, i.e., combine and integrate, the data from each of the sensors 90 including other sensors such as the camera 92, the lidar sensor 94, and optionally the radar sensor 96, etc., and perform data fusion for processing these data. Such data fusion can be implemented in any suitable manner (e.g., in the manner described in U.S. Patent Application Publication No. 2016 / 0291154, which is incorporated herein by reference, or any other known manner).

[0043] The perception module 50 is also configured to generate information regarding the position of the vehicle 10 within the environment 11 by performing vehicle 10 position identification based on data from sensors 90 such as the position sensor 98, the vehicle speed sensor 97, and the IMU 95, thereby determining the position and movement of the vehicle. This information, referred to as "position information", indicates one or more other parameters that depend on the position of the vehicle 10, such as the position (e.g., location and orientation) of the vehicle 10, and / or the movement of the vehicle (e.g., speed, acceleration, etc.) and / or other kinematic aspects of the vehicle 10 that can be defined as ego-motion.

[0044] Accordingly, in this embodiment, the perception information 210 provided by the perception module 50 can include the 3D environment model and position information of the vehicle 10, as well as other information derived from the sensors 90, including the data from the sensors 90 themselves.

[0045] For example, in some embodiments, the perception module 50 can be implemented by LeddarVision (trademark) commercially available from LeddarTech (registered trademark) (e.g., https: / / leddartech.com / leddarvision / ), or any other commercial technology.

[0046] The driving module 54 is configured to control the vehicle 10 by determining how to drive the vehicle 10 (e.g., accelerate, decelerate, and / or steer) based on the perception information 210 provided by the perception module 50, which includes the 3D environment model and position information of the vehicle 10 and, optionally, other information, and sending control signals to actuators 70, such as other components of the vehicle 10 that control the power train 14, the steering system 16, and / or the movement and / or other operating aspects of the vehicle 10.

[0047] For example, in this embodiment, the driving module 54 applies a driving policy, adheres to traffic rules, makes predictions regarding the trajectories of the vehicle 10 and other objects within the environment 11 (e.g., to avoid collisions), and / or executes other suitable operations, etc., to plan a safe route for the vehicle 10, and a control module 56 that generates a control signal transmitted to the actuator 70 to autonomously move the vehicle 10 along the route can be implemented.

[0048] In this embodiment, the controller 80 includes a condition detection module 48 configured to determine whether one or more predetermined conditions that affect the vehicle 10 exist based on the perception information 210 provided by the perception module 50, and generate information 240 regarding the existence of the predetermined conditions that affect the vehicle 10 when they exist. The driving module 54 uses this information, referred to as "detected condition information", to perform one or more operations regarding the vehicle 10, such as adjusting the autonomous driving and / or other operations of the vehicle 10, transmitting a message regarding the vehicle 10, and / or performing another action to enhance the performance, safety, and / or other attributes of the autonomous driving of the vehicle 10. In some cases, this can provide feedback to the driving module 54 that is not otherwise available and / or enable a more rapid adjustment of the autonomous driving of the vehicle 10.

[0049] One of the predetermined conditions that affect the vehicle 10 that the condition detection module 48 can detect and the detected condition information 240 can indicate is environmental, i.e., existing outside the vehicle 10 and originating from the environment 11, and can generally be unrelated to the objects of interest in the scene that the driving module 54 uses to determine the commands transmitted to the actuator 70. Examples of objects of interest include, among others, adjacent vehicles and pedestrians. For example, in some embodiments, among the predetermined conditions that affect the vehicle 10, the environmental conditions are - The shape of Road 19 (e.g., the curvature or straightness of the road, etc.), the state of Road 19 (e.g., the slipperiness of Road 19, the roughness of Road 19, and / or other attributes of the surface of Road 19 that may be related to the wetness or dryness of Road 19, the surface material of Road 19 (e.g., paved road or unpaved road, the type of paving such as asphalt, concrete, gravel, etc.), and / or the damage of Road 19 (e.g., holes), road construction of Road 19, etc.), and / or any other characteristics of Road 19, - An off-road area of the environment 11 of the vehicle 10 where the vehicle 10 may have entered (e.g., deliberately or accidentally), - The meteorology within the environment 11 of the vehicle 10, such as precipitation (e.g., rain, sleet, snow, etc.), wind (e.g., wind speed or wind strength, wind direction, etc.), temperature, fog, and / or any other meteorological characteristics within the environment, - The lighting within the environment 11 of the vehicle 10, such as the type of light (e.g., sunlight, moonlight, artificial light, indoor such as outdoor, parking lot or tunnel lighting, etc.), the intensity of light, and / or any other lighting characteristics within that environment, - The density of the object 32 (e.g., high density indicating a city or other area with relatively high traffic volume, low density indicating a suburb, rural area or other area with relatively low traffic volume, etc.), the distance from the object 32 to the vehicle 10, the time for the object 32 and the vehicle 10 to reach each other (e.g., collide), and / or any other characteristics of the object 32 within the environment of the vehicle 10, and / or, - Any other aspect of the environment 11 of the vehicle 10, can be related to.

[0050] Alternatively, the detection condition information may indicate a condition that is not directly related to the environment 11 in which the vehicle 10 operates in relation to the vehicle 10. These conditions that can be detected by the condition detection module 48 and indicated by the detection condition information 240 are vehicle - related, i.e., inherent in the vehicle 10, and are derived from one or more components of the vehicle 10 such as the power train 14, steering system 16, suspension 18, wheels 20 and / or any other component of the vehicle 10. For example, in some embodiments, among the predetermined conditions affecting the vehicle 10, the vehicle - related conditions are - Malfunctions of components of the vehicle 10 (e.g., excessive vibration of components of the vehicle 10 (such as other motors of the engine or power train 14), wear, damage or other deterioration of components of the vehicle 10 (such as air leakage or wear of the tires of the wheels 20 of the vehicle 10), steering abnormalities within the steering system 16 (such as excessive degrees of freedom of movement), headlight malfunctions, abnormal sounds caused by the power train 14, steering system 16 or suspension 18, etc.), and / or any other malfunctions of one or more components of the vehicle 10, - Output of the power train 14, sensitivity of the steering system 16 (e.g., movement of the steering wheel), rigidity and / or damping of the suspension 18, and / or any other characteristics of the settings of one or more components of the vehicle 10, and / or, - Any other aspect of one or more components of the vehicle 10, can be related to.

[0051] Accordingly, the detection condition information 240 indicating one or more predetermined conditions affecting the vehicle 10 generated by the condition detection module 48 can be related to maintenance, indicate malfunctions, or the need for maintenance or adjustment.

[0052] For example, the perception information 210 provided by the perception module 50 can conceptually be regarded as implementing two detection streams, namely, a main or direct stream that performs detection of objects of interest having an output used by the driving module 54 to determine short-term actuator commands for providing movement control of the vehicle 10 within the 3D environment model, and an auxiliary stream that generally searches for predetermined conditions in the environment 11 that are unrelated to the objects of interest for determining short-term movement control or at least unrelated to the characteristics of the objects of interest. In some embodiments, such both detection streams are carried by the information conveyed by at least the lidar sensor 94 and the camera 92. In other words, the information collected by the lidar sensor 94 and the camera 92 is used to search for objects of interest for short-term movement control and predetermined conditions that affect longer-term driving policies and / or vehicle maintenance.

[0053] Therefore, in addition to generating control signals for movement control in the 3D environment model, the perception information 210 provided by the perception module 50 can be further processed to detect one or more predetermined conditions that affect the vehicle 10.

[0054] In this embodiment, the condition detection module 48 is configured to detect one or more patterns indicating the presence of one or more predetermined conditions within the perception information 210 output by the perception module 50 in order to determine whether one or more predetermined conditions that affect the vehicle 10 exist. Each of these patterns, called "perception fingerprints", indicates a predetermined condition that affects the vehicle 10 such that the detection condition information 240 generated by the condition detection module 48 conveys this perception fingerprint or is otherwise based on this perception fingerprint.

[0055] In various examples, a given one of these perceptual fingerprints can reflect a pattern that exists in the 3D environmental model (e.g., a pattern indicating a predetermined condition related to road 19, weather, lighting, and / or other aspects of the environment 11 of vehicle 10), a pattern that exists in the position information of vehicle 10 (e.g., such as tire wear or deflation of wheels 20, steering anomalies in steering system 16, abnormal vibrations of the motor in power train 14, and / or other aspects of one or more components of vehicle 10, indicating a predetermined condition related to a malfunction of vehicle 10), a pattern that exists in both the 3D environmental model and the position information of vehicle 10, or a pattern that does not exist in either the 3D environmental model or the position information of vehicle 10 (e.g., in the data from sensor 90). Also, a given one of these perceptual fingerprints can be a data pattern from a combination of different sensors of sensor 90 that cannot be detected by considering any of the different sensors of sensor 90 individually.

[0056] Specifically, in this embodiment, the condition detection module 48 includes a perceptual fingerprint identification module 60 configured to detect one or more perceptual fingerprints from the perceptual information 210 provided by the perceptual module 50 and direct the detection condition information 240 generated by the condition detection module 48 to convey these one or more perceptual fingerprints or to be otherwise based on these one or more perceptual fingerprints.

[0057] The perception fingerprint identification module 60 can implement any suitable pattern recognition algorithm for detecting one or more perception fingerprints from the perception information 210 provided by the perception module 50. For example, in this embodiment, the perception fingerprint identification module 60 implements artificial intelligence (also called AI - machine intelligence or machine learning), such as an artificial neural network, a support vector machine, or any other AI unit, in the form of software, hardware, and / or a combination thereof configured to recognize perception fingerprints from the perception information 210 provided by the perception module 50.

[0058] Specifically, in this embodiment, as shown in FIG. 7, the perception fingerprint identification module 60 includes an artificial neural network 64 configured to detect one or more perception fingerprints from the perception information 210 provided by the perception module 50. The artificial neural network 64 can be a deep neural network (e.g., convolutional, recurrent, etc.) and / or can be implemented using any known type of neural network technology.

[0059] The artificial neural network 64 is configured to learn how to detect perception fingerprints from the perception information 210 provided by the perception module 50. The learning by the artificial neural network 64 can be achieved using any known supervised, semi - supervised, or unsupervised technique.

[0060] In some embodiments, the artificial neural network 64 learns by processing, during a learning mode, "training" data that conveys information that it is seeking within the 3D environment model (similar to what is thought to be part of the perception information 210) and / or the position information of the vehicle 10, specifically data that includes one or more perceptual fingerprints indicative of one or more predetermined conditions to be detected and thus affecting the vehicle 10. For example, in a situation characterized by a predetermined condition of interest, by driving a training vehicle having sensors 90, a perception module 50, and an artificial neural network 64 similar to those of the vehicle 10, the perception module of the training vehicle generates training data including perceptual fingerprints (i.e., patterns) indicative of these predetermined conditions, and the artificial neural network of the training vehicle can learn to identify these perceptual fingerprints by processing this training data.

[0061] For example, in some embodiments, if the predetermined conditions to be detected include rough roads, paved roads, slippery roads, winding roads, strong winds, heavy snow, sleet, artificial light, worn tires, flat tires, abnormal vibrations of a motor (e.g., an engine), headlight malfunctions, steering abnormalities, abnormal sounds, or combinations thereof (e.g., rough roads during strong winds, slippery roads during strong winds, slippery winding roads, slippery winding roads during strong winds, slippery roads in artificial light, worn tires on slippery roads, flat tires on rough roads, artificial light during headlight malfunctions, etc.), or any other predetermined conditions to be detected, the learning mode involves driving a training vehicle in one or more situations (e.g., one or more rough roads, one or more paved roads, one or more slippery roads, one or more winding roads, one or more weather events with strong winds, one or more weather events with heavy snow, one or more weather events with sleet, artificial light in one or more areas, one or more worn tires, one or more flat tires, one or more steering abnormalities, one or more abnormal motor vibrations, one or more situations with abnormal sounds, etc.) characterized by each given condition among these predetermined conditions. By doing so, the perception module of the training vehicle can generate training perception information including a perception fingerprint indicating this given predetermined condition, and the artificial neural network of the training vehicle can learn to identify this perception fingerprint.

[0062] Accordingly, in some embodiments, a library or other database can maintain perceptual fingerprints detectable by the perceptual fingerprint identification module 60, and predetermined conditions that these indicate and that affect the vehicle 10. In some cases, the perceptual fingerprint identification module 60 can attempt to identify a perceptual fingerprint that it has not seen before, in which case the perceptual fingerprint identification module 60 can determine that this perceptual fingerprint is different or abnormal with respect to perceptual fingerprints previously encountered. For example, in an implementation of a neural network, the perceptual fingerprint can be information of a class that is trained to be detected by the neural network examining sensor data. In the embodiments of FIGS. 6 and 7, the perceptual fingerprint identification module 60 can continuously output a perceptual fingerprint that differentiates the current environment 11 in which the vehicle 10 operates from the number of environments that the module 60 can identify within the perceptual information 210.

[0063] This perceptual fingerprint can be used as an additional input to the driving module 54 for adjusting the signals sent to the actuators 70 of the vehicle 10. Accordingly, the driving module 54 uses two inputs, both of which are derived from the same perceptual information 210, namely, object-of-interest information that specifically determines short-term movement control and environmental input that adjusts the actual rules for determining short-term movement control. For example, if the environmental input indicates that information generated by the sensors is classified as a fingerprint associated with a slippery road, this input affects the short-term movement control determined by the driving module 54, and thus, for example, the steering input, throttle input, and brake input are adjusted differently to account for the expected slippery road surface.

[0064] The artificial neural network 64 of the condition detection module 48 can be trained to identify a perceptual fingerprint indicating a predetermined condition affecting the vehicle 10 from the perceptual information 210 provided by the perception module 50, even if the sensor 90 is not designed to directly measure the predetermined condition. For example, in some embodiments, the classification of patterns by the artificial neural network 64 can separate or distinguish vibrations and natural frequencies with fingerprints from rough road surfaces and other phenomena outside the vehicle 10 that may have an impact. When this classification indicates the vibration source, the vibration of the motor (e.g., engine) of the powertrain 14 can be identified as an abnormal pattern of the position information of the vehicle 10 (e.g., ego motion) or as a signal from the IMU 95 within the perceptual information 210.

[0065] Accordingly, referring further to FIG. 8, in this embodiment, the controller 80 can implement the following process.

[0066] The perception module 50 generates position information 210 including a 3D environment model and the position information of the vehicle 10 based on the data from the sensor 90, and the driving module 54 uses the perceptual information 210 to determine how to drive the vehicle 10 (e.g., accelerate, decelerate, and steer) and send a signal to the actuator 70 (e.g., the powertrain 14, the steering system 16, etc.), and the vehicle 10 is autonomously driven according to this signal.

[0067] On the other hand, the condition detection module 48 processes the perceptual information 210 provided by the perception module 50 to determine whether one or more perceptual fingerprints indicating one or more predetermined conditions affecting the vehicle 10 are included in this information. If the condition detection module 48 detects one or more perceptual fingerprints indicating one or more predetermined conditions affecting the vehicle 10, the detection condition information 240 generated by the condition detection module 48 conveys these one or more perceptual fingerprints or is otherwise based on these one or more perceptual fingerprints.

[0068] The driving module 54 transmits one or more perceptual fingerprint(s) indicating a predetermined condition affecting the vehicle 10, or executes one or more operations regarding the vehicle 10 using detection condition information 240 based otherwise on the one or more perceptual fingerprint(s).

[0069] For example, in some embodiments, the driving module 54 can adjust the autonomous driving and / or other operations of the vehicle 10 based on the perceptual fingerprint(s) detected by the condition detection module 48. For example, in some cases, if the detected (single or multiple) perceptual fingerprint(s) indicate that the road 19 is rough, slippery and / or winding, strong wind is blowing, one or more of the wheels 20 are worn or deflated, the motor (e.g., engine) of the power train 14 is vibrating abnormally, there is a steering abnormality in the steering system 16, etc., the driving module 54 can determine a short-term actuator command, autonomously drive the vehicle 10 at a lower speed (e.g., reduce the speed of the vehicle 10 when going straight and / or turning), and adjust the logic to reduce the stiffness of the suspension 18 or increase the damping. Conversely, if the detected (single or multiple) perceptual fingerprint(s) indicate that the road 19 is smooth, dry and / or straight, strong wind is not blowing, etc., the driving module 54 can autonomously drive the vehicle 10 at a higher speed (e.g., increase the speed of the vehicle 10 when going straight and / or turning), and adjust the short-term control logic to increase the stiffness of the suspension or reduce the damping. The driving module 54 can send a signal for adjusting the autonomous driving of the vehicle 10 in this way to the actuator 70 such as the power train 14, the steering system 16 and / or the suspension 18.

[0070] As another example, in some embodiments, the driving module 54 can convey a message regarding the vehicle 10 to an individual (e.g., a user of the vehicle 10) or a computer device, etc., based on the (single or plural) perceived fingerprint detected by the condition detection module 48. This message can indicate a malfunction or other problem of one or more components of the vehicle 10. For example, in some cases, the detected (single or plural) perceived fingerprint indicates that one or more tires of the wheels 20 are worn or deflated, one or more headlights are not operating, the motor (e.g., the engine) of the power train 14 is vibrating abnormally, there is a steering abnormality in the steering system 16, etc. In such cases, the driving module 54 can convey a notice to the effect that maintenance, repair, or other services should be performed on the vehicle 10. In some cases, a message regarding the vehicle 10 can be conveyed to the user interface 70 of the vehicle 10. In other embodiments, a message regarding the vehicle 10 can be conveyed to a different (i.e., not part of the vehicle 10 and in some cases external to the vehicle 10) communication device (e.g., a smartphone or a computer) via the communication interface 68 of the vehicle 10.

[0071] In other embodiments, the condition detection module 48 can be configured to determine in various other ways whether one or more predetermined conditions that affect the vehicle 10 exist.

[0072] For example, in some embodiments, as shown in FIG. 9, in order to determine whether one or more predetermined conditions that affect vehicle 10 exist, condition detection module 48 is configured to compare perception information 210 provided by perception module 50 with other information 350 different from the 3D environmental model and the position information of vehicle 10 (e.g., ego motion) that is available to controller 80. This information 350, referred to as "perception-independent reference information", can be obtained from one or more sources independent of sensor 90 used to generate the 3D environmental model and the position information of vehicle 10. When condition detection module 48 determines that perception information 210 does not match perception-independent reference information 350, it determines that a predetermined condition that affects vehicle 10 exists, generates valid detection condition information 240 indicating this predetermined condition, and driving module 54 can use this information to perform one or more operations related to vehicle 10, such as adjusting the autonomous driving and / or other operations of vehicle 10, or transmitting a message related to vehicle 10, as described above.

[0073] In some embodiments, perception-independent reference information 350 is derived from data 67 representing a prediction regarding vehicle 10 (e.g., regarding the environment 11 of the vehicle and / or one or more operating aspects of vehicle 10), stored in memory 70 of controller 80, received via communication interface 68, or otherwise utilized by controller 80.

[0074] As an example, in some embodiments, perception-independent reference information 350 is derived from a map 65 (e.g., a high-resolution map) representing the location of vehicle 10, stored in memory 70 of controller 80, received via communication interface 68, or otherwise utilized by controller 80. Map 65 can provide perception-independent reference information 350 such as the type of road surface of road 19 that vehicle 10 is expected to encounter at a particular location (e.g., paved road, unpaved road, open field, sandy beach, etc.). Driving module 54 can control vehicle 10 based on the information provided by this map 65.

[0075] The condition detection module 48 compares the perception information 210 provided by the perception module 50 with the perception-independent reference information 350 provided by the map 65 to determine whether the surface of the road 19 perceived by the perception module 50 (based on the 3D environment model and / or the ego motion of the vehicle 10) is actually as predicted by the map 65. If not, it can generate detection condition information 240 to indicate what the actual surface of the road 19 is like. As a result, the driving module 54 can determine whether and how to adjust the autonomous driving of the vehicle 10 based on the detection condition information 240. For example, if the driving module 54 determines based on the detection condition information 240 that the actuator setting of the estimated actuator 70 is inappropriate (e.g., sub-optimal or insufficient) for driving smoothness and safety, it can send a signal to the actuator 70 to adjust this setting accordingly.

[0076] As another example, in some embodiments, the perception-independent reference information 350 is derived from an illumination model 34 representing the expected illumination (e.g., light and shadow) around the vehicle 10, stored in the memory 70 of the controller 80, received via the communication interface 68, or otherwise utilized by the controller 80.

[0077] The condition detection module 48 compares the actual illumination conveyed by the perception information 210 provided by the perception module 50 (e.g., based on an image from the camera 92) with the predicted illumination specified by the illumination model 34 of the perception-independent reference information 350, to determine whether the actual illumination perceived by the perception module 50 is actually as predicted by the illumination model 34. If not, it generates detection condition information 240 to indicate the actual illumination. As a result, the driving module 54 can determine whether and how to adjust the autonomous driving of the vehicle 10 based on the detection condition information 240. For example, if the driving module 54 determines based on the detection condition information 240 that the setting of the actuator 70 is inappropriate (e.g., sub-optimal or insufficient) for the smoothness and safety of driving, it can send a signal to the actuator 70 to adjust this setting accordingly. Alternatively or in addition, the driving module 54 can also send a message indicating that maintenance or other services should be performed on the vehicle 10.

[0078] In some embodiments, as shown in FIG. 10, the perception-independent reference information 350 can be derived from the power train 14, the steering system 16, the suspension 18, and / or any other component that controls the movement of the vehicle 10. For example, in some embodiments, the perception-independent reference information 350 can indicate the steering movement of the steered wheels among the wheels 20 performed by the steering system 16, as reported on the vehicle bus 58 (e.g., CAN bus), while the perceived (e.g., past) steering movement of the steered wheels among the wheels 20 can be estimated using the ego motion of the vehicle 10 included in the perception information 210 provided by the perception module 50.

[0079] The condition detection module 48 determines whether the perceived movement of the steering wheel corresponds to the reported movement of the steering wheel on the vehicle bus 58 by comparing the perceived movement of the steering wheel with the reported movement of the steering wheel, and if not, generates detection condition information 240 to indicate the actual movement of the steering wheel. As a result, the driving module 54 can determine whether and how to adjust the autonomous driving of the vehicle 10 based on the detection condition information 240. For example, if the driving module 54 determines based on the detection condition information 240 that the estimated actuator settings of the respective actuators 70 in the steering system 16 are inappropriate (e.g., sub-optimal or insufficient) for maneuverability, it can send a signal to these actuators 70 to adjust this setting accordingly. Alternatively or in addition, the driving module 54 can also send a message indicating that maintenance or other services should be performed on the vehicle 10.

[0080] As another example, in some embodiments, the condition detection module 48 can be configured to monitor the temporal change (change over time) of the perception information 210 provided by the perception module 50 to determine whether one or more predetermined conditions that affect the vehicle 10 exist. For example, the condition detection module 48 monitors the temporal change of parameters that depend on the 3D environmental model, and when it observes that one or more parameters, which are regarded as indicating a predetermined condition that affects the vehicle 10 among these parameters of the 3D environmental model, have changed over time in a predetermined manner, it generates detection condition information 240 to indicate this predetermined condition, and the driving module 54 can use this information to perform one or more operations related to the vehicle 10, such as adjusting the autonomous driving and / or other operations of the vehicle 10, or transmitting a message regarding the vehicle 10, as described above.

[0081] For example, in some embodiments, the condition detection module 48 can monitor the time-dependent statistical behavior of the 3D environment model. For example, it can monitor the distribution of the "distance to an obstacle" or "time to collision" of an object 32 within the environment 11 of the vehicle 10. The desired behavior in a given driving scenario is considered to be that the change to this distribution is slow and smooth (e.g., less than a threshold ratio). The control of the vehicle 10 by the driving module 54 is determined by the driving policy, and tracking the statistics of the environment model distribution can help to evaluate different policies and adjust these policies relative to each other.

[0082] In another variant, the perception fingerprint can be used only for vehicle maintenance purposes without affecting motion control. In such a case, the perception fingerprint identification module 60 can receive inputs from drivetrain sensors configured to detect specific malfunctions or drivetrain conditions in addition to camera and lidar data. In this case, the condition detection module 48 provides a higher level of obstacle detection intelligence and triggers a maintenance message when the actual impact of the obstacle condition reported by the drivetrain sensors is observed within the 3D environment model.

[0083] In the embodiments discussed above, the vehicle 10 moves on the ground, but in other embodiments, the vehicle 10 can move other than on the ground. For example, in other embodiments, the vehicle 10 can fly (e.g., a delivery drone or other unmanned aerial transportation means, a flying car or other personal aircraft, etc.) or move on water (e.g., a water taxi or other ship), and thus "driving" generally means the operation, control, and orientation of the path of the vehicle 10.

[0084] Certain additional elements that may be required for the operation of some embodiments are believed to be within the scope of those skilled in the art, and thus these elements are not described or illustrated. Further, some embodiments do not include, are lacking in, or are capable of functioning without any elements not specifically disclosed herein.

[0085] In some implementation examples, any feature of any of the embodiments described herein can be combined with any feature of any of the other embodiments described herein.

[0086] If there are any contradictions, inconsistencies or other differences between the terms used herein and the terms used in any document incorporated herein by reference, the meaning of the terms used herein shall prevail.

[0087] Although various embodiments and examples have been presented, this presentation is for illustrative purposes and not limiting. Various modifications and enhancements will be apparent to those skilled in the art.

Description of Reference Numerals

[0088] 10 Vehicle 11 Environment 12 Frame 14 Power Train 15 Control System 16 Steering System 18 Suspension 19 Road 20 Wheel 22 Cabin 70 User Interface

Claims

1. 1. A system for autonomous driving of a vehicle, comprising: an interface configured to receive data from sensors of the vehicle, including cameras and lidar sensors; a processing entity including at least one processor; wherein the processing entity comprises: providing sensory information regarding the vehicle's perception of the environment and the state of the vehicle based on the data from the sensors, the sensor information including a 3D model of the vehicle's environment and information regarding the position of the vehicle; generating control signals for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle; and generating the control signals for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle, and processing the sensory information to determine whether a predetermined condition affecting the vehicle exists, and if so, performing an action with respect to the vehicle based on the predetermined condition. It is configured as follows: A system characterized by:

2. the processing entity is configured to detect a pattern indicative of the predetermined condition in the sensory information to determine that the predetermined condition affecting the vehicle exists. The system of claim 1 .

3. the processing entity includes an artificial neural network trained to detect the pattern indicative of the predetermined condition in the sensory information; The system of claim 2.

4. the pattern indicative of the predetermined condition in the sensory information is present in the 3D model of the environment of the vehicle.

4. A system according to claim 2 or 3.

5. the pattern indicative of the predetermined condition in the sensory information is present in the information regarding the position of the vehicle.

4. A system according to claim 2 or 3.

6. the pattern indicative of the predetermined condition in the sensory information is present both in the 3D model of the environment of the vehicle and in the information regarding the position of the vehicle.

4. A system according to claim 2 or 3.

7. the pattern indicative of the predetermined condition in the sensory information is not present in either the 3D model of the environment of the vehicle or in the information regarding the position of the vehicle; 4. A system according to claim 2 or 3.

8. the pattern indicative of the predetermined condition in the sensory information arises from a combination of different ones of the sensors and is not detectable individually from any of the different ones of the sensors.

4. A system according to claim 2 or 3.

9. the processing entity is configured to compare the sensory information with reference information to determine the presence of the predetermined condition affecting the vehicle. The system of claim 1 .

10. The reference information is derived from a map representative of the location of the vehicle. The system of claim 9.

11. the vehicle travels on a road, and the reference information indicates a condition of the road according to the map; The system of claim 10.

12. the reference information is derived from a component that controls the movement of the vehicle; The system of claim 9.

13. the reference information is derived from a vehicle bus connected to the components that control the movement of the vehicle; The system of claim 12.

14. the component that controls the movement of the vehicle is a powertrain of the vehicle; 14. A system according to claim 12 or 13.

15. the component that controls the movement of the vehicle is a steering system of the vehicle; 14. A system according to claim 12 or 13.

16. the processing entity is configured to monitor temporal changes in the sensory information to determine the presence of the predetermined condition affecting the vehicle. The system of claim 1 .

17. the processing entity is configured to perform data fusion on the data from each of the sensors, including the camera and the LIDAR sensor, to provide the perception information.

17. A system according to any one of claims 1 to 16.

18. the predetermined condition affecting the vehicle is external to the vehicle and originates from the environment of the vehicle; 18. A system according to any one of claims 1 to 17.

19. the predetermined condition affecting the vehicle is related to a road along which the vehicle travels; 20. The system of claim 18.

20. the predetermined condition affecting the vehicle is related to a condition of the road; 20. The system of claim 19.

21. the predetermined condition affecting the vehicle is related to slipperiness of the road; 21. The system of claim 20.

22. the predetermined condition affecting the vehicle is related to the roughness of the road; 22. A system according to claim 20 or 21.

23. the predetermined condition affecting the vehicle is related to a surface material of the road; 23. A system according to any one of claims 20 to 22.

24. the predetermined condition affecting the vehicle is related to the geometry of the road; 24. A system according to any one of claims 19 to 23.

25. the predetermined condition affecting the vehicle is related to the curvature of the road; 25. The system of claim 24.

26. the predetermined condition affecting the vehicle is related to weather in the environment of the vehicle; 20. The system of claim 18.

27. the predetermined condition affecting the vehicle is related to an amount of precipitation in the environment of the vehicle; 27. The system of claim 26.

28. the predetermined condition affecting the vehicle is related to wind in the environment of the vehicle; 28. A system according to claim 26 or 27.

29. the predetermined condition affecting the vehicle relates to lighting in the environment of the vehicle; 20. The system of claim 18.

30. the predetermined condition affecting the vehicle is related to a density of objects in the environment of the vehicle; 20. The system of claim 18.

31. the predetermined condition affecting the vehicle is inherent to the vehicle and originates from a component of the vehicle; 18. A system according to any one of claims 1 to 17.

32. the predetermined condition affecting the vehicle is related to functionality of the components of the vehicle; 32. The system of claim 31.

33. the predetermined condition affecting the vehicle is related to a malfunction of the component of the vehicle; 33. The system of claim 32.

34. the predetermined condition affecting the vehicle is related to deterioration of the component of the vehicle; 34. The system of claim 33.

35. the components of the vehicle are tires on wheels of the vehicle, and the predetermined conditions affecting the vehicle relate to wear on the tires; 35. The system of claim 34.

36. the component of the vehicle is a tire on a wheel of the vehicle, and the predetermined condition affecting the vehicle is related to deflation of the tire.

35. The system of claim 34.

37. the predetermined condition affecting the vehicle is related to vibration of the component of the vehicle; 32. The system of claim 31.

38. the component of the vehicle is a steering system of the vehicle, and the predetermined condition affecting the vehicle is related to a steering anomaly of the steering system.

32. The system of claim 31.

39. the component of the vehicle is a headlight of the vehicle, and the predetermined condition affecting the vehicle is related to a malfunction of the headlight of the vehicle.

32. The system of claim 31.

40. the predetermined condition affecting the vehicle is related to a configuration of the component of the vehicle; 32. The system of claim 31.

41. and wherein the action with respect to the vehicle includes adjusting autonomous driving of the vehicle based on the predetermined condition.

41. A system according to any one of claims 1 to 40.

42. The adjustment of the autonomous driving of the vehicle includes a change in a speed of the vehicle.

42. The system of claim 41.

43. and wherein the action with respect to the vehicle includes generating a signal directed to a component of the vehicle based on the predetermined condition.

41. A system according to any one of claims 1 to 40.

44. the component of the vehicle is a powertrain of the vehicle; 44. The system of claim 43.

45. the component of the vehicle is a steering system of the vehicle; 44. The system of claim 43.

46. the component of the vehicle is a suspension of the vehicle; 44. The system of claim 43.

47. the action with respect to the vehicle includes communicating a message with respect to the vehicle.

41. A system according to any one of claims 1 to 40.

48. the message regarding the vehicle is communicated to a user interface of the vehicle.

48. The system of claim 47.

49. the message regarding the vehicle is communicated to a communication device separate from the vehicle; 48. The system of claim 47.

50. the message regarding the vehicle is indicative of a malfunction of a component of the vehicle.

50. A system according to any one of claims 47 to 49.

51. the predetermined condition is one of a plurality of predetermined conditions affecting the vehicle, and the processing entity is configured to process the sensory information to determine whether any of the predetermined conditions affecting the vehicle are present, and if so, to perform an action with respect to the vehicle based on each of the predetermined conditions determined to be present.

51. A system according to any one of claims 1 to 50.

52. 1. A system for autonomous driving of a vehicle, comprising: an interface configured to receive data from sensors of the vehicle, including cameras and lidar sensors; a processing entity including at least one processor; wherein the processing entity comprises: providing sensory information regarding the vehicle's perception of the environment and the state of the vehicle based on the data from the sensors, the sensor information including a 3D model of the vehicle's environment and information regarding the position of the vehicle; generating control signals for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle; processing the sensory information to detect patterns in the sensory information indicative of predetermined conditions affecting the vehicle; It is configured as follows: A system characterized by:

53. 1. A non-transitory computer-readable medium comprising instructions executable by a processing device for autonomous driving of a vehicle, the instructions, when executed by the processing device, performing: receiving data from sensors of the vehicle, including cameras and lidar sensors; providing sensory information regarding the vehicle's perception of the environment and the state of the vehicle based on the data from the sensors, the sensor information including a 3D model of the vehicle's environment and information regarding the position of the vehicle; generating control signals for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle; and generating the control signals for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle, and processing the sensory information to determine whether a predetermined condition affecting the vehicle exists, and if so, performing an action with respect to the vehicle based on the predetermined condition. A non-transitory computer-readable medium for causing the processor to:

54. 1. A non-transitory computer-readable medium comprising instructions executable by a processing device for autonomous driving of a vehicle, the instructions, when executed by the processing device, performing: receiving data from sensors of the vehicle, including cameras and lidar sensors; providing sensory information regarding the vehicle's perception of the environment and the state of the vehicle, based on the data from the sensors, the sensor information including a 3D model of the vehicle's environment and information regarding the position of the vehicle; generating control signals for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle; processing the sensory information to detect patterns in the sensory information indicative of predetermined conditions affecting the vehicle; A non-transitory computer-readable medium for causing the processor to:

55. 1. A method for autonomous driving of a vehicle, comprising: receiving data from sensors of the vehicle, including cameras and lidar sensors; providing sensory information regarding the vehicle's perception of the environment and the state of the vehicle based on the data from the sensors, the sensor information including a 3D model of the vehicle's environment and information regarding the vehicle's position; generating control signals for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle; In addition to generating the control signals for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle, processing the sensory information to determine whether a predetermined condition affecting the vehicle exists, and if so, performing an action with respect to the vehicle based on the predetermined condition; The method according to claim 1, further comprising:

56. 1. A method for autonomous driving of a vehicle, comprising: receiving data from sensors of the vehicle, including cameras and lidar sensors; providing sensory information regarding the vehicle's perception of the environment and the state of the vehicle based on the data from the sensors, the sensor information including a 3D model of the vehicle's environment and information regarding the vehicle's position; generating control signals for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle; processing the sensory information to detect patterns in the sensory information indicative of predetermined conditions affecting the vehicle; The method according to claim 1, further comprising:

Images