Functional Safety in Autonomous Driving

Abstract

Autonomous operation of a vehicle that uses computer-controlled perception by the vehicle of the environment and the vehicle itself (e.g., its egomotion) to drive the vehicle autonomously and may also provide feedback to enhance the performance, safety, and / or other attributes of the autonomous operation of the vehicle, such as by adjusting the autonomous operation of the vehicle, communicating messages regarding the vehicle, and / or performing other actions regarding the vehicle (e.g., when it is determined that some condition affecting the vehicle exists by detecting or otherwise analyzing patterns of what is perceived by the vehicle).

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 903,845, filed Sep. 22, 2019, which is incorporated herein by reference.

[0002] This disclosure relates to vehicles (e.g., cars, trucks, buses, and other road vehicles) that have autonomous (also known as self-driving) capabilities. [Background technology]

[0003] Vehicles capable of autonomous driving (i.e., self-driving), that is, capable of traveling without human control (e.g., by steering, accelerating, and / or decelerating autonomously), are becoming more common.

[0004] For example, cars, trucks, and other road vehicles may feature various levels of driving automation (e.g., any one of Levels 2-5 of the SAE J3016 driving automation levels), ranging from partial driving automation using one or more advanced driver assistance systems (ADAS) to full driving automation.

[0005] A computerized perception of the environment and the vehicle itself (e.g., ego-motion) by these vehicles based on various sensors (e.g., cameras, LIDAR (Light Detection and Ranging) devices, RADAR devices, GPS or other position sensors, Inertial Measurement Units (IMUs), etc.) is used to determine how and where to safely move these vehicles and to drive them autonomously by controlling actuators (e.g., powertrain, steering systems, etc.) to move the vehicles accordingly. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent No. 8,818,608 [Patent Document 2] US Patent Application Publication No. 2014 / 0303827 [Patent Document 3] US Patent Application Publication No. 2016 / 0291154 [Non-patent literature]

[0007] [Non-Patent Document 1] internet<https: / / waymo.com / tech / > [Non-Patent Document 2] internet<https: / / waymo.com / safetyreport / > [Non-Patent Document 3] Pendleton et al., "Perception, Planning, Control, and Coordination for Autonomous Vehicles," MDPI, February 17, 2017. Summary of the Invention [Problem to be solved by the invention]

[0008] Although great advances have been made in computer-controlled perception by these vehicles, in some cases it is still underutilized, which may result in suboptimal performance, safety and / or other attributes of the autonomous driving of these vehicles.

[0009] For these and other reasons, there is a need to look to improvements in vehicles with autonomous driving capabilities. [Means for solving the problem]

[0010] According to various aspects, the present disclosure relates to autonomous driving of a vehicle or various levels of driving assistance that use computer-controlled perception by the vehicle of the environment and the vehicle itself (e.g., its ego-motion) to drive the vehicle autonomously and also to provide feedback to enhance performance, safety and / or other attributes of the autonomous driving of the vehicle, such as by adjusting the autonomous driving of the vehicle (e.g., when it is determined that a condition affecting the vehicle exists by detecting or otherwise analyzing patterns of what is perceived by the vehicle), communicating messages regarding the vehicle, and / or performing other actions regarding the vehicle.

[0011] For example, according to one aspect, the present disclosure relates to a system for autonomous driving of a vehicle or various levels of driving assistance. The system includes an interface configured to receive data from sensors of the vehicle, including, among others, a camera and a lidar sensor. The system also includes a processing entity including at least one processor, which is configured to provide sensory information regarding a perception of the vehicle's environment and a state of the vehicle based on the data from the sensors, including information regarding a 3D model of the vehicle's environment and a position of the vehicle, generate control signals for autonomously driving the vehicle based on the 3D model of the vehicle's environment and the information regarding the position of the vehicle, process the sensory information to determine whether a predetermined condition affecting the vehicle exists, and if so, perform an action regarding the vehicle based on the predetermined condition.

[0012] According to another aspect, the present disclosure relates to a system for autonomous driving or various levels of driving assistance of a vehicle. The system includes an interface configured to receive data from sensors of the vehicle, including, among others, a camera and a lidar sensor. The system also includes a processing entity including at least one processor, the processing entity configured to provide, based on the data from the sensors, perception information regarding the perception of the vehicle's environment and vehicle state, including information regarding a 3D model of the vehicle's environment and a position of the vehicle, generate control signals for autonomously driving the vehicle based on the 3D model of the vehicle's environment and the information regarding the position of the vehicle, and process the perception information to detect a pattern indicative of a predetermined condition affecting the vehicle in the perception information.

[0013] According to another aspect, the present disclosure relates to a non-transitory computer-readable medium including instructions executable by a processing device for autonomous driving or various levels of driving assistance of a vehicle, which when executed by the processing device causes the processing device to, among other things, receive data from sensors of the vehicle including cameras and lidar sensors, provide sensory information regarding a perception of the vehicle's environment and a state of the vehicle based on the data from the sensors, including information regarding a 3D model of the vehicle's environment and a position of the vehicle, generate control signals for autonomously driving the vehicle based on the 3D model of the vehicle's environment and the information regarding the vehicle's position, process the sensory information to determine whether a predetermined condition impacting the vehicle exists, and if so, perform an action with respect to the vehicle based on the predetermined condition, in addition to generating control signals for autonomously driving the vehicle based on the 3D model of the vehicle's environment and the information regarding the vehicle's position.

[0014] According to another aspect, the present disclosure relates to a non-transitory computer-readable medium including instructions executable by a processing device for autonomous driving or various levels of driving assistance of a vehicle, which instructions, when executed by the processing device, cause the processing device to, among other things, receive data from sensors of the vehicle including cameras and lidar sensors; provide sensory information regarding a perception of the vehicle's environment and a state of the vehicle based on the data from the sensors, including information regarding a 3D model of the vehicle's environment and a position of the vehicle; generate control signals for autonomously driving the vehicle based on the 3D model of the vehicle's environment and the information regarding the position of the vehicle; and process the sensory information to detect a pattern indicative of a predetermined condition affecting the vehicle in the sensory information.

[0015] According to another aspect, the present disclosure relates to a method for autonomous driving of a vehicle or various levels of driving assistance, the method including receiving data from sensors of the vehicle including, inter alia, a camera and a lidar sensor, providing sensory information regarding the vehicle's perception of an environment and a state of the vehicle based on the data from the sensors, the sensory information including information regarding a 3D model of the vehicle's environment and a location of the vehicle, generating control signals for autonomously driving the vehicle based on the 3D model of the vehicle's environment and the information regarding the vehicle's location, and processing the sensory information to determine whether a predetermined condition affecting the vehicle exists and, if so, performing an action on the vehicle based on the predetermined condition, other than generating control signals for autonomously driving the vehicle based on the 3D model of the vehicle's environment and the information regarding the vehicle's location.

[0016] According to another aspect, the present disclosure relates to a method for autonomous driving of a vehicle or various levels of driving assistance, the method including receiving data from sensors of the vehicle, including inter alia a camera and a lidar sensor, providing sensory information regarding the vehicle's perception of an environment and a state of the vehicle based on the data from the sensors, the sensory information including a 3D model of the vehicle's environment and information regarding the vehicle's position, generating control signals for autonomously driving the vehicle based on the 3D model of the vehicle's environment and information regarding the vehicle's position, and processing the sensory information to detect a pattern indicative of a predetermined condition affecting the vehicle in the sensory information.

[0017] These and other aspects of the disclosure will become apparent to those skilled in the art upon review of the description of the embodiments in conjunction with the accompanying drawings.

[0018] A detailed description of embodiments will now be given, by way of example only, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 illustrates an embodiment of an autonomously capable vehicle. [Diagram 2] FIG. 2 illustrates an example scene of a vehicle's environment. [Diagram 3] FIG. 1 illustrates an example of vehicle components. [Figure 4] FIG. 1 illustrates an embodiment of a control system for a vehicle. [Diagram 5] FIG. 2 illustrates an embodiment of a controller for a vehicle control system. [Figure 6] FIG. 2 illustrates an embodiment of a controller for a vehicle control system. [Figure 7] FIG. 2 illustrates an embodiment of a controller for a vehicle control system. [Figure 8] FIG. 2 illustrates an example process performed by a controller. [Figure 9] FIG. 13 is a diagram showing a modified example of a controller in another embodiment. [Figure 10]FIG. 13 is a diagram showing a modified example of a controller in another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] It is to be expressly understood that the description and drawings are merely for the purpose of illustrating some embodiments and are intended to aid in understanding, they are not intended to be and should not be limiting.

[0021] 1-5 show an embodiment of a vehicle 10 capable of autonomous driving (i.e., automated driving) within an environment 11 of the vehicle 10. In this embodiment, the vehicle 10 is a road vehicle, and the environment 11 includes a road 19. The vehicle 10 is designed to lawfully carry people and / or cargo on the road 19, which is part of a public road infrastructure (e.g., a highway, a highway, etc.). In this example, the vehicle 10 is a motor vehicle (e.g., a passenger car).

[0022] Vehicle 10 is autonomously operable in that it is capable of traveling without direct human control, such as by automatically steering, accelerating, and / or decelerating (e.g., braking) to travel toward a destination for at least a portion of its use. While vehicle 10 is autonomously operable in some embodiments, it may also be controlled or managed by a human driver in some circumstances. Thus, vehicle 10 may feature various levels of driving automation, ranging from partial to full driving automation using one or more advanced driver assistance systems (ADAS) (e.g., any one of SAE J3016 driving automation levels 2-5).

[0023] As described further below, this embodiment may use computer-controlled perception by vehicle 10 of the environment 11 and vehicle 10 itself (e.g., its egomotion) to drive vehicle 10 autonomously, and may also provide feedback to enhance the performance, safety and / or other attributes of the autonomous driving of vehicle 10, such as by adjusting the autonomous driving of vehicle 10 (e.g., when it is determined that some condition affecting vehicle 10 exists by detecting or otherwise analyzing patterns of what is perceived by vehicle 10), communicating messages regarding vehicle 10, and / or performing other actions regarding vehicle 10.

[0024] In this embodiment, the vehicle 10 includes a frame 12, a powertrain 14, a steering system 16, a suspension 18, wheels 20, a cabin 22, and a control system 15 configured to operate the vehicle 10 autonomously (i.e., without human control) for at least a portion of its use.

[0025] Powertrain 14 is configured to generate power for vehicle 10, including motive power for wheels 20 to propel vehicle 10 along roadway 19. To this end, powertrain 14 includes a power source (e.g., prime mover) including one or more motors. For example, in some embodiments, the power source of powertrain 14 may include an internal combustion engine, an electric motor (e.g., battery-powered), or a combination of different types of motors (e.g., an internal combustion engine and an electric motor). Powertrain 14 may transfer power from the power source to one or more wheels 20 in any suitable manner (e.g., via a transmission, a differential, a shaft that engages (i.e., directly couples) the motor, and a given one of the wheels 20).

[0026] The steering system 16 is configured to steer the vehicle 10 on a road 19. In this embodiment, the steering system 16 is configured to turn the front wheels of the wheels 20 to change their orientation relative to the frame 12 of the vehicle 10 in order to move the vehicle 10 in a desired direction.

[0027] The suspension 18 is connected between the frame 12 and the wheels 20 to allow relative movement between the frame 12 and the wheels 20 as the vehicle 10 moves along the road 19. For example, the suspension 18 may enhance handling of the vehicle 10 on the road 19 by absorbing shocks and helping to maintain static friction between the wheels 20 and the road 19. The suspension 18 may include one or more springs, dampers, and / or other resilient devices.

[0028] The cabin 22 is configured to be occupied by one or more occupants of the vehicle 10. In this embodiment, the cabin 22 is configured to interact with one or more occupants of the vehicle and includes a user interface 70 including an input portion including one or more input devices (e.g., a series of buttons, levers, dials, etc., a touch screen, a microphone, etc.) that allow the occupants of the vehicle 10 to input commands and / or other information to the vehicle 10, and an output portion including one or more output devices (e.g., a display, a speaker, etc.) that provide information to the occupants of the vehicle 10. The output portion of the user interface 70 may include an instrument panel (e.g., a dashboard) that provides indicators related to the operation of the vehicle 10 (e.g., a speedometer indicator, a tachometer indicator, etc.).

[0029] Control system 15 is configured to operate vehicle 10 autonomously (i.e., without human control), including steering, accelerating, and / or decelerating (e.g., braking) autonomous vehicle 10 as vehicle 10 travels along a route on roads 19 toward a destination. In particular, control system 15 includes controller 80 and sensing devices 82 that perform actions (such as steering, accelerating, decelerating, etc.) to control vehicle 10 to move along roads 19 toward its destination based, among other things, on a computer-controlled perception of vehicle 10's environment 11 and vehicle 10 itself (e.g., egomotion).

[0030] The vehicle 10 may be driven by its control system 15, but may also be controlled in some circumstances by a human driver, such as a passenger in the cabin 22. For example, in some embodiments, the control system 15 may selectively enable the vehicle 10 to operate autonomously (i.e., without human control) or under human control (i.e., by a human driver) in various environments (e.g., the vehicle 10 may operate in either an autonomous mode of operation or a human-controlled mode of operation). For example, in this embodiment, the user interface 70 of the cabin 22 may include an acceleration device (e.g., an accelerator pedal), a braking device (e.g., a brake pedal), and a steering device (e.g., a steering wheel) that may be operated by the human driver in the cabin 22 to control the vehicle 10 on the road 19.

[0031] Controller 80 is a processing device configured to process information received from sensing devices 82, and possibly other sources, to perform control actions for vehicle 10, including steering, accelerating, and / or decelerating vehicle 10 toward a destination on road 19. In this embodiment, controller 80 includes an interface 166, a processing entity 168, and a memory 170, which are implemented by suitable hardware and software.

[0032] The interface 166 includes one or more inputs and outputs (e.g., input / output interfaces) that enable the controller 80 to receive input signals and send output signals to other components to which the controller 80 is connected (i.e., directly or indirectly connected), including other components such as the sensing devices 82, the powertrain 14, the steering system 16, the suspension 18, and possibly a user interface 70, a communication interface 68 configured to communicate via a communication network (e.g., a cellular or other wireless network for the Internet and / or other communications) and / or with one or more other vehicles in the vicinity of the vehicle 10 (i.e., vehicle-to-vehicle communications). The controller 80 may communicate with the other components of the vehicle 10 via a vehicle bus 58 (e.g., a Controller Area Network (CAN) bus or other suitable vehicle bus).

[0033] Processing entity 168 includes one or more processors that perform processing operations to implement the functionality of controller 80. The processor of processing entity 168 may be a general-purpose processor that executes program code stored in memory 170. Alternatively, the processor of processing entity 168 may be a special-purpose processor that includes one or more pre-programmed hardware or firmware elements (e.g., an application specific integrated circuit (ASIC), an electrically erasable programmable read only memory (EEPROM), etc.), or other related elements.

[0034] Memory 170 includes one or more storage elements that store program code executed by processing entity 168 and / or data used during operation of processing entity 168 (e.g., maps, vehicle parameters, etc.). The storage elements of memory 170 may be semiconductor media (including, e.g., solid-state memory), magnetic storage media, optical storage media, and / or any other suitable type of memory. The storage elements of memory 170 may include, for example, read-only memory (ROM) elements and / or random access memory (RAM) elements.

[0035] In some embodiments, controller 80 may be associated with (e.g., include and / or interact with) one or more other control units of vehicle 10. For example, in some embodiments, controller 80 may include and / or interact with powertrain control units of powertrain 14, such as an engine control unit (ECU), a transmission control unit (TCU), etc.

[0036] The detection device 82 includes sensors 90 configured to detect aspects of the environment 11 including objects 32 (e.g., people, animals, other vehicles, inanimate objects, traffic control devices such as traffic lights and traffic signs, other obstacles, lanes, free travel areas, and / or any other tangible static or dynamic objects) in the environment 11 of the vehicle 10 and aspects of the state of the vehicle 10 including the position (e.g., location, orientation and / or movement) of the vehicle 10 and generate data indicative of these aspects, which are provided to the controller 80, which can process the data and determine actions to be autonomously performed by the vehicle 10 to continue moving towards its destination.

[0037] The sensor 90 may include any suitable sensing device. For example, in some embodiments, the sensor 90 may include: - one or more passive sensors such as a camera 92, a sound sensor, a light sensor, etc. - one or more active sensors such as a LIDAR (Light Detection and Ranging) sensor 94 (e.g. a solid-state LIDAR device that does not include rotating mechanical parts, such as a Micro-Electro-Mechanical System (MEMS) LIDAR, Flash LIDAR, Optical Phased Array LIDAR or Frequency Modulated Continuous Wave (FMCW) LIDAR, or a mechanical LIDAR that includes a rotating assembly), a radar sensor 96, an ultrasonic sensor, etc. - a location sensor 98 (e.g. based on GPS), - Vehicle speed sensor 97, - Inertial Measurement Unit (IMU) 95, including accelerometers, gyroscopes, etc.; and / or - any other detection device, may include.

[0038] Vehicle 10 may be implemented in any suitable manner. For example, in some embodiments, vehicle 10, including control system 15, may be implemented using the techniques described in https: / / waymo.com / tech / and https: / / waymo.com / safetyreport / , U.S. Patent No. 8,818,608, or U.S. Patent Application Publication No. 2014 / 0303827, or any other suitable autonomous driving techniques (e.g., one or more advanced driver assistance systems (ADAS)), all of which are incorporated herein by reference.

[0039] 5, in this embodiment, the controller 80 includes a number of modules, including a perception module 50 and a driving module 54, that autonomously drive (e.g., accelerate, decelerate, steer, etc.) and otherwise control the vehicle 10 on the road 19 toward a destination. In various embodiments, these modules may be implemented in any suitable manner (e.g., as described in Pendleton et al., “Perception, Planning, Control, and Coordination for Autonomous Vehicles,” MDPI, Feb. 17, 2017, which is incorporated herein by reference, or in any known manner).

[0040] The perception module 50 is configured to provide information 210 regarding the perception of the environment 11 of the vehicle 10 and the state of the vehicle 10 in real time based on data from the sensors 90. This information 210, referred to as "perception information," conveys knowledge of the environment 11 of the vehicle 10 and the state of the vehicle 10 (e.g., position, ego-motion, etc.) and is used by the driving module 54 to drive the vehicle 10 autonomously.

[0041] Specifically, in this embodiment, the perception module 50 is configured to generate a 3D model of the environment 11 of the vehicle 10 based on data from the sensors 90. This 3D model, referred to as a "3D environment model," includes information that provides a representation of the environment 11, including the objects 32 within the environment 11 of the vehicle 10. The 3D environment model may include properties of the objects 32, such as the class (i.e., type) of these objects 32, their shape, distance to the vehicle 10, speed, position relative to some reference point, etc. The perception module 50 may detect and potentially classify various objects 32 within the scene of the environment 11 of the vehicle 10 using any suitable known technique, such as frame-based processing, segmentation, deep learning or other machine learning algorithms, such as using deep neural networks or other artificial neural networks.

[0042] 6, perception module 50 may include a sensor data fusion module 55 configured to perform data fusion to fuse, i.e., combine, integrate and process, data from each of sensors 90, including cameras 92, lidar sensors 94, and possibly other sensors, such as radar sensors 96. Such data fusion may be implemented in any suitable manner (e.g., as described in U.S. Patent Application Publication No. 2016 / 0291154, which is incorporated herein by reference, or in any other known manner).

[0043] The perception module 50 is also configured to generate information regarding the position of the vehicle 10 within the environment 11 by performing localization of the vehicle 10 based on data from sensors 90, such as position sensor 98, vehicle speed sensor 97, and IMU 95, to determine the vehicle's position and movement. This information, referred to as "position information," is indicative of the position (e.g., location and orientation) of the vehicle 10 and / or one or more other parameters that depend on the position of the vehicle 10, such as the vehicle's movement (e.g., speed, acceleration, etc.) and / or other kinematic aspects of the vehicle 10 that can be defined as egomotion.

[0044] Thus, in this embodiment, the perception information 210 provided by the perception module 50 may include a 3D environment model and position information for the vehicle 10, as well as other information derived from the sensors 90, including data from the sensors 90 themselves.

[0045] For example, in some embodiments, perception module 50 may be implemented with LeddarVision™, available from LeddarTech®, Inc. (e.g., https: / / leddartech.com / leddarvision / ), or any other commercially available technology.

[0046] The driving module 54 is configured to determine how to drive (e.g., accelerate, decelerate, and / or steer) the vehicle 10 based on the perception information 210 provided by the perception module 50, which includes a 3D environmental model and position information of the vehicle 10, and possibly other information, and to control the vehicle 10 by sending control signals to actuators 70, such as the powertrain 14, the steering system 16, and / or other components of the vehicle 10 that control the movement and / or other operating aspects of the vehicle 10.

[0047] For example, in this embodiment, the driving module 54 may implement a planning module 40 that plans a safe path for the vehicle 10, such as by applying driving policies, adhering to traffic rules, making predictions regarding the trajectories of the vehicle 10 and other objects in the environment 11 (e.g., to avoid collisions), and / or performing other suitable actions, and a control module 56 that generates control signals that are sent to actuators 70 to autonomously move the vehicle 10 along the path.

[0048] In this embodiment, the controller 80 includes a condition detection module 48 configured to determine whether one or more predefined conditions affecting the vehicle 10 exist based on the perception information 210 provided by the perception module 50, and, if so, generate information 240 regarding the existence of the predefined conditions affecting the vehicle 10. The driving module 54 may use this information, referred to as “detected condition information,” to perform one or more actions with respect to the vehicle 10, such as adjusting the autonomous operation and / or other operation of the vehicle 10, communicating a message with respect to the vehicle 10, and / or taking another action that enhances the performance, safety, and / or other attributes of the autonomous operation of the vehicle 10. In some instances, this may provide feedback to the driving module 54 that would not otherwise be available and / or enable more rapid adjustments to the autonomous operation of the vehicle 10.

[0049] One of the predefined conditions affecting the vehicle 10 that the condition detection module 48 may detect and that the detected condition information 240 may indicate may be environmental, i.e., external to the vehicle 10 and originating from the environment 11, and generally unrelated to objects of interest in the scene that the driving module 54 uses to determine commands to send to the actuators 70. Examples of objects of interest include adjacent vehicles and pedestrians, among others. For example, in some embodiments, an environmental condition among the predefined conditions affecting the vehicle 10 may be: - the shape of the road 19 (e.g., curvature or straightness of the road, etc.), the condition of the road 19 (e.g., slipperiness of the road 19, roughness of the road 19, and / or other attributes of the surface of the road 19 that may be related to the wetness or dryness of the road 19, the surface material of the road 19 (e.g., type of pavement such as paved or unpaved road, asphalt, concrete, gravel, etc.), and / or damage to the road 19 (e.g., holes), road works on the road 19, etc.), and / or any other characteristics of the road 19, - off-road areas of the environment 11 of the vehicle 10 into which the vehicle 10 may have entered (e.g. deliberately or accidentally), - the weather in the environment 11 of the vehicle 10, such as precipitation (e.g. rain, sleet, snow, etc.), wind (e.g. wind speed or strength, wind direction, etc.), temperature, fog, and / or any other weather characteristics in the environment; the lighting in the environment 11 of the vehicle 10, such as the type of light (e.g. daylight, moonlight, artificial light, outdoors, indoors such as parking lot or tunnel lighting, etc.), the light intensity, and / or any other lighting characteristics in that environment; - objects 32 in the environment 11 of the vehicle 10, such as the density of objects 32 (e.g. high density indicating an urban or other area with relatively high traffic, low density indicating a suburban, rural or other area with relatively low traffic, etc.), the distance from the objects 32 to the vehicle 10, the time at which the objects 32 and the vehicle 10 reach each other (e.g. collide), and / or any other characteristics of the objects 32 in that environment; - any other aspect of the environment 11 of the vehicle 10, can be related to.

[0050] Alternatively, the detected condition information may indicate conditions associated with the vehicle 10 that are not directly related to the environment 11 in which the vehicle 10 operates. These conditions that the condition detection module 48 may detect and that the detected condition information 240 may indicate are vehicular, i.e., intrinsic to the vehicle 10 and originate from one or more components of the vehicle 10, such as the powertrain 14, steering system 16, suspension 18, wheels 20, and / or any other components of the vehicle 10. For example, in some embodiments, the vehicular conditions among the predetermined conditions affecting the vehicle 10 include: - a malfunction of one or more components of the vehicle 10 (e.g., excessive vibration of a component of the vehicle 10 (such as the engine or other motor of the powertrain 14), wear, damage or other deterioration of a component of the vehicle 10 (e.g., deflated or worn tires on the wheels 20 of the vehicle 10), steering abnormalities (e.g., excessive freedom of movement) in the steering system 16, headlight malfunction, abnormal noises generated by the powertrain 14, the steering system 16 or the suspension 18, etc.), and / or any other malfunction of a component of the vehicle 10; - the settings of one or more components of the vehicle 10, such as the output of the powertrain 14, the sensitivity (e.g. steering wheel movement) of the steering system 16, the stiffness and / or damping of the suspension 18, and / or any other characteristic of the settings of one or more components of the vehicle 10; and / or any other aspect of one or more components of the vehicle 10; can be related to.

[0051] Thus, the detected condition information 240 generated by the condition detection module 48 indicating one or more predetermined conditions affecting the vehicle 10 may be maintenance related and may indicate a malfunction or a need for maintenance or adjustment.

[0052] For example, the perception information 210 provided by the perception module 50 can be conceptually considered as implementing two detection streams: a main or direct stream that performs detection of objects of interest with outputs used by the driving module 54 to determine short-term actuator commands for providing motion control of the vehicle 10 within the 3D environment model, and an auxiliary stream that looks for predefined conditions within the environment 11 that are generally unrelated to the objects of interest that determine short-term motion control, or at least unrelated to the characteristics of the objects of interest. In some embodiments, both such detection streams are carried by information conveyed by at least the lidar sensor 94 and the camera 92. In other words, the information collected by the lidar sensor 94 and the camera 92 is used to look for objects of interest for short-term motion control and predefined conditions that impact longer-term driving policies and / or vehicle maintenance.

[0053] Thus, the perception information 210 provided by the perception module 50 may be further processed to detect one or more predetermined conditions affecting the vehicle 10, in addition to generating control signals for motion control in the 3D environment model.

[0054] In this embodiment, condition detection module 48 is configured to detect one or more patterns indicative of the presence of one or more predefined conditions in sensory information 210 output by perception module 50 to determine whether one or more predefined conditions affecting vehicle 10 are present. Each of these patterns, referred to as a "perceptual fingerprint," is indicative of a predefined condition affecting vehicle 10, such that detected condition information 240 generated by condition detection module 48 conveys or is otherwise based on the perceptual fingerprint.

[0055] In various examples, a given one of these perceptual fingerprints may reflect a pattern present in the 3D environment model (e.g., indicative of a predetermined condition related to road 19, weather, lighting, and / or other aspects of the environment 11 of the vehicle 10), a pattern present in the position information (e.g., of egomotion) of the vehicle 10 (e.g., indicative of a predetermined condition related to a malfunction of the vehicle 10, such as worn or deflated tires on the wheels 20, steering anomalies in the steering system 16, abnormal vibrations in the motor of the powertrain 14, and / or another aspect of one or more components of the vehicle 10), a pattern present in both the 3D environment model and the position information of the vehicle 10, or a pattern absent in either the 3D environment model or the position information of the vehicle 10 (e.g., in the data from the sensors 90). A given one of these perceptual fingerprints may also be a pattern of data from a combination of different sensors of the sensors 90 that cannot be detected by considering any of the different sensors of the sensors 90 individually.

[0056] Specifically, in this embodiment, the condition detection module 48 includes a perceptual fingerprint identification module 60 configured to detect one or more perceptual fingerprints from the perceptual information 210 provided by the perception module 50 and to cause the detected condition information 240 generated by the condition detection module 48 to convey or otherwise be based on these one or more perceptual fingerprints.

[0057] The perceptual fingerprint identification module 60 may implement any suitable pattern recognition algorithm for detecting one or more perceptual fingerprints from the perceptual information 210 provided by the perception module 50. For example, in this embodiment, the perceptual fingerprint identification module 60 implements artificial intelligence (also referred to as AI - machine intelligence or machine learning), such as an artificial neural network, a support vector machine, or any other AI unit, in the form of software, hardware, and / or a combination thereof configured to recognize a perceptual fingerprint from the perceptual information 210 provided by the perception module 50.

[0058] Specifically, in this embodiment, as shown in Figure 7, the perceptual fingerprint identification module 60 includes an artificial neural network 64 configured to detect one or more perceptual fingerprints from the perceptual information 210 provided by the perception module 50. The artificial neural network 64 may be a deep neural network (e.g., convolutional, recurrent, etc.) and / or may be implemented using any known type of neural network technology.

[0059] The artificial neural network 64 is configured to learn how to detect perceptual fingerprints from the perceptual information 210 provided by the perception module 50. Learning by the artificial neural network 64 can be accomplished using any known supervised, semi-supervised or unsupervised techniques.

[0060] In some embodiments, the artificial neural network 64 may learn by processing "training" data during a learning mode that conveys information it is looking for in the 3D environment model (similar to what would be part of the perception information 210) and / or the location of the vehicle 10, specifically data that includes one or more perceptual fingerprints indicative of one or more predefined conditions to be detected and thus affecting the vehicle 10. For example, in a situation characterized by predefined conditions of interest, a training vehicle having a sensor, perception module and artificial neural network similar to the sensor 90, perception module 50 and artificial neural network 64 of the vehicle 10 may be driven such that the perception module of the training vehicle generates training data including perceptual fingerprints (i.e., patterns) indicative of those predefined conditions, and the artificial neural network of the training vehicle may be trained to identify those perceptual fingerprints by processing the training data.

[0061] For example, in some embodiments, if the predetermined conditions to be detected include rough roads, paved roads, slippery roads, winding roads, high winds, heavy snow, sleet, artificial light, worn tires, deflated tires, abnormal vibrations in a motor (e.g., engine), headlight failure, abnormal steering, abnormal sounds, or combinations thereof (e.g., rough roads with high winds, slippery roads with high winds, slippery winding roads, slippery winding roads with high winds, slippery roads in artificial light, worn tires on slippery roads, deflated tires on rough roads, artificial light with headlight failure, etc.), or any other predetermined condition to be detected, the learn mode may include, for each given one of these predetermined conditions, determining one or more of the following characteristics that are characterized by the given predetermined condition: By driving the training vehicle in the above conditions (e.g., one or more of rough roads, one or more paved roads, one or more slippery roads, one or more winding roads, one or more weather events with high winds, one or more weather events with heavy snow, one or more weather events with sleet, conditions with artificial light in one or more areas, one or more worn tires, one or more deflated tires, one or more steering anomalies, one or more abnormal motor vibrations, one or more abnormal sounds, etc.), a perception module of the training vehicle can generate training perception information including a perceptual fingerprint indicative of the given predetermined condition, and an artificial neural network of the training vehicle can be trained to identify the perceptual fingerprint.

[0062] Thus, in some embodiments, the perceptual fingerprints detectable by the perceptual fingerprint identification module 60 and the predefined conditions they indicate affecting the vehicle 10 may be maintained in a library or other database. In some cases, the perceptual fingerprint identification module 60 may attempt to identify a perceptual fingerprint that has not been seen before, in which case the perceptual fingerprint identification module 60 may determine that the perceptual fingerprint is different or anomalous to previously encountered perceptual fingerprints. For example, in a neural network implementation, the perceptual fingerprint may be a class of information that the neural network is trained to detect by examining the sensor data. In the embodiment of Figures 6 and 7, the perceptual fingerprint identification module 60 may continually output a perceptual fingerprint that distinguishes the current environment 11 in which the vehicle 10 is operating from among the other environments that the module 60 may identify in the perceptual information 210.

[0063] This perceptual fingerprint can be used as a further input to the driving module 54 to adjust the signals sent to the actuators 70 of the vehicle 10. Thus, the driving module 54 uses two inputs, both derived from the same perceptual information 210: specifically the object of interest information that determines the short-term motion control, and the environmental input that adjusts the actual rules that determine the short-term motion control. For example, if the environmental input indicates that the information generated by the sensor falls into a fingerprint associated with a slippery road, this input will affect the short-term motion control determined by the driving module 54, such that, for example, the steering input, throttle input and brake input are adjusted differently to take into account the expected slippery road surface.

[0064] The artificial neural network 64 of the condition detection module 48 can be trained to identify perceptual fingerprints indicative of predetermined conditions affecting the vehicle 10 from the perceptual information 210 provided by the perception module 50, even if the sensors 90 are not designed to directly measure the predetermined conditions. For example, in some embodiments, the classification of patterns by the artificial neural network 64 can separate or distinguish between vibrations and natural frequencies associated with the fingerprint and rough road surfaces and other phenomena external to the vehicle 10 that may affect the vehicle 10, such that vibrations of a motor (e.g., engine) of the powertrain 14 can be identified as an anomalous pattern in the position information (e.g., egomotion) of the vehicle 10 or as a signal from the IMU 95 in the perceptual information 210 when this classification indicates a vibration source.

[0065] Thus, with further reference to FIG. 8, in this embodiment, the controller 80 may implement a process as follows.

[0066] The perception module 50 generates position information 210 including a 3D environment model and position information of the vehicle 10 based on data from the sensors 90, and the driving module 54 uses the perception information 210 to determine how to drive (e.g., accelerate, decelerate, and steer) the vehicle 10 and sends signals to actuators 70 (e.g., powertrain 14, steering system 16, etc.) so that the vehicle 10 is driven autonomously according to the signals.

[0067] Meanwhile, condition detection module 48 processes sensory information 210 provided by perception module 50 to determine whether this information contains one or more sensory fingerprints indicative of one or more predefined conditions affecting vehicle 10. If condition detection module 48 detects one or more sensory fingerprints indicative of one or more predefined conditions affecting vehicle 10, then detected condition information 240 generated by condition detection module 48 conveys or is otherwise based on those one or more sensory fingerprints.

[0068] The driving module 54 communicates one or more sensory fingerprints indicative of predetermined conditions affecting the vehicle 10, or uses the detected condition information 240 based otherwise on these one or more sensory fingerprints to perform one or more actions with respect to the vehicle 10.

[0069] For example, in some embodiments, the driving module 54 can adjust the autonomous driving and / or other operation of the vehicle 10 based on the sensory fingerprints detected by the condition detection module 48. For example, in some instances, if the detected sensory fingerprint(s) indicate that the road 19 is rough, slippery and / or winding, that there is high winds, that one or more of the wheels 20 are worn or deflated, that a motor (e.g., engine) of the powertrain 14 is vibrating abnormally, that there is a steering anomaly in the steering system 16, etc., the driving module 54 can determine short-term actuator commands and adjust logic to autonomously drive the vehicle 10 slower (e.g., slow the vehicle 10 down when going straight and / or when turning), reduce the stiffness or increase the damping of the suspension 18. Conversely, if the detected sensory fingerprint(s) indicate that the road 19 is gentle, dry and / or straight, there are no strong winds, etc., the driving module 54 may adjust the short-term control logic to drive the vehicle 10 autonomously at a higher speed (e.g., increase the speed of the vehicle 10 when going straight and / or when turning) and to increase suspension stiffness or decrease damping. The driving module 54 may send signals to actuators 70, such as the powertrain 14, steering system 16 and / or suspension 18, to adjust the autonomous driving of the vehicle 10 in this manner.

[0070] As another example, in some embodiments, the driving module 54 may communicate a message regarding the vehicle 10 to an individual (e.g., a user of the vehicle 10) or a computing device, etc., based on the sensory fingerprint(s) detected by the condition detection module 48. The message may indicate a malfunction or another problem with one or more components of the vehicle 10. For example, in some cases, if the detected sensory fingerprint(s) indicate that one or more tires on the wheels 20 are worn or deflated, one or more headlights are not working, a motor (e.g., an engine) of the powertrain 14 is vibrating abnormally, there is a steering anomaly in the steering system 16, etc., the driving module 54 may communicate a notice that the vehicle 10 should be serviced for maintenance, repair, or other service. In some cases, the message regarding the vehicle 10 may be communicated to a user interface 70 of the vehicle 10. In other embodiments, the message regarding the vehicle 10 may be communicated to a different (i.e., not part of the vehicle 10, and possibly external to the vehicle 10) communication device (e.g., a smartphone or computer) via the communication interface 68 of the vehicle 10.

[0071] In other embodiments, condition detection module 48 may be configured to determine whether one or more predetermined conditions affecting vehicle 10 exist in a variety of other ways.

[0072] For example, in some embodiments, as shown in FIG. 9 , the condition detection module 48 may be configured to compare the perception information 210 provided by the perception module 50 with other information 350 available to the controller 80, different from the 3D environment model and the (e.g., egomotion) position information of the vehicle 10, to determine whether one or more predefined conditions affecting the vehicle 10 exist. This information 350, referred to as “perception-independent reference information,” may be obtained from one or more sources independent of the sensors 90 used to generate the 3D environment model and the position information of the vehicle 10. If the condition detection module 48 determines that the perception information 210 does not match the perception-independent reference information 350, it may determine that a predefined condition affecting the vehicle 10 exists and generate valid detected condition information 240 indicative of the predefined condition, which the driving module 54 may use to perform one or more actions regarding the vehicle 10, such as adjusting the autonomous driving and / or other operation of the vehicle 10 or communicating a message regarding the vehicle 10, as described above.

[0073] In some embodiments, perception-independent reference information 350 may be derived from data 67 representing predictions regarding the vehicle 10 (e.g., regarding the vehicle's environment 11 and / or one or more operational aspects of the vehicle 10) and stored in memory 70 of the controller 80, received via communications interface 68, or otherwise utilized by the controller 80.

[0074] As an example, in some embodiments, perception-independent reference information 350 may be derived from a map 65 (e.g., a high-resolution map) representing the location of the vehicle 10 and stored in the memory 70 of the controller 80, received via the communications interface 68, or otherwise utilized by the controller 80. The map 65 may provide perception-independent reference information 350, such as the type of surface of the road 19 (e.g., paved road, unpaved road, open plain, sandy beach, etc.) that the vehicle 10 should expect to encounter at a particular location. The driving module 54 may control the vehicle 10 based on the information provided by this map 65.

[0075] By comparing the perception information 210 provided by the perception module 50 with the perception-independent reference information 350 provided by the map 65, the condition detection module 48 can determine whether the surface of the road 19 perceived by the perception module 50 (based on the 3D environment model and / or the ego-motion of the vehicle 10) is actually as predicted by the map 65, and if not, generate the detected condition information 240 to indicate what the surface of the road 19 actually is. As a result, the driving module 54 can determine whether and how to adjust the autonomous driving of the vehicle 10 based on the detected condition information 240. For example, if the driving module 54 determines based on the detected condition information 240 that the actuator settings of the estimated actuators 70 are inappropriate (e.g., suboptimal or insufficient) for driving smoothness and safety, it can send a signal to the actuators 70 to adjust the settings accordingly.

[0076] As another example, in some embodiments, perception-independent reference information 350 may be derived from a lighting model 34 representing the expected lighting (e.g., light and shadow) surrounding the vehicle 10 and stored in memory 70 of the controller 80, received via the communications interface 68, or otherwise utilized by the controller 80.

[0077] The condition detection module 48 compares the actual lighting conveyed by the perception information 210 provided by the perception module 50 (e.g., based on images from the camera 92) with the expected lighting specified by the lighting model 34 of the perception-independent reference information 350 to determine whether the actual lighting perceived by the perception module 50 is indeed as predicted by the lighting model 34, and generates the detected condition information 240 to indicate the actual lighting if not. As a result, the driving module 54 can determine whether and how to adjust the autonomous driving of the vehicle 10 based on the detected condition information 240. For example, if the driving module 54 determines based on the detected condition information 240 that the settings of the actuators 70 are inappropriate (e.g., suboptimal or insufficient) for driving smoothness and safety, it can send a signal to the actuators 70 to adjust the settings accordingly. Alternatively or additionally, the driving module 54 can also send a message indicating that maintenance or other service should be performed on the vehicle 10.

[0078] 10, the perception-independent reference 350 may be derived from the powertrain 14, the steering system 16, the suspension 18, and / or any other components that control the motion of the vehicle 10. For example, in some embodiments, the perception-independent reference 350 may indicate the steering motion of the steered ones of the wheels 20 performed by the steering system 16 as reported on the vehicle bus 58 (e.g., CAN bus), while the ego-motion of the vehicle 10 included in the perception information 210 provided by the perception module 50 may be used to estimate the perceived (e.g., past) steering motion of the steered ones of the wheels 20.

[0079] The condition detection module 48 compares the perceived steering wheel movement with the reported steering wheel movement to determine whether the steering wheel movement perceived by the perception module 50 actually corresponds to the steering wheel movement reported on the vehicle bus 58, and generates the detected condition information 240 to indicate the actual steering wheel movement if not. As a result, the driving module 54 can determine whether and how to adjust the autonomous driving of the vehicle 10 based on the detected condition information 240. For example, if the driving module 54 determines based on the detected condition information 240 that the estimated actuator settings of each of the actuators 70 in the steering system 16 are inappropriate (e.g., suboptimal or insufficient) for maneuverability, it can send a signal to those actuators 70 to adjust the settings accordingly. Alternatively or additionally, the driving module 54 can also send a message indicating that maintenance or other service should be performed on the vehicle 10.

[0080] As another example, in some embodiments, the condition detection module 48 may be configured to monitor temporal changes (changes in time) in the sensory information 210 provided by the perception module 50 to determine whether one or more predefined conditions affecting the vehicle 10 are present. For example, the condition detection module 48 may monitor temporal changes in parameters dependent on the 3D environment model, and upon observing that one or more of the parameters of the 3D environment model that are deemed to be indicative of a predefined condition affecting the vehicle 10 have changed over time in a predefined manner, generate detected condition information 240 to indicate the predefined condition, and the driving module 54 may use this information as described above to perform one or more actions with respect to the vehicle 10, such as adjusting the autonomous driving and / or other operation of the vehicle 10, or communicating a message with respect to the vehicle 10.

[0081] For example, in some embodiments, the condition detection module 48 can monitor the time-dependent statistical behavior of the 3D environment model. For example, the distribution of "distance to obstacle" or "time to collision" of objects 32 in the environment 11 of the vehicle 10 can be monitored. The desired behavior in a given driving scenario is considered to be slow and smooth changes to this distribution (e.g., below a threshold rate). The control of the vehicle 10 by the driving module 54 is determined by a driving policy, and tracking the statistics of the environment model distribution can help evaluate different policies and align them with each other.

[0082] In another variation, the sensory fingerprint may be used solely for vehicle maintenance purposes without affecting motion control. In such a case, sensory fingerprint identification module 60 may receive input from driveline sensors configured to detect specific malfunctions or driveline conditions, in addition to camera and lidar data. In this case, condition detection module 48 provides a higher level of fault detection intelligence, triggering maintenance messages when the actual effects of a fault condition reported by the driveline sensors are observed within the 3D environment model.

[0083] While the embodiments discussed above have vehicle 10 moving on land, in other embodiments vehicle 10 may move other than on land. For example, in other embodiments vehicle 10 may fly (e.g., a delivery drone or other unmanned aerial vehicle, a flying car or other personal aircraft, etc.) or move on water (e.g., a water taxi or other watercraft), and thus "driving" generally refers to the steering, control, and direction of the path of vehicle 10.

[0084] Certain additional elements that may be necessary for the operation of some embodiments are not described or shown because these elements are believed to be within the purview of those skilled in the art. Furthermore, some embodiments do not include, are devoid of, or can function without any element not specifically disclosed herein.

[0085] In some implementations, any feature of any embodiment described herein can be combined with any feature of any other embodiment described herein.

[0086] In the event of any discrepancy, inconsistency or other difference between the terms used in this specification and the terms used in any document incorporated by reference herein, the meaning of the terms used in this specification shall prevail.

[0087] While various embodiments and examples have been presented, this presentation is for purposes of illustration and not limitation, and various modifications and enhancements will become apparent to those skilled in the art. [Explanation of symbols]

[0088] 10 Vehicles 11 Environment 12 Frames 14 Powertrain 15 Control System 16 Steering system 18 Suspension 19 Road 20 wheels 22 Cabin 70 User Interface

Claims

1. 1. A system for autonomous driving of a vehicle, comprising: an interface configured to receive data from sensors of the vehicle, including cameras and lidar sensors; a processing entity including at least one processor; wherein the processing entity comprises: providing sensory information regarding the vehicle's perception of the environment and the state of the vehicle, based on the data from the sensors, the sensor information including a three-dimensional (3D) model of the vehicle's environment and information regarding the position of the vehicle; generating control signals for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle; In addition to generating the control signals for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle, processing the sensory information to determine whether a predetermined condition exists that affects the vehicle and is unrelated to other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment, and if so, performing an action with respect to the vehicle based on the predetermined condition. It is configured as follows: the processing entity is configured to detect patterns indicative of the predetermined condition in the sensory information to determine the presence of the predetermined condition independent of other vehicles, pedestrians and other objects of interest affecting the vehicle and detected in the vehicle's environment, A system characterized by:

2. the processing entity includes an artificial neural network trained to detect the pattern indicative of the predetermined condition in the sensory information; The system of claim 1 .

3. the pattern indicative of the predetermined condition in the sensory information arises from a combination of different ones of the sensors and is not detectable individually from any of the different ones of the sensors.

3. A system according to claim 1 or 2.

4. the processing entity is configured to monitor temporal changes in the sensory information to determine the presence of the predetermined condition affecting the vehicle and independent of other vehicles, pedestrians and other objects of interest detected in the vehicle's environment, A system according to any one of claims 1 to 3.

5. the predetermined conditions are external to the vehicle and originate from the vehicle's environment, and are independent of other vehicles, pedestrians, and other objects of interest that may be detected within the vehicle's environment and that may affect the vehicle; A system according to any one of claims 1 to 4.

6. the predetermined conditions, which affect the vehicle and are independent of other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment, are related to the road along which the vehicle travels; The system of claim 5.

7. the predetermined condition, which affects the vehicle and is independent of other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment, is related to at least one of the slipperiness and roughness of the road; The system of claim 6.

8. the predetermined conditions, which affect the vehicle and are independent of other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment, are related to the geometry of the road; A system according to any one of claims 6 to 7.

9. the predetermined condition, which affects the vehicle and is independent of other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment, is related to weather in the vehicle's environment; The system of claim 5.

10. the predetermined condition, which affects the vehicle and is independent of other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment, is related to an amount of precipitation in the vehicle's environment; The system of claim 9.

11. the predetermined condition, which affects the vehicle and is independent of other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment, is related to lighting in the vehicle's environment; The system of claim 5.

12. the predetermined conditions are internal to the vehicle and originate from components of the vehicle, the predetermined conditions impacting the vehicle and being independent of other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment; A system according to any one of claims 1 to 4.

13. the predetermined condition, which affects the vehicle and is independent of other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment, is related to a malfunction of the component of the vehicle; The system of claim 12.

14. the components of the vehicle are tires on wheels of the vehicle, and the predetermined condition, which affects the vehicle and is independent of other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment, relates to at least one of wear and deflation of the tires; The system of claim 13.

15. the component of the vehicle is a steering system of the vehicle, and the predetermined condition, which affects the vehicle and is independent of other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment, is associated with a steering anomaly of the steering system; The system of claim 12.

16. the component of the vehicle is a headlight of the vehicle, and the predetermined condition, which affects the vehicle and is independent of other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment, is related to a failure of the headlight of the vehicle; The system of claim 12.

17. and wherein the action with respect to the vehicle includes adjusting autonomous driving of the vehicle based on the predetermined condition.

17. A system according to any one of claims 1 to 16.

18. The adjustment of the autonomous driving of the vehicle includes a change in a speed of the vehicle.

20. The system of claim 17.

19. the action with respect to the vehicle includes communicating a message with respect to the vehicle.

17. A system according to any one of claims 1 to 16.

20. the message regarding the vehicle is indicative of a malfunction of a component of the vehicle.

20. The system of claim 19.

21. 1. A non-transitory computer-readable medium comprising instructions executable by a processing device for autonomous driving of a vehicle, the instructions, when executed by the processing device, performing: receiving data from sensors of the vehicle, including cameras and lidar sensors; providing sensory information regarding the vehicle's perception of the environment and the state of the vehicle, based on the data from the sensors, the sensor information including a three-dimensional (3D) model of the vehicle's environment and information regarding the position of the vehicle; generating control signals for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle; In addition to generating the control signals for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle, processing the sensory information to determine whether a predetermined condition exists that affects the vehicle and is unrelated to other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment, and if so, performing an action with respect to the vehicle based on the predetermined condition. causing the processing device to perform the above steps; The processing entity is configured to detect patterns indicative of the predetermined condition in the sensory information to determine the presence of the predetermined condition that affects the vehicle and is independent of other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment.

22. 1. A method for autonomous driving of a vehicle, comprising: receiving data from sensors of the vehicle, including cameras and lidar sensors; providing sensory information regarding the vehicle's perception of the environment and the state of the vehicle based on the data from the sensors, the sensor information including a three-dimensional (3D) model of the vehicle's environment and information regarding the position of the vehicle; generating control signals for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle; In addition to generating the control signals for autonomously driving the vehicle based on the 3D model of the environment of the vehicle and the information regarding the position of the vehicle, processing the sensory information to determine whether a predetermined condition exists that affects the vehicle and is unrelated to other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment, and if so, performing an action with respect to the vehicle based on the predetermined condition; Including, The method, characterized in that the processing entity is configured to detect patterns indicative of the predetermined condition in the sensory information to determine the presence of the predetermined condition that affects the vehicle and is independent of other vehicles, pedestrians, and other objects of interest detected in the vehicle's environment.

Images