Method and system for remote monitoring of a fleet of autonomous vehicles

Abstract

The present invention relates to a method for remotely monitoring a fleet of autonomous vehicles (10, 20, 30, 40), each vehicle equipped with a sensor, a storage unit and an autonomous driving unit capable of detecting obstacles and of stopping the vehicle if an obstacle is detected. The method comprises: - after the vehicle is stopped, establishing a wireless communication session between the vehicle and a remote monitoring system (50); - During a communication session, - From vehicles to surveillance systems, the information currently being acquired by the sensors, and Information previously acquired by sensors and stored in the vehicle's storage unit and transmitting - transmitting a restart command or a restart prevention command from the monitoring system to the vehicle; and then - terminating the communications session. Includes.

Description

[Technical field]

[0001] The field of the invention is that of self-driving vehicles, also called autonomous vehicles, intended to transport goods and / or people, and more particularly, the invention relates to monitoring and remote assistance of self-driving vehicles during their operation to ensure the safety of their functioning. [Background technology]

[0002] An autonomous vehicle comprises a suite of sensors, including inter alia exteroceptive sensors (e.g., cameras, Lidar), each of which can provide information about the environment in which the vehicle is located. Such a vehicle also comprises an autonomous driving unit configured to develop a spatiotemporal path for the vehicle based on information obtained by the suite of sensors, consistent with a tasked operational mission.

[0003] Unlike an automated subway, for example, an autonomous vehicle is operated in an open environment. The environment in which the vehicle is operated can therefore change very quickly and put the vehicle in a disruptive situation in an instant. A change in the environment can force the vehicle out of its operating design domain. This must be detected in order to be able to bring the vehicle to a stable and safe standstill. The autonomous driving unit of the vehicle can therefore detect in particular nearby obstacles and can stop the vehicle if a nearby obstacle is detected.

[0004] Currently deployed autonomous vehicles require a safety handoff or assistance from an on-board human driver to accommodate deviations from their operational design domain, and thus it is the human driver who makes the decision as to whether to resume the vehicle in autonomous mode after a safety stop.

[0005] Future autonomous vehicles, especially SAE Level 4 vehicles, will operate without a human driver on board. These require a solution to reliably resume autonomous driving following a safety stop automatically initiated by the vehicle as a result of the vehicle detecting a deviation from its operational design domain. Summary of the Invention [Means for solving the problem]

[0006] The present invention aims to meet this need and therefore proposes a method for remote monitoring of a fleet of autonomous vehicles, in which each vehicle in the fleet comprises exteroceptive sensors each capable of providing information about the vehicle's environment, a storage unit capable of storing the information obtained by the exteroceptive sensors, and an autonomous driving unit capable of detecting nearby obstacles based on the information obtained by the exteroceptive sensors and of stopping the vehicle in case a nearby obstacle is detected.

[0007] The method comprises the following steps: - after the vehicle is stopped by an autonomous driving unit of the vehicle, establishing a wireless communication session between the vehicle and a monitoring system remote from the vehicle; - during a wireless communication session, - From vehicles to surveillance systems, information currently acquired by the vehicle's exteroceptive sensors, constituting information about the vehicle's real-time environment; and information stored in a storage unit of the vehicle, which corresponds to information previously acquired by exteroceptive sensors of the vehicle and constitutes information about the vehicle's past environment; and transmitting - transmitting a resume command or a resume inhibit command from the monitoring system to the vehicle; and then - terminating the wireless communication session; Includes.

[0008] Some preferred, but non-limiting aspects of this method are as follows: The method further comprises selecting a time window during which information about the historical environment of the vehicle is transmitted to the monitoring system; - the time window includes the time when the vehicle was stopped, - establishing a wireless communication session includes transmitting a stop signal by the vehicle to the monitoring system; - the stop signal is sent when the duration during which the vehicle is stopped exceeds a threshold; - Stop signals convey information about why a vehicle has stopped, - Stop signals convey information about the category of nearby obstacles, - the exteroceptive sensor comprises a camera, and the step of transmitting information about the real-time environment of the vehicle comprises transmitting a video stream currently captured by the camera, and the step of transmitting information about the historical environment of the vehicle comprises transmitting an excerpt of a video previously captured by the camera and stored in a storage unit of the vehicle; - the method further comprises the step of displaying, on a display device of the monitoring system, information about the real-time environment of the vehicle and information about the historical environment of the vehicle, the information about the real-time environment of the vehicle being displayed on the right side of the display device and the information about the historical environment of the vehicle being displayed on the left side of the display device; - the method further includes the step of transmitting, by the monitoring system, to the vehicle, a command to sound the vehicle horn or a command to flash the vehicle headlights; The method further comprises the step of transmitting, by the monitoring system, a request for intervention to a local operator.

[0009] Other aspects, objects, features and advantages of the present invention will become more apparent on reading the following detailed description of some preferred embodiments thereof, given as non-limiting examples and made with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates a fleet of autonomous vehicles and a remote monitoring system for the fleet. [Diagram 2] FIG. 2 illustrates one possible arrangement of various types of information on a display device of a surveillance system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] With reference to figure 1, the present invention relates to a method for remote monitoring of a fleet of autonomous vehicles 10, 20, 30, 40. The vehicles in the fleet may operate at the same site or, conversely, may be deployed at different sites.

[0012] Each vehicle in the fleet may be equipped with exteroceptive sensors, each of which may provide information regarding the vehicle's environment. Each vehicle in the fleet may also be equipped with proprioceptive sensors, each of which may provide information regarding the vehicle's state.

[0013] Each vehicle in the fleet also carries a storage unit capable of storing the information acquired by the various sensors. The storage unit thus: - raw obstacle detection measurements, - location information derived from the fusion of various data streams, e.g. coming from Lidar, inertial measurement units and satellite-based positioning systems; - Video streams captured by cameras such as four external cameras (one capturing the front of the vehicle, one capturing the rear of the vehicle, one capturing the right of the vehicle, and one capturing the left of the vehicle) and an internal camera capturing the inside of the vehicle The storage unit may further store: - Vehicle commands, such as commands for propel, brake, steer, emergency braking, lights, or door position; - Vehicle status information, such as its speed or engine temperature, - Events such as vehicle emergency stops or sensor failures can be stored.

[0014] Each vehicle in the fleet also has an autonomous driving unit that can detect nearby obstacles based on information obtained by the exteroceptive sensors and can stop the vehicle if a nearby obstacle is detected.

[0015] To ensure a resumption of service in autonomous driving mode while guaranteeing safety, the method according to the invention comprises, after a vehicle of the fleet has been stopped by its autonomous driving unit, establishing a wireless communication session between the stopped vehicle and a monitoring system 50 remote from the vehicle. This communication session can in particular be performed via a cloud service 60. This communication session is thus not established continuously between the vehicle and the remote monitoring system, but rather as a result of the vehicle being stopped, which can limit the amount of data exchanged between the vehicle and the monitoring system.

[0016] During the wireless communication session, the vehicle transmits to the remote monitoring system both information currently being acquired by the vehicle's exteroceptive sensors (constituting information about the vehicle's real-time environment) and information stored in the vehicle's memory unit that corresponds to information previously acquired by the vehicle's exteroceptive sensors (constituting information about the vehicle's past environment).

[0017] The remote monitoring system then sends a resume or inhibit resume command to the vehicle, and the wireless communication session is then interrupted.

[0018] As mentioned above, the exteroceptive sensor comprises a camera, and in one possible embodiment, transmitting information about the vehicle's real-time environment includes transmitting a video stream currently captured by one or more cameras, and transmitting information about the vehicle's historical environment includes transmitting an abstract of video previously captured by one or more cameras and stored in a storage unit of the vehicle.

[0019] In another embodiment, which may be implemented independently of or in addition to the previous one, transmitting information regarding the vehicle's real-time environment and regarding the vehicle's historical environment includes transmitting data from one or more exteroceptive sensors other than a camera, such as, for example, information from Lidar or radar.

[0020] Surveillance systems typically include a display device on which information about the vehicle's real-time environment and information about the vehicle's historical environment is displayed. This typically involves displaying one or more real-time video streams, as well as one or more historical video excerpts. When information from other types of sensors is delivered, it is formatted to provide a human-friendly representation of the vehicle's perception.

[0021] The operator of the monitoring system can thus view information about the vehicle's past environment, which allows the operator to contextualize what occurred before the stop and understand the reason for the stop. The operator can also view the vehicle's real-time environment and examine the presence or absence of obstacles. Observing information about the past environment also allows the operator to resolve ambiguities, for example, a pedestrian does not appear in the real-time information, but analysis of the past information provides the understanding that a pedestrian is under the chassis of the vehicle. Based on the operator's analysis of the displayed information, the operator authorizes or prevents the vehicle from resuming in autonomous mode.

[0022] For example, in this way, the operator can verify that an obstacle detected by the vehicle is a false positive (a swarm of gnats, grass blown by a mower operating along the edge of the road, a cloud of dust, or bags and boxes on the lane) and can therefore be ignored. The operator can also verify that an interfering obstacle has been removed from the lane, or decide that a detected obstacle is not an inconvenience / nuisance and can be bypassed (e.g. a vehicle parked on the shoulder). The operator can also ensure that a moving obstacle (person, animal) no longer poses a danger.

[0023] The display device can be subdivided into several display parts or sections, for example by several screens, as shown in Fig. 2. In one possible embodiment, four screens are used, i.e. three screens arranged horizontally (left screen EG, center screen EC, right screen ED), with an upper screen ES arranged above the center screen EC.

[0024] Information about the vehicle's real-time environment can be displayed on the right side of the display device (e.g., right screen ED) and information about the vehicle's historical environment can be displayed on the left side of the display device (e.g., left screen EG). This mode of presenting the past on the left and the real-time on the right helps the operator perform the task of deciding whether to authorize the vehicle to resume in autonomous mode.

[0025] In one possible embodiment using four external cameras and one internal camera, each of the left and right screens is subdivided into five zones, with the center zone containing information captured by the internal camera, the upper zone containing information acquired by the external camera capturing the front of the vehicle, the lower zone containing information acquired by the external camera capturing the rear of the vehicle, the left zone containing information acquired by the external camera capturing the left of the vehicle, and the right zone containing information acquired by the external camera capturing the right of the vehicle.

[0026] The central screen EC can be used to display and / or specify control and planning information for the fleet of vehicles under surveillance. Instructions, commands or requests for intervention can be sent using it.

[0027] An upper screen ES, located above the central screen, can be used to display, for example, a map of the deployment sites for the vehicles of the fleet, together with a representation of the route that each of the vehicles will follow.

[0028] The display device therefore allows the operator to visualize the current situation (with access to the vehicle's video, a human-friendly representation of the recognition by the vehicle), to review the scene stored before the vehicle was stopped (in the form of a video, a recognition representation), to send commands to the vehicle (a command to resume in autonomous mode, a command not to resume, a command to sound its horn, a command to flash its headlights), to issue instructions (such as a speed limit on the route) or to configure the vehicle's behavior (according to the location or service, the definition of zones where any stop should generate a high safety warning, the definition of specific danger zones, a configurable time period before sending a stop warning).

[0029] In one possible embodiment, the monitoring method includes selecting, for example by said operator, a time window for which information about the vehicle's past environment is sent to the monitoring system. This time window preferably includes the time when the vehicle was stopped, i.e. the "stop time". This time window typically includes a period before the stop time (e.g. lasting 10 seconds) and a period after the stop time (e.g. lasting 20 seconds).

[0030] In one possible embodiment, establishing the wireless communication session includes transmitting a stop signal by the vehicle to the monitoring system. This stop signal may be transmitted, inter alia, when the duration for which the vehicle has been stopped exceeds a threshold. This threshold is preferably configurable, for example, depending on the area of ​​the site (e.g., for areas near stop signs where the threshold is typically higher).

[0031] The stop signal preferably conveys information about the reason the vehicle is stopped (e.g., the presence of a fixed obstacle completely blocking the lane, or the presence of a moving obstacle), which is determined by the automated driving unit of the vehicle. The stop signal also preferably conveys information about the category of the nearby obstacle (e.g., of the type resulting from an object classification algorithm, and that the obstacle is a person 90% of the time, etc.), which is determined by the automated driving unit of the vehicle.

[0032] A first use case for the surveillance system involves an improperly parked car that is parallel parked in the path of an autonomous vehicle. The vehicle automatically stops in front of the car and notifies the surveillance system. The surveillance system operator views real-time video and abstracts of past video. The surveillance system operator can command the vehicle to honk its horn or flash its headlights, and can use the real-time video to verify that the original improperly parked car is no longer impeding the vehicle's operation.

[0033] Another use case concerns an object that has fallen from a car. The vehicle automatically stops in front of this object and informs the surveillance system. The surveillance system operator views the real-time video and an excerpt of the past video. The surveillance system operator can request local operator intervention to remove the object from the lane. The surveillance system operator can verify the actual removal of the object with the real-time video and can initiate the resumption of the vehicle in automatic mode.

[0034] Yet another use case concerns a pedestrian or animal crossing close to the vehicle. The vehicle automatically stops in front of this moving object and informs the surveillance system. The operator of the surveillance system views the real-time video, as well as abstracts of past videos. The operator can command the vehicle to sound its horn. After the lane is clear, and after consulting abstracts of past videos, if necessary, to understand where the pedestrian or animal has gone, the operator can authorize the resumption of the vehicle in automatic mode.

[0035] Another use case is the loss of a sensor, for example due to deliberate damage. The vehicle automatically stops after this loss and informs the surveillance system. The surveillance system operator sees real-time video and past video clips. The operator can request local operator intervention.

[0036] In one possible embodiment, each vehicle further comprises a unit for detecting accident-prone situations, configured to detect warning signs of accidents based on information provided by the sensors. This unit can identify repetitions of unexpected behaviors in the deployment state of the vehicle, such as repeated sudden braking of the vehicle in the same area. These sudden brakings are for example the result of the installation of a new pedestrian crossing on the path of the vehicle, the repeated presence of vehicles regularly parked just in front of a pedestrian crossing resulting in occlusion and sudden braking, or the repeated presence of vehicles imperfectly parked along the path of the vehicle causing a deceleration. The warning signs thus detected can be transmitted to a monitoring unit, which can understand their cause, for example by analyzing a video captured by the vehicle's camera. Actions can then be initiated to reduce the occurrence of these dangers, such as raising awareness among certain ongoing delivery drivers, placing posts to reduce occlusion of pedestrian crossings, requesting the installation of speed bumps, requesting the correction of painted white lines affecting the path used, or updating the deployment state of the vehicle, for example by adding a new pedestrian crossing to the path.

[0037] The present invention is not limited to the method described above, but also extends to the remote monitoring system described above, in particular comprising: during a wireless communication session established with one of the vehicles of the fleet after the vehicle has been stopped by the autonomous driving unit of the vehicle, - Controlling receiving from the vehicle information about the real-time environment of the vehicle and information about the historical environment of the vehicle; - Send a resume or inhibit resume command to the vehicle The present invention also extends to a system comprising a processor configured to:

[0038] Such a monitoring system advantageously comprises a display device as described above.

[0039] Thus, one possibility is the implementation of a method for remote monitoring of a fleet of autonomous vehicles 10, 20, 30, 40, the vehicles in the fleet may operate at the same site or may be deployed at different sites, each vehicle in the fleet being equipped with an exteroceptive sensor. In addition, each vehicle may be equipped with proprioceptive sensors, each capable of providing information about the environment and / or the vehicle's state, a storage unit capable of storing the information obtained by the sensors by data fusion, vehicle commands, vehicle states and events, and an autonomous driving unit capable of detecting nearby obstacles based on the information obtained by the exteroceptive sensors and capable of stopping the vehicle if a nearby obstacle is detected. The method comprises the following steps: - after the vehicle has been stopped by its autonomous driving unit, establishing a wireless communication session between the vehicle and a monitoring system 50 remote from the vehicle; - during a wireless communication session, - From vehicles to surveillance systems, Currently acquired information about the vehicle's real-time environment, and - information stored in a storage unit of the vehicle which corresponds to previously acquired information and constitutes information about the vehicle's past environment; and transmitting - transmitting a resume command or a resume inhibit command from the monitoring system to the vehicle; and then - terminating the wireless communication session; Includes.

[0040] A monitoring system for a fleet of autonomous vehicles can also be proposed. The vehicles in the fleet can operate at the same site or can be deployed at different sites, where each vehicle in the fleet comprises exteroceptive and possibly proprioceptive sensors, each capable of providing information about the environment and / or the vehicle's state, a storage unit capable of storing the information obtained by the sensors by data fusion, vehicle commands, vehicle states and events, and an autonomous driving unit capable of detecting nearby obstacles based on the information obtained by the exteroceptive sensors and of stopping the vehicle if a nearby obstacle is detected. The system is configured to: during a wireless communication session established with one of the vehicles of the fleet after the vehicle has been stopped by its autonomous driving unit: - controlling the reception from the vehicle of information stored in a storage unit of the vehicle, the information constituting information about the real-time environment of the vehicle corresponding to information currently acquired by sensors of the vehicle and information previously acquired by sensors of the vehicle and constituting information about the historical environment of the vehicle; - Send a resume or inhibit resume command to the vehicle The device further comprises a processor configured to:

[0041] This possible method and this possible system can advantageously be supplemented by the following features, taken alone or in any technically feasible combination:

[0042] One use case for a monitoring system is for a sensor loss to automatically stop the vehicle after said loss and so inform the monitoring system.

[0043] Each vehicle further comprises an accident prone situation detection unit arranged to detect accident warning indications, which are transmitted to a monitoring unit capable of understanding their causes.

[0044] The unit is capable of identifying repetitions of unexpected behavior in the vehicle's deployment conditions.

[0045] The unit is able to identify repeated sudden braking of a vehicle in the same area.

[0046] Actions can be initiated to reduce the occurrence of the hazard.

[0047] Actions to reduce the occurrence of the hazard consist of one or more of the following actions: - Raising awareness among delivery drivers on a consistent basis; - Placing posts to reduce occlusion of pedestrian crossings; - Request the installation of speed bumps; - requesting the correction of drawn white lines; - Updating vehicle deployment status It is.

[0048] The storage unit stores a video stream capturing the interior of the vehicle.

[0049] When information is submitted from a sensor type other than one providing a video stream, this is formatted to provide a human-friendly representation of the perception by the vehicle.

[0050] A display device is subdivided into several display parts or sections, for example by several screens.

[0051] The display device of the monitoring system is used to display and / or assign management and planning information for the fleet of vehicles under monitoring, displaying maps of deployment sites for the vehicles in the fleet, and representations of the routes each vehicle will follow.

[0052] The method may further include transmitting vehicle instructions or operational configurations to the vehicle by the monitoring system. [Explanation of symbols]

[0053] 10. Self-driving cars 20 Self-driving cars 30 Self-driving cars 40 Self-driving cars 50 Surveillance System 60 Cloud Services EC center screen ED right screen EG left screen ES upper screen

Claims

1. A method for remotely monitoring a fleet of autonomous vehicles (10, 20, 30, 40), each vehicle in the fleet comprising exteroceptive sensors each capable of providing information about the vehicle's environment, a storage unit capable of storing the information obtained by the exteroceptive sensors, and an autonomous driving unit capable of detecting nearby obstacles based on the information obtained by the exteroceptive sensors and capable of stopping the vehicle if a nearby obstacle is detected, The method comprises: - after the vehicle has been stopped by the automated driving unit of the vehicle, establishing a wireless communication session between the vehicle and a monitoring system (50) remote from the vehicle; - during said wireless communication session; - from the vehicle to the monitoring system; - information currently acquired by the exteroceptive sensors of the vehicle constituting information about the real-time environment of the vehicle; and - information stored in the storage unit of the vehicle, which corresponds to information previously acquired by the exteroceptive sensors of the vehicle and constitutes information about the past environment of the vehicle; and transmitting - transmitting a restart command or a restart prohibition command from the monitoring system to the vehicle; and - terminating the wireless communication session. A method comprising:

2. The method of claim 1 , further comprising selecting a time window during which the information regarding the historical environment of the vehicle is transmitted to the monitoring system.

3. The method of claim 2 , wherein the time window includes a time when the vehicle is stopped.

4. 4. The method of claim 1, wherein the step of establishing the wireless communication session comprises the step of transmitting a stop signal by the vehicle to the monitoring system.

5. The method of claim 4 , wherein the stop signal is transmitted when the duration the vehicle has been stopped exceeds a threshold.

6. 6. The method of claim 4, wherein the stop signal conveys information regarding why the vehicle has stopped.

7. The method of claim 4 , wherein the stop signal conveys information about a category of the nearby obstacle.

8. 8. The method of claim 1, wherein the exteroceptive sensor comprises a camera, and wherein transmitting the information about the real-time environment of the vehicle comprises transmitting a video stream currently captured by the camera, and wherein transmitting the information about the historical environment of the vehicle comprises transmitting an excerpt of a video previously captured by the camera and stored in the storage unit of the vehicle.

9. 9. The method of claim 1, further comprising the step of displaying on a display device of the monitoring system the information regarding the real-time environment of the vehicle and the information regarding the historical environment of the vehicle, wherein the information regarding the real-time environment of the vehicle is displayed on the right side of the display device and the information regarding the historical environment of the vehicle is displayed on the left side of the display device.

10. 10. The method of claim 1, further comprising the step of transmitting, by the monitoring system to the vehicle, a command to sound the horn of the vehicle or a command to flash the headlights of the vehicle.

11. The method of claim 1 , further comprising the step of transmitting a request for intervention by the monitoring system to a local operator.

12. A monitoring system for a fleet of autonomous vehicles, each vehicle in the fleet comprising exteroceptive sensors each capable of providing information about the vehicle's environment, a storage unit capable of storing the information obtained via the exteroceptive sensors, and an autonomous driving unit capable of detecting nearby obstacles based on the information obtained by the exteroceptive sensors and capable of stopping the vehicle if a nearby obstacle is detected, The system further comprises: during a wireless communication session established with one of the vehicles of the fleet after the vehicle is stopped by an autonomous driving unit of the vehicle; - controlling the reception from the vehicle of information currently acquired by the exteroceptive sensors of the vehicle, the information constituting information about a real-time environment of the vehicle, and information stored in the storage unit of the vehicle corresponding to information previously acquired by the exteroceptive sensors of the vehicle and constituting information about a historical environment of the vehicle; - Transmitting a restart command or a restart prohibition command to the vehicle A monitoring system comprising a processor configured to:

Images