Remote monitoring device, remote monitoring system, and remote monitoring method

Abstract

The remote monitoring device associates blind spot management information related to a blind spot area with infrastructure support management information related to a detectable area of ​​an infrastructure sensor determined to detect the state of the blind spot area, determines that a blind spot area exists for a vehicle position based on the vehicle position and the blind spot management information, determines that infrastructure support management information associated with the blind spot management information exists, and determines that the vehicle is being remotely operated, and then determines that information acquired by the infrastructure sensor should be displayed to the monitor.

Description

[Technical field]

[0001] The present disclosure relates to a remote monitoring device, a remote monitoring system, and a remote monitoring method. [Background technology]

[0002] In recent years, the development of autonomous driving systems based on cameras and surrounding monitoring sensors (e.g., millimeter wave radar) mounted on vehicles such as automobiles has progressed, and demonstration experiments of unmanned autonomous driving systems have been conducted in limited areas. In such unmanned autonomous driving systems, development is also being conducted to realize remote operation and remote monitoring of vehicles from monitors as a safety measure. For example, Patent Document 1 proposes a technology that integrates vehicle sensors and roadside sensors to automatically switch between controlling an autonomous vehicle locally on the vehicle and controlling it remotely. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-185758 A Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, the system automatically switches between controlling an autonomous vehicle locally and remotely, but does not determine whether to use the video from an infrastructure sensor for remote control depending on whether the vehicle has a blind spot. This means that the monitor must determine whether the state of the vehicle's blind spot can be seen from the video from the infrastructure sensor, which places a burden on the monitor.

[0005] Therefore, the present disclosure has been made in consideration of the above-mentioned problems, and aims to provide technology that can reduce the burden on monitors. [Means for solving the problem]

[0006] The remote monitoring device according to the present disclosure is a vehicle that can be remotely operated by a monitor. Sensors installed in Obtained in Multiple An information integration unit that integrates first sensing information to generate a first integration result and integrates second sensing information acquired by an infrastructure sensor attached to a road to generate a second integration result; In the above, the area that cannot be detected by the first sensing information is an area that can be detected by the sensor. Static blind spot area and An area that cannot be detected by a part of the first sensing information among the detectable areas. a blind spot determination unit that determines a blind spot area including a static blind spot area and a dynamic blind spot area and determines whether the infrastructure sensor detects the state of the static blind spot area based on the first integration result and the second integration result; and an assistance determination unit that, when it determines that the blind spot area exists for the position of the vehicle, determines that the infrastructure sensor detects the state of the static blind spot area, and determines that the vehicle is being remotely operated, determines that information acquired by the infrastructure sensor should be displayed to the monitor from a human interface. Effect of the Invention

[0007] According to the present disclosure, when it is determined that infrastructure support management information corresponding to blind spot management information exists and it is determined that the vehicle is being remotely operated, it is determined that the information acquired by the infrastructure sensor should be displayed to the monitor. With such a configuration, it is possible to reduce the burden on the monitor.

[0008] The objects, features, aspects and advantages of the present disclosure will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of a remote monitoring system including a remote monitoring device according to a first embodiment. [Diagram 2] 1 is a block diagram showing a configuration of a remote monitoring system including a remote monitoring device according to a first embodiment. [Diagram 3]FIG. 2 is a diagram illustrating an example of a detectable area of ​​a vehicle sensor. [Figure 4] FIG. 2 is a diagram illustrating an example of a detectable area of ​​an infrastructure sensor. [Diagram 5] FIG. 2 is a diagram showing an example of a blind spot area of ​​a vehicle sensor; [Figure 6] 1A and 1B are diagrams illustrating an example in which an infrastructure sensor detects a state of a blind spot area of ​​a vehicle sensor. [Figure 7] 1A and 1B are diagrams illustrating an example in which an infrastructure sensor detects a state of a blind spot area of ​​a vehicle sensor. [Figure 8] FIG. 11 is a diagram illustrating an example of blind spot management information. [Figure 9] FIG. 11 is a diagram illustrating an example of infrastructure support management information. [Figure 10] 4 is a schematic diagram for explaining an example of the operation of the remote monitoring device according to the first embodiment; FIG. [Figure 11] 5 is a flowchart showing an operation example of the remote monitoring device according to the first embodiment. [Figure 12] 5 is a flowchart showing an operation example of the remote monitoring device according to the first embodiment. [Figure 13] 5 is a flowchart showing an operation example of the remote monitoring device according to the first embodiment. [Figure 14] 5 is a flowchart showing an operation example of the remote monitoring device according to the first embodiment. [Figure 15] 5 is a flowchart showing an operation example of the remote monitoring device according to the first embodiment. [Figure 16] FIG. 11 is a block diagram showing a configuration of a remote monitoring system including a remote monitoring device according to a second embodiment. [Figure 17] FIG. 13 is a diagram for explaining an example of a pattern. [Figure 18] FIG. 13 is a diagram for explaining an example of a pattern. [Figure 19] 10 is a flowchart showing an example of the operation of the remote monitoring device according to the second embodiment. [Figure 20] 10 is a flowchart showing an example of the operation of the remote monitoring device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] <Embodiment 1> 1 and 2 are block diagrams showing a configuration of a remote monitoring system including a remote monitoring device 1 according to the present embodiment 1. As shown in Fig. 1, the remote monitoring system includes the remote monitoring device 1, an autonomous vehicle 2 that can be remotely operated by an observer, an RSU (Road Side Unit) 3, and an observer HMI 4 that is a human interface. In the present embodiment 1, the remote monitoring device 1 will be described on the assumption that it is mounted on a cloud server, but it may also be mounted on an edge server or a roadside device, for example.

[0011] The autonomous vehicle 2 is equipped with peripheral recognition sensors such as millimeter wave radar and LiDAR (Light Detection and Ranging Laser Imaging Detection and Ranging), cameras, a Global Navigation Satellite System (GNSS), and a control ECU (Electronic Control Unit) that controls the accelerator, brakes, and steering. The autonomous vehicle 2 can determine areas in which it can travel autonomously using the control ECU, etc., and can travel autonomously through autonomous driving.

[0012] The autonomous vehicle 2 transmits information such as the position, speed, and type of surrounding obstacles detected by the mounted surrounding recognition sensor, and the position of the autonomous vehicle 2 acquired from the GNSS, as first sensing information to the remote monitoring device 1. The autonomous vehicle 2 also transmits camera images captured by the mounted camera and vehicle information to the remote monitoring device 1. The vehicle information includes information on whether the autonomous vehicle 2 is being remotely operated and information on whether the autonomous vehicle 2 is being driven autonomously. The autonomous vehicle 2 may transmit the first sensing information and the like spontaneously (for example, periodically), or may transmit the information after receiving an acquisition request from the remote monitoring device 1. The autonomous vehicle 2 may also transmit the camera images and vehicle information included in the first sensing information. In the present embodiment 1, the surrounding recognition sensor and the camera are included in the concept of a vehicle sensor, and for convenience in the following description, the vehicle sensor and the autonomous vehicle 2 will not be distinguished from each other.

[0013] The autonomous vehicle 2 receives information obtained by integrating the detection results of the sensors from the remote monitoring device 1 and uses the information for determining the driving route during autonomous driving. The autonomous vehicle 2 is also configured to be controllable by remote operation from a monitor.

[0014] The RSU3 is connected to peripheral monitoring sensors such as millimeter-wave radar, LiDAR, and GNSS, and is installed at intersections, bus stops, and the like. The RSU3 sequentially transmits detection results of the peripheral monitoring sensors as second sensing information to the remote monitoring device 1. The RSU3 is also connected to a camera, and transmits camera images captured by the camera to the remote monitoring device 1. The RSU3 may transmit the second sensing information spontaneously (e.g., periodically), or may transmit the second sensing information after receiving an acquisition request from the remote monitoring device 1. The RSU3 may transmit the camera images by including them in the second sensing information. In the first embodiment, the RSU3, the peripheral monitoring sensors, and the cameras are included in the concept of infrastructure sensors attached to roads, and in the following description, for convenience, the infrastructure sensors and the RSU3 are described without distinction.

[0015] The monitor HMI 4 displays to the monitor the camera images transmitted from the autonomous vehicle 2 and the RSU 3 via the remote monitoring device 1. When the monitor HMI 4 receives the first sensing information and the second sensing information from the autonomous vehicle 2 and the RSU 3 via the remote monitoring device 1, the monitor HMI 4 may display the first sensing information and the second sensing information to the monitor.

[0016] Furthermore, the monitor HMI 4 transmits control information for the monitor to remotely operate the accelerator, brake, steering, and so on of the autonomous vehicle 2 to the autonomous vehicle 2 via the remote monitoring device 1. The monitor may request to change the camera image displayed on the monitor HMI 4 for remote operation, and the monitor HMI 4 transmits the request from the monitor to change the camera image to the remote monitoring device 1. This enables the monitor HMI 4 to receive the changed camera image from the remote monitoring device 1 and display the changed camera image.

[0017] Fig. 2 is a block diagram specifically showing the configuration of Fig. 1. The remote monitoring device 1 of Fig. 2 includes a processor 10, a storage device 11, and an external interface 12.

[0018] The processor 10 is connected to other hardware via a signal line and controls the other hardware. The processor 10 is, for example, an integrated circuit (IC) for executing instructions written in a program to perform processes such as data transfer, calculation, processing, control, and management. The processor 10 has, for example, an arithmetic circuit, and a register and cache memory in which instructions and information are stored. The processor is, for example, a central processing unit (CPU), a digital signal processor (DSP), or a graphics processing unit (GPU). As a result of executing the above processes, the functions of an information integration unit 101, a blind spot determination unit 102, a management information generation unit 103, an assistance determination unit 104, a road-vehicle cooperation assistance unit 105, a video transfer unit 106, and a management information update unit 107 are realized in the processor 10.

[0019] 2, the number of processors 10 is one. However, the number of processors 10 may be multiple, and for example, the multiple processors 10 may execute a program in cooperation with each other, so that each processor 10 realizes each function.

[0020] The storage device 11 stores the blind spot management information and the infrastructure support management information. The storage device 11 may be, for example, a random access memory (RAM), a hard disk drive (HDD), or a solid state drive (SSD). The storage device 11 may also be a portable storage medium such as a secure digital (SD) memory card (registered trademark), a compact flash (CF), a NAND flash, a flexible disk, an optical disk, a compact disk, a Blu-ray disk (registered trademark), or a digital versatile disk (DVD).

[0021] The external interface 12, which is a communication unit, communicates with the autonomous vehicle 2, the RSU 3, and the monitor HMI 4. In the first embodiment, the information integration unit 101, the road-vehicle cooperative support unit 105, the video transfer unit 106, and the management information update unit 107 of the remote monitoring device 1 use the external interface 12 to transmit and receive at least one of data to and from the autonomous vehicle 2, the RSU 3, and the monitor HMI 4. In this specification, for example, at least one of A, B, C, ..., and Z means any one of all combinations of one or more items extracted from the group of A, B, C, ..., and Z.

[0022] The external interface 12 includes, for example, a receiver that receives data from a communication device and a transmitter that transmits data to the communication device. The external interface 12 uses a communication method such as a cellular network such as LTE (Long Term Evolution) (registered trademark) and a fifth generation mobile communication system (5G), a dedicated short range communication (DSRC) dedicated to vehicle communication, and a cellular-V2X (vehicle-to-x) system.

[0023] The information integrator 101 fuses the first sensing information acquired by the autonomous vehicle 2 to generate a first integrated result, and fuses the second sensing information acquired by the RSU3 to generate a second integrated result. For example, when the autonomous vehicle 2 acquires a plurality of pieces of first sensing information, the information integrator 101 generates a first integrated result by integrating the plurality of pieces of first sensing information with reference to the same obstacle indicated by the plurality of pieces of first sensing information, and outputs the first integrated result to the blind spot determination unit 102. Similarly, when the RSU3 acquires a plurality of pieces of second sensing information, the information integrator 101 generates a second integrated result by integrating the plurality of pieces of second sensing information with reference to the same obstacle indicated by the plurality of pieces of second sensing information, and outputs the second integrated result to the blind spot determination unit 102.

[0024] The blind spot determination unit 102 determines whether the RSU 3 detects a state of a blind spot area of ​​the autonomously driven vehicle 2, that is, whether the RSU 3 is capable of supporting the autonomously driven vehicle 2, based on the first and second integration results generated by the information integration unit 101. The blind spot determination unit 102 may estimate a blind spot area common to the vehicle sensor and the infrastructure sensor based on the first and second integration results, information on the detectable area of ​​the vehicle sensor, and information on the detectable area of ​​the infrastructure sensor, and output the estimated blind spot area to the management information generation unit 103. The detectable area of ​​the vehicle sensor and the detectable area of ​​the infrastructure sensor may be set in advance in the blind spot determination unit 102.

[0025] FIG. 3 is a diagram showing an example of a detectable area 2a of a vehicle sensor. As shown in FIG. 3, it is assumed that the vehicle sensor is installed so as to be able to monitor the surroundings (360 degrees) of the autonomous vehicle 2. The detectable area 2a of the vehicle sensor includes the position and traveling direction of the autonomous vehicle 2. The detectable area 2a of the vehicle sensor also includes a detection angle defined clockwise around the autonomous vehicle 2, with the traveling direction of the autonomous vehicle 2 being 0 degrees, and a detectable distance defined for each detection angle. FIG. 3 shows an example of an elliptical detectable area 2a in which the detection angle is in 0.5 degree increments, the detectable distance in the forward and backward directions (0,180 degrees) of the autonomous vehicle 2 is 100 m, and the detectable distance in the left and right directions (90,270 degrees) of the autonomous vehicle 2 is 50 m.

[0026] Fig. 4 is a diagram showing an example of a detectable area 3a of an infrastructure sensor. In the example of Fig. 4, the infrastructure sensor is installed at a predetermined angle, and the detectable area 3a of the infrastructure sensor is different from the detectable area 2a of the vehicle sensor. Fig. 4 shows an example of a detectable area 3a where the installation angle of the infrastructure sensor is 0 degrees and the detectable area 3a spreads at 30 degrees in the left and right directions.

[0027] Fig. 5 is a diagram showing an example of a blind spot area 2b of a vehicle sensor. As the detectable area of ​​the vehicle sensor in Fig. 5, an elliptical detectable area 2a similar to that in Fig. 3 is shown. In the example of Fig. 5, a large vehicle and a building are present in front and to the left rear of the autonomously driven vehicle 2. In this case, the blind spot determination unit 102 estimates, as the blind spot area 2b of the vehicle sensor, the area on the opposite side of the large vehicle and the building as seen from the autonomously driven vehicle 2, among the areas inside the outline of the detectable area 2a of the vehicle sensor.

[0028] 6 and 7 are diagrams showing an example in which an infrastructure sensor detects the state of a blind spot area 2b of a vehicle sensor.

[0029] In the example of Fig. 6, the detectable area 3a of the infrastructure sensor includes the entire blind spot area 2b of the vehicle sensor in a situation in which the autonomous vehicle 2, a large vehicle, and a building are present in Fig. 5. In this case, the second sensing information and camera video of the infrastructure sensor make it possible to recognize the state of the entire blind spot area 2b of the autonomous vehicle 2, and the blind spot determination unit 102 determines that the infrastructure sensor can detect the state of the blind spot area 2b of the autonomous vehicle 2 and assist the autonomous vehicle 2.

[0030] In the example of Fig. 7, the detectable area 3a of the infrastructure sensor includes a part of the blind spot area 2b of the vehicle sensor. In this case, the second sensing information and camera image of the infrastructure sensor enable recognition of the state of a part of the blind spot area 2b of the autonomously driven vehicle 2, and the blind spot determination unit 102 determines that the infrastructure sensor can detect the state of the blind spot area 2b of the autonomously driven vehicle 2 and assist the autonomously driven vehicle 2. However, in the example of Fig. 7, a blind spot area 2c common to the vehicle sensor and the infrastructure sensor occurs.

[0031] When a common blind spot area 2c is generated by another vehicle such as a large vehicle and the remote monitoring device 1 is capable of receiving sensing information from the other vehicle, the remote monitoring device 1 may transmit the sensing information from the other vehicle to the autonomously driving vehicle 2. With this configuration, the autonomously driving vehicle 2 can continue autonomous driving in the blind spot area 2c common to the vehicle sensor and the infrastructure sensor. Alternatively, the remote monitoring device 1 may transmit camera images from the other vehicle to the monitor HMI 4. With this configuration, the monitor can check the state of the blind spot area 2c common to the vehicle sensor and the infrastructure sensor.

[0032] The management information generation unit 103 in Figure 2 associates blind spot management information regarding the blind spot area 2b of the autonomously driving vehicle 2 with infrastructure support management information regarding the detectable area 3a of the infrastructure sensor that is determined to detect the state of the blind spot area 2b, and stores them in the storage device 11.

[0033] FIG. 8 is a diagram showing an example of blind spot management information. The management information generating unit 103 generates the blind spot management information based on, for example, the position of the autonomous vehicle 2, the sensor result, and the blind spot area 2b. The blind spot management information defines, for example, the blind spot area 2b of the autonomous vehicle 2 as a list of latitude and longitude on the driving route of the autonomous vehicle 2. The blind spot area 2b is expressed, for example, by a boundary point sequence (polygon) of the blind spot area 2b. When the infrastructure sensor detects the state of the blind spot area 2b, the blind spot determining unit 102 determines that the infrastructure sensor can support the autonomous vehicle 2 for the blind spot area 2b, and the identifiers of the RSU 3 and the surrounding monitoring sensor are associated with the blind spot area 2b by the supporting RSU. This allows the remote monitoring device 1 to quickly extract information required for the blind spot area 2b of the autonomous vehicle 2.

[0034] 9 is a diagram showing an example of infrastructure support management information. The management information generating unit 103 generates the infrastructure support management information based on, for example, the position of the infrastructure sensor, the sensor result, and the detectable area 3a. The infrastructure support management information defines, for example, the detectable area 3a of the infrastructure sensor and the blind spot area as a list of latitude and longitude on the driving route of the autonomous driving vehicle 2. By managing the detectable area 3a of the infrastructure sensor and the blind spot area of ​​the infrastructure sensor in advance, it is expected that the calculation processing of the remote monitoring device 1 can be accelerated and the processing load can be reduced.

[0035] In the above, the blind spot management information and the infrastructure support management information are defined as a list of latitude and longitude, but the present invention is not limited to this. For example, the blind spot management information and the infrastructure support management information may include an altitude, and may be defined in a relative coordinate system based on a specific position, rather than in an absolute coordinate system.

[0036] The assistance determination unit 104 in FIG. 2 determines whether or not a blind spot area 2b exists relative to the position of the autonomous vehicle 2 based on the position of the autonomous vehicle 2 and the blind spot management information. When a driving route is set in advance for the autonomous vehicle 2, the assistance determination unit 104 determines whether or not a blind spot area 2b exists on the driving route. The assistance determination unit 104 also determines whether or not infrastructure assistance management information associated with the blind spot management information exists. That is, the assistance determination unit 104 determines whether or not an infrastructure sensor can detect the state of the blind spot area 2b. The assistance determination unit 104 also determines whether or not the autonomous vehicle 2 is autonomously driving and whether or not the autonomous vehicle 2 is being remotely controlled based on the vehicle information from the autonomous vehicle 2.

[0037] Then, when the assistance determination unit 104 determines that a blind spot area 2b exists for the position of the autonomously driven vehicle 2, determines that infrastructure assistance management information associated with the blind spot management information exists, and determines that the autonomously driven vehicle 2 is being remotely operated, it determines that the camera footage acquired by the infrastructure sensor that can provide assistance should be displayed to the observer from the observer HMI 4. Note that the determination that the footage should be displayed to the observer from the observer HMI 4 includes a determination to display the footage to the observer from the observer HMI 4.

[0038] On the other hand, when the assistance determination unit 104 determines that a blind spot area 2b exists for the position of the autonomously driving vehicle 2, determines that infrastructure assistance management information corresponding to the blind spot management information exists, and determines that the autonomously driving vehicle 2 is operating autonomously, it determines that the second integrated result of the infrastructure sensor that can provide assistance should be transmitted to the autonomously driving vehicle 2.

[0039] When the assistance determination unit 104 determines that the second integrated result of the infrastructure sensors capable of providing assistance should be transmitted to the autonomously driven vehicle 2, the road-vehicle cooperation assistance unit 105 transmits the second integrated result to the autonomously driven vehicle 2 using the external interface 12. Note that the second integrated result may include information on the blind spot areas 2b and 2c.

[0040] When the assistance determination unit 104 determines that the camera image acquired by the infrastructure sensor capable of providing assistance should be displayed to the monitor by the monitor HMI 4, the image transfer unit 106, which is an information transfer unit, transmits the camera image to the monitor HMI 4 using the external interface 12.

[0041] When a monitor requests to change the display of the camera video on the monitor HMI 4, the management information update unit 107 preferentially displays the changed camera video on the monitor HMI 4. As a result, for example, when a monitor requests to change the shooting angle of the camera video of the displayed infrastructure sensor, the camera video of the infrastructure sensor with the changed shooting angle is displayed first on the monitor HMI 4 from the next time onwards. For example, when a monitor requests to change the camera video of the displayed infrastructure sensor to another camera video that can display the same blind spot area 2b as the camera video, the other camera video is displayed first on the monitor HMI 4 from the next time onwards.

[0042] <Operation> 10 to 15 are diagrams for explaining an example of the operation of the remote monitoring device 1 according to the present embodiment 1. The operation of the remote monitoring device 1 according to the present embodiment 1 corresponds to the operation of the remote monitoring method or remote monitoring system according to the present embodiment 1. Moreover, the operation of the remote monitoring device 1 according to the present embodiment 1 corresponds to the processing of the remote monitoring program according to the present embodiment 1.

[0043] Fig. 10 is a schematic diagram for explaining an example of the operation of the remote monitoring device 1. Fig. 10 illustrates a situation in which an autonomous vehicle 2 is traveling and a blind spot area of ​​the autonomous vehicle 2 is present near an intersection ahead of the autonomous vehicle 2. Fig. 10 illustrates a traffic jam 6 occurring in the blind spot area of ​​the autonomous vehicle 2, and the RSU 3 detects the state of the traffic jam 6 and notifies the remote monitoring device 1. In this case, the remote monitoring device 1 is configured to transmit the second integration result of the RSU 3 to the autonomous vehicle 2, and transmit the video ahead of the autonomous vehicle 2 and the camera video of the RSU 3 detecting the blind spot area to the monitor HMI 4 that displays the video to the monitor.

[0044] 11 is a flowchart showing an example of an operation for generating blind spot management information from among the operation examples of the remote monitoring device 1 according to the present embodiment 1. This operation is performed as a pre-processing.

[0045] In step S1, the autonomous vehicle 2 acquires first sensing information from a vehicle sensor that indicates the position of the vehicle on the travel route, etc. The acquisition may be performed once, but it is preferable to acquire the information multiple times.

[0046] In step S2, the information integration unit 101 integrates the first sensing information for the same point on the travel route to generate a first integration result. By integrating the first sensing information for the same position, it becomes possible to identify the blind spot area 2b of the autonomously driven vehicle 2 at that position.

[0047] In step S3, the blind spot determination unit 102 determines whether the blind spot area is a dynamic blind spot area or a static blind spot area based on the first integration result. For example, the blind spot determination unit 102 determines, in the first integration result of all the first sensing information, an area of ​​the detectable area 2a that cannot be detected by the vehicle sensor as a static blind spot area. Also, the blind spot determination unit 102 determines, in the first integration result of only a part of the first sensing information, an area of ​​the detectable area 2a that cannot be detected by the vehicle sensor as a dynamic blind spot area.

[0048] In step S4, the management information generation unit 103 generates blind spot management information regarding the static blind spot areas, and stores and manages it in the storage device 11. Note that dynamic blind spot areas are not managed as blind spot management information because they differ for each travel of the autonomously driven vehicle 2. Also, at this point, the blind spot management information and the infrastructure support management information are not associated with each other. After that, the processing of FIG. 11 ends.

[0049] In the process described below, the blind spot determination unit 102 determines the blind spot area 2b including the static blind spot area and the dynamic blind spot area based on the first integration result and the static blind spot area indicated by the blind spot management information managed in advance. With this configuration, the blind spot determination unit 102 can determine the entire blind spot area 2b simply by determining the dynamic blind spot area from the detectable area 2a other than the static blind spot area based on the first integration result in real time. This reduces the load of the blind spot determination unit 102 in determining the blind spot area 2b.

[0050] 12 is a flowchart showing an example of an operation for generating infrastructure support management information among the operation examples of the remote monitoring device 1 according to the embodiment 1. This operation is performed as a pre-processing, and is basically the same as the generation of the blind spot management information in FIG.

[0051] In step S11, the RSU 3 acquires the second sensing information of the infrastructure sensor. The acquisition may be performed once, but it is preferable to acquire the second sensing information multiple times.

[0052] In step S12, the information integration unit 101 integrates the second sensing information to generate a second integration result. By integrating the sensing information at the same position, it becomes possible to identify the detectable area 3a of the infrastructure sensor at that position and the blind spot area.

[0053] In step S13, the blind spot determination unit 102 determines a dynamic detectable area and a static detectable area based on the second integration result. For example, the blind spot determination unit 102 determines an area that can be detected by an infrastructure sensor in the second integration result of all the second sensing information as a static detectable area. Also, the blind spot determination unit 102 determines an area that can be detected by an infrastructure sensor in the second integration result of only a part of the second sensing information as a dynamic detectable area.

[0054] In step S14, the management information generation unit 103 generates infrastructure support management information related to the static detectable areas, and stores and manages the information in the storage device 11. Areas that cannot be detected by the infrastructure sensors are set as blind spot areas of the infrastructure sensors. Note that dynamic detectable areas are not managed as infrastructure support management information because they differ for each travel of the autonomously driven vehicle 2. Also, at this point, the blind spot management information and the infrastructure support management information are not associated with each other. Thereafter, the processing of FIG. 12 ends.

[0055] In the process described below, the blind spot determination unit 102 determines the detectable area 3a including the static detectable area and the dynamic detectable area based on the second integration result and the static detectable area managed in advance. With this configuration, the blind spot determination unit 102 can determine the entire detectable area 3a simply by determining the dynamic detectable area from the areas other than the static detectable area based on the real-time second integration result. This reduces the load of the process of determining the detectable area 3a on the blind spot determination unit 102.

[0056] Fig. 13 is a flowchart showing an example of an association operation between blind spot management information and infrastructure support management information among the operation examples of the remote monitoring device 1 according to embodiment 1. This operation is performed sequentially (e.g., periodically) after the processes of Figs. 11 and 12, for example.

[0057] In step S21, the information integrator 101 acquires first sensing information from the autonomously driven vehicle 2 and acquires second sensing information from the RSU 3.

[0058] In step S22, the information integrating unit 101 integrates the first sensing information to generate a first integrated result, and integrates the second sensing information to generate a second integrated result. When sensing information of other vehicles is available, the information integrating unit 101 may also integrate that information.

[0059] In step S23, the blind spot determination unit 102 determines a blind spot area 2b including a static blind spot area and a dynamic blind spot area based on the first integration result and a static blind spot area managed in advance. The blind spot determination unit 102 determines a detectable area 3a including a static detectable area and a dynamic detectable area based on the second integration result and a static detectable area managed in advance. Note that the blind spot area 2b may include a static blind spot area without including a dynamic blind spot area, and the detectable area 3a may include a static detectable area without including a dynamic detectable area.

[0060] The blind spot determination unit 102 determines whether the detectable area 3a of the RSU 3 overlaps with at least a part of the blind spot area 2b of the autonomously driven vehicle 2. In other words, it determines whether the RSU 3 detects the state of the blind spot area 2b of the autonomously driven vehicle 2.

[0061] If it is determined that the RSU3 detects the state of the blind spot area 2b of the autonomous vehicle 2, processing proceeds to step S24, and if it is not determined that the RSU3 detects the state of the blind spot area 2b of the autonomous vehicle 2, the operation of Figure 13 ends.

[0062] In step S24, the management information generation unit 103 associates the blind spot management information related to the blind spot area 2b of the autonomously driven vehicle 2 with the infrastructure support management information related to the detectable area 3a of the infrastructure sensor determined to detect the state of the blind spot area 2b. The management information generation unit 103 performs this association using the support RSU in Fig. 8. After that, the operation in Fig. 13 ends.

[0063] 14 is a flowchart showing an example of the overall operation of the remote monitoring device 1 according to the embodiment 1. This operation is performed sequentially (for example, periodically) after the processes of FIGS.

[0064] In step S31, the autonomous vehicle 2 travels according to a specified travel route. During travel, the information integrator 101 periodically receives first sensing information, camera footage, and vehicle information from the autonomous vehicle 2. The information integrator 101 periodically receives second sensing information and camera footage from the RSU 3.

[0065] In step S32, the information integrator 101 outputs the position of the autonomously driven vehicle 2 and the vehicle information to the assistance determination unit 104.

[0066] In step S33, the assistance determination unit 104 determines, based on the position of the autonomous vehicle 2 and the blind spot management information, whether or not a blind spot area 2b of the autonomous vehicle 2 exists on the driving route of the autonomous vehicle 2. If it is determined that a blind spot area 2b exists, the process proceeds to step S34, and if it is not determined that a blind spot area 2b exists, the operation in FIG. 14 ends.

[0067] In step S34, the assistance determination unit 104 determines whether or not infrastructure assistance management information associated with the blind spot management information exists, that is, whether or not the state of the blind spot area 2b of the autonomously driven vehicle 2 can be assisted (i.e., can be detected) by an infrastructure sensor. If it is determined that infrastructure assistance management information associated with the blind spot management information exists, the process proceeds to step S35, and if it is not determined that the infrastructure assistance management information exists, the operation of Fig. 14 ends. Note that, if sensing information of another vehicle can be used, the assistance determination unit 104 may determine whether or not the state of the blind spot area 2b of the autonomously driven vehicle 2 can be detected by a sensor of the other vehicle, similar to the determination for the infrastructure sensor.

[0068] In step S35, the assistance determination unit 104 determines whether the autonomous vehicle 2 is autonomously driven or remotely controlled based on the vehicle information. If it is determined that the autonomous vehicle 2 is autonomously driven, the process proceeds to step S36, and if it is determined that the autonomous vehicle 2 is remotely controlled, the process proceeds to step S38.

[0069] In step S36, the assistance determination unit 104 outputs, to the road-vehicle cooperation assistance unit 105, the infrastructure assistance management information for which it has been determined in step S34 that assistance is possible.

[0070] In step S37, the road-vehicle cooperation assistance unit 105 transmits the second integration result of the infrastructure sensors that can provide assistance corresponding to the infrastructure assistance management information from the assistance determination unit 104 to the autonomously driven vehicle 2, and the autonomously driven vehicle 2 uses the second integration result for autonomous driving. After that, the operation in FIG. 14 ends.

[0071] In step S38, the assistance determination unit 104 outputs the infrastructure assistance management information for which it has been determined in step S34 that assistance is possible, to the video transfer unit 106.

[0072] In step S39, the image transfer unit 106 transmits the camera image of the infrastructure sensor capable of providing support corresponding to the infrastructure support management information from the support determination unit 104 to the monitor HMI 4, and the monitor HMI 4 displays the camera image to the monitor. After that, the operation in FIG. 14 ends.

[0073] FIG. 15 is a flowchart showing an example of a video changing operation among the operational examples of the remote monitoring device 1 according to the first embodiment.

[0074] In step S41, the monitor is requested by the monitor HMI 4 to remotely operate the autonomous vehicle 2.

[0075] In step S42, the monitor HMI 4 displays to the monitor the camera video of the autonomous vehicle 2 and the camera video of the infrastructure sensor transmitted from the video transfer unit 106. As a result, when the autonomous vehicle 2 is being remotely operated, the monitor can remotely operate the autonomous vehicle 2 while viewing the camera video of the autonomous vehicle 2 and the camera video of the infrastructure sensor that can assist in visual recognition of blind spot areas of the camera video.

[0076] In step S43, if the monitor determines that the camera image of the infrastructure sensor displayed on the monitor HMI 4 is inappropriate for remote operation, the monitor requests the monitor HMI 4 to change the display of the camera image of the infrastructure sensor.

[0077] In step S44, when the management information update unit 107 receives the change request from the monitor HMI 4, it transmits the changed camera image of the infrastructure sensor from the image transfer unit 106 to the monitor HMI 4, and the monitor HMI 4 displays the transmitted changed camera image. As a result, for example, the camera image of an appropriate infrastructure sensor out of multiple infrastructure sensors that can support the viewing of one blind spot area 2b is displayed on the monitor HMI 4.

[0078] In step S45, the management information update unit 107 updates the infrastructure support management information of the changed camera image. As a result, in the next step S42, the changed camera image is preferentially displayed on the monitor HMI 4. Then, the operation in FIG. 15 ends.

[0079] <Summary of the first embodiment> According to the remote monitoring device 1 of the present embodiment 1 as described above, when it is determined that infrastructure support management information associated with the blind spot management information exists and it is determined that the autonomous vehicle 2 is being remotely operated, it is determined that the camera image of the infrastructure sensor should be displayed to the monitor. With this configuration, it is possible to reduce the burden on the monitor in determining whether the state of the blind spot area 2b of the autonomous vehicle 2 can be visually confirmed by the camera image of the infrastructure sensor. Note that, as in the present embodiment 1, if it is determined in advance whether the blind spot area 2b of the autonomous vehicle 2 exists before the autonomous vehicle 2 travels, it is possible to quickly provide appropriate support and reduce the processing load of the remote monitoring device 1 while the autonomous vehicle 2 is traveling.

[0080] Furthermore, in the first embodiment, when it is determined that infrastructure assistance management information associated with the blind spot management information exists and it is determined that the autonomously driven vehicle 2 is operating autonomously, the second integrated result of the infrastructure sensors is transmitted to the autonomously driven vehicle 2. With this configuration, it is possible to provide appropriate assistance to the autonomously driven vehicle 2 depending on whether the autonomously driven vehicle 2 is being remotely operated or operating autonomously.

[0081] Furthermore, in the first embodiment, when a monitor requests to change the display of the infrastructure sensor camera video on the monitor HMI 4, the changed camera video is preferentially displayed on the monitor HMI 4. With this configuration, the changed camera video is preferentially displayed from the next time onwards, which reduces the burden on the monitor in selecting the camera video.

[0082] In the first embodiment, a blind spot area including a static blind spot area and a dynamic blind spot area is determined based on the first integration result and the static blind spot area. According to this configuration, the static blind spot area and the dynamic blind spot area can be easily distinguished and determined, so that the load of the blind spot area determination process can be reduced.

[0083] <Variation 1> In the first embodiment, when the assistance determination unit 104 determines that a blind spot area 2b exists with respect to the position of the autonomous vehicle 2, determines that infrastructure assistance management information associated with the blind spot management information exists, and determines that the autonomous vehicle 2 is being remotely operated, it determines that the camera video acquired by the supportable infrastructure sensor should be displayed to the monitor from the monitor HMI 4. However, the assistance determination unit 104 may determine that the information acquired by the supportable infrastructure sensor should be displayed to the monitor from the monitor HMI 4, rather than determining that the camera video acquired by the supportable infrastructure sensor should be displayed to the monitor from the monitor HMI 4. The information acquired by the supportable infrastructure sensor includes, for example, at least one of the camera video, the position of the autonomous vehicle 2, and the speed of the autonomous vehicle 2.

[0084] When the assistance determination unit 104 determines that information acquired by an infrastructure sensor capable of assistance should be displayed to the monitor from the monitor HMI 4, the video transfer unit 106 transmits the information to the monitor HMI 4 using the external interface 12. When the transmitted information includes the position and speed of the autonomously driving vehicle 2, the monitor HMI 4 may display the position and speed of the autonomously driving vehicle 2 to the monitor from a bird's-eye view, may display them using AR (Augmented Reality), or may display them on a map.

[0085] Furthermore, when an observer makes a request to change the information displayed on the observer HMI 4, the management information update unit 107 may give priority to displaying the changed information on the observer HMI 4.

[0086] <Variation 2> In the first embodiment, the remote monitoring device 1 and the monitor HMI 4 are provided separately as shown in Fig. 1, but this is not limited to the above. For example, the remote monitoring device 1 may be provided with a monitor HMI 4. In addition, in the first embodiment, the map used by the remote monitoring device 1 and the autonomously driving vehicle 2 may be a map with medium accuracy, a map with high accuracy, or a map with low accuracy.

[0087] Also, in the first embodiment, for convenience, the blind spot areas of the perimeter recognition sensor and the camera of the vehicle sensor are the same. However, the blind spot areas of the perimeter recognition sensor and the camera may be different, in which case the determination in step S23 may be performed separately for the sensor and the camera. Similarly, in the first embodiment, for convenience, the detectable areas of the perimeter monitoring sensor and the camera of the infrastructure sensor are the same, but they may be different, in which case the determination in step S23 may be performed separately for the sensor and the camera.

[0088] Moreover, the above modifications can be similarly applied to the second and subsequent embodiments.

[0089] <Embodiment 2> Fig. 16 is a block diagram showing the configuration of a remote monitoring system including a remote monitoring device 1 according to the second embodiment, and is a block diagram corresponding to Fig. 2. Hereinafter, among the components according to the second embodiment, components that are the same as or similar to the components described above are given the same or similar reference numerals, and different components will be mainly described.

[0090] The configuration in Fig. 16 is the same as the configuration in Fig. 2 except that a pattern classification unit 109 is added. In Fig. 16, in order to simplify the drawing, the road-vehicle cooperation support unit 105 and the video transfer unit 106 are omitted.

[0091] The pattern classification unit 109 generates a pattern capable of classifying the state of the detectable area 3a of the infrastructure sensor by modeling the second integration result of the infrastructure sensor. The monitor checks the second integration result used to generate the pattern, and requests the monitor HMI 4 to register the pattern and the state indicating the pattern. When the monitor HMI 4 receives a registration request from the monitor, the pattern classification unit 109 registers the pattern and the state in the storage device 11. This registration may be performed by the management information update unit 107 instead of the pattern classification unit 109.

[0092] 17 and 18 are diagrams for explaining examples of patterns generated by the pattern classification unit 109. As shown in FIG. 17, the patterns are largely classified into normal states and abnormal states. As normal states, patterns of a state where there is nothing, a state where the autonomously driven vehicle 2 is stopped, a state where passengers are getting on and off the autonomously driven vehicle 2, and a state where passengers are waiting at the stop of the autonomously driven vehicle 2 are registered. As abnormal states, patterns of a state where there is a fallen object, and a state where a passenger is abnormal at the stop of the autonomously driven vehicle 2 (for example, a state where a passenger has collapsed) are registered. As shown in FIG. 18, a pattern identifier, a state, a normal or abnormal state, and information on whether to request assistance from a monitor are set for each pattern.

[0093] The support determination unit 104 determines whether the observer HMI 4 should request support from an observer, based on the pattern newly generated by the pattern classification unit 109 and the patterns preregistered in the storage device 11. For example, when the newly generated pattern is classified as a pattern preregistered as an abnormal state, or when the newly generated pattern is not classified as a pattern preregistered regardless of whether it is a normal state or an abnormal state, the support determination unit 104 determines that a request should be made to an observer.

[0094] Furthermore, the assistance determination unit 104 determines whether or not the autonomously driving vehicle 2 should continue autonomous driving, based on the pattern newly generated by the pattern classification unit 109 and the patterns registered in advance in the storage device 11. For example, if the newly generated pattern is classified as a pattern registered in advance as a normal state, the assistance determination unit 104 determines that the autonomously driving vehicle 2 should continue autonomous driving.

[0095] <Operation> FIG. 19 is a flowchart showing an example of a pattern registration operation among the operational examples of the remote monitoring device 1 according to the second embodiment.

[0096] In step S51, the RSU 3 acquires second sensing information from the infrastructure sensor. The acquisition may be performed once, but it is preferable to acquire the second sensing information multiple times.

[0097] In step S52, the information integration unit 101 integrates the second sensing information to generate a second integration result.

[0098] In step S53, the pattern classification unit 109 extracts and models feature points of the second integration result, and generates a pattern that can classify the state of the detectable area 3a of the infrastructure sensor.

[0099] In step S54, the monitor visually checks the state of the detectable area 3a of the infrastructure sensor displayed on the monitor HMI 4, and if it is determined that the state should be registered, the monitor requests the monitor HMI 4 to register the state.

[0100] In step S55, when the observer HMI 4 receives a registration request from the observer, the pattern classification unit 109 associates the state requested by the observer with the generated pattern and registers them in the storage device 11. Thereafter, the operation in FIG. 19 ends.

[0101] Although the above describes an example in which an observer requests the registration of a state, the pattern classification unit 109 may automatically register the state by machine learning (training).

[0102] 20 is a flowchart showing an example of a support determination operation among the operational examples of the remote monitoring device 1 according to the embodiment 2. This operation is performed sequentially (for example, periodically) after the process of FIG.

[0103] In step S61, the RSU 3 acquires the second sensing information, and the information integrator 101 integrates the second sensing information to generate a second integration result. Then, the support determination unit 104 acquires the second integration result from the information integrator 101.

[0104] In step S62, the pattern classification unit 109 extracts and models feature points of the second integration result to generate a pattern capable of classifying the state of the detectable area 3a of the infrastructure sensor. Then, the assistance determination unit 104 determines whether the generated pattern is classified into a pattern registered in advance as a normal state. If it is determined that the generated pattern is classified into a pattern registered in advance as a normal state, the process proceeds to step S63. If it is not determined that the generated pattern is classified into a pattern registered in advance as a normal state, the process proceeds to step S65.

[0105] In step S63, the assistance determination unit 104 determines that the autonomously driven vehicle 2 should continue autonomous driving, and issues an action instruction to the autonomously driven vehicle 2, such as enabling it to start.

[0106] In step S64, the autonomously driving vehicle 2 continues autonomous driving based on instructions from the assistance determination unit 104. Thereafter, the operation in Fig. 20 ends.

[0107] In step S65, the assistance determination unit 104 determines that the observer HMI 4 should request assistance from the observer, and the observer HMI 4 notifies the observer of the abnormal state and the assistance request.

[0108] In step S66, the monitor checks the camera image displayed on the monitor HMI 4 and checks the second integrated result as appropriate. If the monitor determines that there is no problem with the state detected by the infrastructure sensor, the monitor requests the pattern classification unit 109 to register the pattern of that state as a normal state. If the monitor determines that there is a problem with the state detected by the infrastructure sensor, the monitor remotely operates the autonomous vehicle 2 and requests the pattern classification unit 109 to register the pattern of that state as an abnormal state.

[0109] In step S67, the pattern classification unit 109 associates the state requested by the observer with the generated pattern and additionally registers them in the storage device 11. Thereafter, the operation of FIG. 20 ends.

[0110] Although the above describes an example in which an observer requests the registration of a state, the pattern classification unit 109 may automatically register the state by machine learning (training).

[0111] <Summary of the second embodiment> According to the remote monitoring device 1 of the second embodiment as described above, a pattern is generated by modeling the second integration result, and the monitor HMI 4 judges whether or not to request assistance from the monitor based on the generated pattern and a preregistered pattern. With this configuration, the burden on the monitor in confirming whether remote operation is required can be reduced. Note that if the above judgment is made in advance before the autonomous vehicle 2 starts traveling, appropriate assistance can be provided quickly, and the processing load of the remote monitoring device 1 during the traveling of the autonomous vehicle 2 can be reduced.

[0112] In the second embodiment, a pattern is generated by modeling the second integration result, and based on the generated pattern and a pre-registered pattern, it is determined whether or not the autonomous vehicle 2 should continue autonomous driving. With this configuration, the autonomous vehicle 2 can appropriately continue autonomous driving.

[0113] In the second embodiment, if a generated pattern is not classified into a pre-registered pattern, the pattern is added to the pre-registered patterns at the request of the monitor. With this configuration, the remote monitoring device 1 can be customized to suit the monitor.

[0114] <Modification> In embodiment 2, patterns at bus stops for autonomous vehicle 2 are described, but this is not limited to this, and patterns at other locations such as normal roads and intersections other than bus stops may also be generated and managed.

[0115] It should be noted that the embodiments and modifications may be freely combined, and the embodiments and modifications may be modified or omitted as appropriate.

[0116] The above description is illustrative in all respects and is not restrictive. It is understood that countless variations not illustrated can be envisioned. [Explanation of symbols]

[0117] 1 Remote monitoring device, 2 Autonomous vehicle, 2b Blind spot area, 3 RSU, 3a Detectable area, 4 Monitor HMI, 101 Information integration unit, 102 Blind spot determination unit, 103 Management information generation unit, 104 Assistance determination unit, 105 Road-vehicle cooperation assistance unit, 106 Video transfer unit, 107 Management information update unit, 109 Pattern classification unit.

Claims

1. an information integration unit that integrates first sensing information acquired by a vehicle that can be remotely operated by a monitor to generate a first integration result, and integrates second sensing information acquired by an infrastructure sensor installed on the road to generate a second integration result; a blind spot determination unit that determines a blind spot area including a static blind spot area and a dynamic blind spot area that are managed in advance of the vehicle based on the first integration result, and determines whether the infrastructure sensor detects a state of the static blind spot area based on the first integration result and the second integration result; a support determination unit that determines that the blind spot area exists with respect to the position of the vehicle, determines that the infrastructure sensor detects a state of the static blind spot area, and when it determines that the vehicle is being remotely operated, determines that information acquired by the infrastructure sensor should be displayed to the observer from a human interface; A remote monitoring device comprising:

2. A remote monitoring device as claimed in claim 1, The remote monitoring device further includes a management information generation unit that associates blind spot management information regarding the static blind spot area with infrastructure support management information regarding a static detectable area of ​​the infrastructure sensor that is determined to detect the state of the static blind spot area.

3. 3. The remote monitoring device according to claim 2, The vehicle is capable of autonomous driving. a road-vehicle cooperation support unit that determines that the blind spot area exists for a position of the vehicle, determines that the infrastructure support management information associated with the blind spot management information exists, and transmits the second integrated result of the static detectable area to the vehicle when the support determination unit determines that the vehicle is in autonomous driving; an information transfer unit that transmits the information to the human interface when the assistance determination unit determines that the information should be displayed from the human interface; A remote monitoring device comprising:

4. 2. The remote monitoring device according to claim 1, The remote monitoring device further comprises a management information update unit which, when the monitor requests a change to the display of the information in the human interface, causes the changed information to be preferentially displayed in the human interface.

5. 2. The remote monitoring device according to claim 1, a pattern classifier that generates a pattern capable of classifying a state of a detectable area of ​​the infrastructure sensor by modeling the second integration result, The support determination unit, The remote monitoring device determines whether or not the human interface should request assistance from the monitor based on the pattern and a previously registered pattern.

6. 2. The remote monitoring device according to claim 1, a pattern classifier that generates a pattern capable of classifying a state of a detectable area of ​​the infrastructure sensor by modeling the second integration result, The vehicle is capable of autonomous driving; The support determination unit, A remote monitoring device that determines whether or not the vehicle should continue autonomous driving based on the pattern and a pre-registered pattern.

7. The remote monitoring device according to claim 5 or 6, If the pattern is not classified into the pre-registered patterns, the pattern is added to the pre-registered patterns at the request of the monitor.

8. A remote monitoring device; a human interface capable of communicating with the remote monitoring device; Equipped with The remote monitoring device includes: an information integration unit that integrates first sensing information acquired by a vehicle that can be remotely operated by a monitor to generate a first integration result, and integrates second sensing information acquired by an infrastructure sensor installed on the road to generate a second integration result; a blind spot determination unit that determines a blind spot area including a static blind spot area and a dynamic blind spot area that are managed in advance of the vehicle based on the first integration result, and determines whether the infrastructure sensor detects a state of the static blind spot area based on the first integration result and the second integration result; a support determination unit that determines that the blind spot area exists with respect to the position of the vehicle, determines that the infrastructure sensor detects a state of the static blind spot area, and when it determines that the vehicle is being remotely operated, determines that information acquired by the infrastructure sensor should be displayed to the observer from the human interface; A remote monitoring system comprising:

9. generating a first integrated result by integrating first sensing information acquired by a vehicle that can be remotely operated by a monitor, and generating a second integrated result by integrating second sensing information acquired by an infrastructure sensor attached to the road; Based on the first integration result, a blind spot area including a static blind spot area and a dynamic blind spot area that are managed in advance of the vehicle is determined, and based on the first integration result and the second integration result, it is determined whether the infrastructure sensor detects a state of the static blind spot area; A remote monitoring method comprising: determining that the blind spot area exists for the position of the vehicle; determining that the infrastructure sensor detects the state of the static blind spot area; and, if it is determined that the vehicle is being remotely operated, determining that information obtained by the infrastructure sensor should be displayed to the monitor via a human interface.

Images