Remote Support System and Remote Support Method for Autonomous Vehicles

Abstract

The present invention provides a remote support system and a remote support method for an autonomous vehicle. A technology is provided such that when performing identification support for the indication content of a traffic signal existing in front of the vehicle, the vehicle can travel safely based on real-time information of the indication content. The vehicle determines whether the vehicle passing permission conditions according to the indication content of the traffic signal in front of the vehicle are satisfied. When it is determined that the passing permission conditions are not satisfied, the vehicle temporarily stops in front of the traffic signal and sends a support request signal to a remote facility. In the display device of the remote facility, reference image information representing the captured image information during the temporary stop of the vehicle is displayed. An operator of the remote facility inputs the position information of the pixels constituting the traffic signal included in the reference image information. The remote facility sends support information including the position information to the vehicle. The vehicle re-determines whether the passing permission conditions are satisfied based on the position information.

Description

Technical Field

[0001] The present invention relates to a system and method for supporting the automatic driving of an autonomous vehicle. Background Art

[0002] Japanese Unexamined Patent Application Publication No. 2019-156192 discloses a control device for the automatic driving of a vehicle in a one-way alternating traffic section. A one-way alternating traffic section is provided, for example, when construction work is being carried out on one side of a two-way traffic road, to ensure the passage of vehicles across the construction section. The conventional control device identifies the presence of the construction section and the structures peculiar to the construction section based on the image information obtained by a camera. As the peculiar structures, construction vehicles, traffic cones, and temporary traffic lights are exemplified.

[0003] The temporary traffic light is installed in front of the construction section. The conventional control device determines whether a vehicle can pass in the one-way alternating traffic section based on the information indicated by the temporary traffic light (e.g., display color, time display). In the case where the temporary traffic light is not identified, the control device determines whether the vehicle can pass based on other information. As the other information, the activities of traffic guides and the actions of preceding vehicles are exemplified. In the case where these information are not identified, the conventional control device executes control to urge the driver to perform manual driving.

[0004] Prior Art Documents

[0005] Patent Documents

[0006] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2019-156192

[0007] Patent Document 2: Japanese Unexamined Patent Application Publication No. 2019-156196

[0008] Patent Document 3: Japanese Unexamined Patent Application Publication No. 2019-036100 Summary of the Invention

[0009] As a reason for not identifying the temporary traffic light, it can be cited that it is difficult for the vehicle to grasp the installation position of the temporary traffic light on the map, and thus it is difficult to identify in the camera image compared to a permanent traffic light. This problem of identifiability also exists in traffic lights installed on private property (e.g., the parking lot of a commercial facility).

[0010] Therefore, consider a situation where the recognition accuracy of the indication content of a traffic signal existing in front of a vehicle is low and the operator of a remote facility is required to support the recognition of the indication content. In this situation, while the vehicle temporarily stops in front of the traffic signal, a signal requesting recognition support is sent to the remote facility. When this signal is sent, the image information of the in-vehicle camera is provided to the operator. Therefore, it is easy for the operator to recognize the indication content of the traffic signal. Then, according to the indication content of the traffic signal, the operator inputs a passage permission signal to a computer. This passage permission signal is sent from the remote facility to the vehicle.

[0011] However, due to the transmission speed of the communication line and the distance from the remote facility to the vehicle, communication delay occurs. Therefore, there is a possibility that at the timing when the vehicle receives the passage permission signal, or at the timing when the vehicle starts according to this passage permission signal, the indication content of the traffic signal becomes an indication content that does not permit the vehicle to pass. Therefore, an improvement for appropriately coping with such a change in the indication content is required.

[0012] One object of the present invention is to provide a technology that enables a vehicle to travel safely according to real-time information of the indication content of a traffic signal existing in front of the vehicle when supporting the recognition of the indication content.

[0013] The first aspect is a remote support system for a vehicle, having the following features.

[0014] The remote support system includes a vehicle that performs autonomous driving and a remote facility that communicates with the vehicle.

[0015] The vehicle is equipped with a control device that performs autonomous driving control of the vehicle based on driving environment information including captured image information in front of the vehicle.

[0016] The remote facility includes an input device, a display device, and an information processing device. The input device is operated by an operator. The display device displays the captured image information. The information processing device performs input information processing for processing input information received by the input device and display control processing for the display device.

[0017] In the autonomous driving control, the control device

[0018] determines whether the passage permission condition of the vehicle according to the indication content of the traffic signal in front of the vehicle is satisfied based on the driving environment information,

[0019] and when it is determined that the passage permission condition is not satisfied, causes the vehicle to temporarily stop in front of the traffic signal.

[0020] Send a support request signal requesting support from the operator to the remote facility.

[0021] The information processing device

[0022] In the case of receiving the support request signal, in the display control process, perform a process of displaying reference image information representing the captured image information during the temporary stop of the vehicle.

[0023] In the case of accepting the position information of the pixels constituting the traffic signal included in the reference image information as the input information, in the input information process, send support information including the position information to the vehicle.

[0024] The control device further in the automatic driving control

[0025] In the case of receiving the support information, based on the position information, re-determine whether the traffic permission condition is satisfied.

[0026] In the case of determining that the traffic permission condition is satisfied, release the temporary stop state of the vehicle and perform vehicle control for driving according to the indication content of the traffic signal.

[0027] The second aspect further has the following features in the first aspect.

[0028] In the case of the information processing device receiving the traffic permission signal of the vehicle as the input information, in the input information process, send the support information including the traffic permission signal to the vehicle.

[0029] The control device further in the automatic driving control

[0030] In the case of determining that the traffic permission condition is satisfied after receiving the traffic permission signal, release the temporary stop state and perform the vehicle control.

[0031] The third aspect is a remote support method for a vehicle that remotely supports the automatic driving of the vehicle, and has the following features.

[0032] In the remote support method,

[0033] The control device of the vehicle performs automatic driving control of the vehicle according to driving environment information including captured image information in front of the vehicle.

[0034] An information processing device of a remote facility that communicates with the vehicle performs: input information processing to process input information received by an input device operated by an operator; and display control processing of a display device that displays the captured image information.

[0035] In the automatic driving control, the control device

[0036] determines, based on the driving environment information, whether the vehicle's passage permission conditions in accordance with the indication content of the traffic signal in front of the vehicle are satisfied.

[0037] When it is determined that the passage permission conditions are not satisfied, the vehicle is temporarily stopped in front of the traffic signal.

[0038] A support request signal requesting support to be performed by the operator is sent to the remote facility.

[0039] The information processing device

[0040] When receiving the support request signal, in the display control processing, it performs processing for displaying reference image information representing the captured image information during the temporary stop of the vehicle.

[0041] When receiving the position information of the pixels of the traffic signal included in the reference image information as the input information, in the input information processing, support information including the position information is sent to the vehicle.

[0042] In the automatic driving control, the control device further

[0043] When receiving the support information, based on the position information, it determines again whether the passage permission conditions are satisfied.

[0044] When it is determined that the passage permission conditions are satisfied, the temporary stop state of the vehicle is released, and vehicle control for traveling in accordance with the indication content of the traffic signal is performed.

[0045] The fourth aspect has the following further features in the third aspect.

[0046] When the information processing device receives the vehicle's passage permission signal as the input information, in the input information processing, support information including the passage permission signal is sent to the vehicle.

[0047] In the automatic driving control, the control device further

[0048] When it is determined that the passing permission conditions are met after the reception of the passing permission signal, the temporary stop state is released and the vehicle control is performed.

[0049] According to the first or third aspect, even when the recognition accuracy of the indication content of the traffic signal is low, a support request signal is still responded to, and support information is provided to the vehicle from a remote facility. The support information includes the position information of the pixels of the traffic signal included in the captured image information (reference image information) during the temporary stop of the vehicle. Therefore, the situation where the recognition accuracy of the indication content of the traffic signal is low can be improved. Therefore, real-time information on the indication content of the traffic signal can be recognized in the vehicle, and the determination of the passing permission conditions can be performed. Therefore, the driving safety of the vehicle according to the indication content of the traffic signal can be improved.

[0050] According to the second or fourth aspect, even when support information including a passing permission signal is provided to the vehicle, real-time information on the indication content of the traffic signal can be recognized in the vehicle, and thus the determination of the passing permission conditions can be performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a diagram showing an outline of an embodiment of the present invention.

[0052] Figure 2 is a diagram showing an outline of an embodiment of the present invention.

[0053] Figure 3 is a diagram showing an outline of an embodiment of the present invention.

[0054] Figure 4 is a diagram showing a structural example of a remote support system according to an embodiment of the present invention.

[0055] Figure 5 is a flowchart showing the flow of an autonomous driving control process performed in an autonomous driving vehicle when an impassable section is recognized.

[0056] Figure 6 is a flowchart showing the flow of another autonomous driving control process performed in an autonomous driving vehicle when an impassable section is recognized.

[0057] Figure 7 is a flowchart showing the flow of information processing performed in a remote facility.

[0058] (Reference Signs)

[0059] 1: Remote support system; 2: Network; 3: Autonomous vehicle; 4: Remote facility; 37: Communication device; 38: Control device; 41: Communication device; 42: Information processing device; 43: Display device; 44: Input device; DR: Focus area; L1, L2: Traffic lanes; TS: Traffic signal; ASS: Support information; IPS: Non-drivable section; OAS: One-side alternate section; PPS: Position information; REF: Reference image information; REQ: Support request signal; STA: Passage permission signal. Detailed implementation manners

[0060] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In addition, the remote support method related to the embodiments is implemented by computer processing performed in the remote support system related to the embodiments. In addition, in each figure, the same or corresponding parts are assigned the same reference numerals to simplify or omit the description thereof.

[0061] 1. Summary of the embodiment

[0062] Figures 1 to 3 This is a diagram for explaining the summary of the embodiment. In Figure 1 and 2, an autonomous vehicle 3 traveling on the traffic lane L1 is depicted. The autonomous vehicle 3 forms a part of the remote support system for vehicles related to the embodiment. In front of the autonomous vehicle 3, there is a construction machine CM. The area around the construction machine CM constitutes a non-drivable section IPS (impassable section) for vehicles. That is, in Figure 1 and the example of 2, an IPS as a construction section is set. However, the example of the IPS is not limited to this. For example, an IPS can also be set in the area around a stopped vehicle (such as a malfunctioning vehicle).

[0063] The traffic lane L2 adjacent to the traffic lane L1 is a road for oncoming vehicles to travel. There is no structure such as a median strip between the traffic lane L1 and the traffic lane L2. Therefore, the traffic lanes L1 and L2 constitute an oncoming traffic road with one traffic lane on each side. When an IPS is set in the oncoming traffic road, the area around the IPS sometimes constitutes an one-side alternate section OAS (one-side alternate section) for vehicles to alternate. The OAS includes the areas in front of and behind the IPS in the traffic lane L1, the area of the traffic lane L2 adjacent to the IPS, and the areas in front of and behind the adjacent area.

[0064] In front of the section IPS in the traffic lane L1, a signal lamp TS is provided. The signal lamp TS is a temporarily installed signal lamp for a vehicle (i.e., the autonomous vehicle 3) traveling on the traffic lane L1 to pass through the section OAS. The signal lamp TS has a lighting part (for example, a green and a red lighting part). The lighting state of the lighting part is switched at a predetermined interval. The signal lamp TS may also have a display part that displays the time until the lighting state is switched.

[0065] In Figure 1 this case, the autonomous vehicle 3 decelerates in front of the section OAS. As an example of the autonomous vehicle 3 decelerating, it can be cited that the indication content of the lighting part corresponds to "stop" (specifically, the red lighting part is lit). As another example, it can be cited that the accuracy of the autonomous vehicle 3 recognizing the signal lamp TS, the lighting part, or the lighting state is low (including the case where the signal lamp TS cannot be recognized, etc.). In the case where the recognition accuracy of the signal lamp TS, etc. is low, the autonomous vehicle 3 temporarily stops in front of the section OAS.

[0066] In Figure 2 this case, the autonomous vehicle 3 that temporarily stops in front of the section OAS is depicted. In the case where the recognition accuracy of the signal lamp TS, etc. is low, after the completion of the stop operation (or during the implementation of the deceleration operation) of the autonomous vehicle 3, a signal (support request signal) REQ that requests support for passing through the section OAS is sent to the remote facility 4. The operator stationed at the remote facility 4 responds to the support request signal REQ and implements the support. When the support request signal REQ is sent, the captured image information in front of the autonomous vehicle 3 when the autonomous vehicle 3 is stopped is sent as "reference image information REF".

[0067] Figure 3 is a diagram schematically showing a part of the reference image information REF. In the remote facility 4, the Figure 3 shown reference image information REF is displayed in a visually recognizable manner by the operator. Therefore, the operator who sees the reference image information REF recognizes the signal lamp TS. The operator who recognizes the signal lamp TS sends support information ASS including the position information PPS of the pixels constituting the signal lamp TS to the autonomous vehicle 3. The position information PPS of the pixels is information for improving the recognition accuracy of the signal lamp TS, etc. by the autonomous vehicle 3.

[0068] The reference image information REF is composed of a plurality of pixels. In Figure 3In the example shown, the attention area DR is set in such a way as to surround the signal lamp TS. The attention area DR is an area designated by an operator who views the reference image information REF. The attention area DR is a rectangular area with coordinates (i - 1, j - 1), (i + 2, j - 1), (i - 1, j + 2), and (i + 2, j + 2) as vertices, and the coordinate information constituting the attention area DR corresponds to the position information PPS.

[0069] The autonomous driving vehicle 3 that has received the support information ASS including the position information PPS determines whether the passing permission conditions for automatically passing through the section OAS according to the indication content of the signal lamp TS are satisfied based on the position information PPS and the captured image information. The passing permission conditions include the following conditions, for example.

[0070] (i) Identify the signal lamp TS

[0071] (ii) The indication content of the lighting part corresponds to "passing permission" (specifically, the green lighting part is lit)

[0072] When it is determined that the passing permission conditions are satisfied, the autonomous driving vehicle 3 releases the temporary stop state and starts to automatically pass through the section OAS.

[0073] The operator who has identified the signal lamp TS can also send a passing permission signal STA to the autonomous driving vehicle 3 when the indication content of the lighting part corresponds to "passing permission". In this case, the passing permission signal STA is sent to the autonomous driving vehicle 3 as additional information of the support information ASS. When the passing permission signal STA is received, the autonomous driving vehicle 3 determines whether the passing permission conditions are satisfied after receiving the passing permission signal STA. Then, when it is determined that the passing permission conditions are satisfied, the autonomous driving vehicle 3 releases the temporary stop state and starts to automatically pass through the section OAS. That is, when it is determined that the passing permission conditions are satisfied after receiving the passing permission signal STA, the autonomous driving vehicle 3 automatically passes through the section OAS.

[0074] In this way, according to the embodiment, when the accuracy of identifying the signal lamp TS or the like is low, the operator supports the passing in the section OAS. In this support, the operator who views the reference image information REF provides the support information ASS including the position information PPS to the autonomous driving vehicle 3. According to the position information PPS, the accuracy of identifying the signal lamp TS or the like by the autonomous driving vehicle 3 is improved. Therefore, it is possible to determine whether the passing permission conditions are satisfied in the autonomous driving vehicle 3.

[0075] When only the traffic permission signal STA is provided to the autonomous vehicle 3, the time lag that occurs in the transmission and reception of the traffic permission signal STA becomes a problem. In particular, when communication delay occurs, the time lag expands. Therefore, at the timing of starting the autonomous vehicle 3, there is a possibility that the indication content of the lighting unit changes from "traffic permission" to "stop". Regarding this point, based on the determination of the traffic permission conditions according to the position information PPS, the real-time information indicated by the traffic signal TS can be considered. Therefore, the safety when automatically passing through the section OAS according to the indication content of the traffic signal TS can be improved.

[0076] In addition, in Figures 1 to 3 an example of setting the traffic signal TS in the section OAS is described. However, in the traffic signal TS of the embodiment, it also includes traffic signals installed on private property (hereinafter also referred to as "privately installed traffic signals"). Privately installed traffic signals are security devices installed in specific areas of private property (such as the intersection or merging point of multiple roads). As typical installation locations of privately installed traffic signals, the sides or above of specific areas are envisioned. By replacing the "section OAS" in the description of Figures 1 to 3 with "specific area", the outline of the embodiment applied to privately installed traffic signals is described.

[0077] Hereinafter, the vehicle remote support system according to the embodiment will be described in detail.

[0078] 2. Structural example of the remote support system

[0079] 2-1. Overall structural example

[0080] The vehicle remote support system according to the present embodiment is a system that connects the autonomous vehicle 3 and the remote facility 4 with a network. Figure 4 It is a diagram showing a structural example of the remote support system 1 according to the present embodiment.

[0081] The remote support system 1 includes the autonomous vehicle 3 and a remote facility 4 that communicates with the autonomous vehicle 3 via a network (i.e., the Internet) 2. The driving status of the autonomous vehicle 3 is monitored by the remote facility 4. The remote facility 4 can either exclusively monitor one autonomous vehicle 3 or simultaneously monitor multiple autonomous vehicles 3.

[0082] In the remote facility 4, the driving status of the autonomous vehicle 3 is monitored by an operator. To enable the monitoring by the operator, the remote facility 4 includes a communication device (facility-side communication device) 41, an information processing device 42, a display device 43, and an input device 44.

[0083] The communication device 41 is a device for receiving the captured image information from the autonomous driving vehicle 3. The information processing device 42 is composed of a microcomputer having at least one processor and at least one memory. In the information processing device 42, display control processing is performed as processing for processing the captured image information and displaying it on the display device 43. The input device 44 is an HMI (Human Machine Interface) for an operator to input an instruction to the information processing device 42. When an instruction is input from the operator, the information processing device 42 performs input information processing for processing the input information. The communication device 41 sends a part or all of the information after the input information processing performed by the information processing device 42 to the autonomous driving vehicle 3.

[0084] In the autonomous driving vehicle 3, a camera 31, a millimeter-wave radar 32, and a lidar 33 are provided as devices for obtaining driving environment information for its autonomous driving. The camera 31 is a device for obtaining the captured image information around the autonomous driving vehicle 3 and is an essential structure of the autonomous driving vehicle 3. The millimeter-wave radar 32 and the lidar 33 are devices for obtaining target information around the autonomous driving vehicle 3. The millimeter-wave radar 32 and the lidar 33 can also be omitted. The camera 31 is installed, for example, on the windshield of the autonomous driving vehicle 3. The information obtained by the camera 31, the millimeter-wave radar 32, and the lidar 33 is sent to the control device 38.

[0085] The control device 38 takes in the driving environment information including the information from the camera 31, the millimeter-wave radar 32, and the lidar 33, and the support information from the remote facility 4. Then, the control device 38 operates various actuators with the operation signals obtained by processing these information. Among the various actuators, there are a drive actuator 34, a brake actuator 35, and a steering actuator 36. The drive actuator 34 drives the autonomous driving vehicle 3. The brake actuator 35 applies a braking force to the autonomous driving vehicle 3. The steering actuator 36 steers the autonomous driving vehicle 3.

[0086] In the driving environment information, it includes the information related to the vehicle state obtained by in-vehicle sensors such as a vehicle speed sensor and an acceleration sensor (not shown). In the driving environment information, it also includes the position information indicating the current location of the autonomous driving vehicle 3 obtained by a GPS receiver (not shown) and the map information in the map database. The support information from the remote facility 4 is obtained through wireless communication using the communication device (vehicle-side communication device) 37. As the communication standard of the wireless communication used by the communication device 37, the standards of mobile communications such as 4G, LTE, and 5G are exemplified.

[0087] The control device 38 is composed of a microcomputer having at least one processor 38a and at least one memory 38b. In the memory 38b, at least one program for autonomous driving is stored. Various information including driving environment information is stored in the memory 38b. By reading out the program stored in the memory 38b and executing it by the processor 38a, various functions of the control device 38 are realized.

[0088] 2-2. Examples of functions of the control device

[0089] 2-2-1. Autonomous driving control function

[0090] The control device 38 (processor 38a) calculates the driving route of the autonomous driving vehicle 3 based on, for example, the position information of the autonomous driving vehicle 3 and the map information. Then, the control device 38 performs autonomous driving control to control the driving, steering, and braking of the autonomous driving vehicle 3 so that it travels along the calculated driving route. When executing the autonomous driving control, the captured image information from the camera 31, the target information from the millimeter-wave radar 32 and the lidar 33, and the vehicle state information from the in-vehicle sensors are appropriately used. In addition, there are various known methods regarding the method of autonomous driving, and the method itself is not limited in the present invention. Therefore, in this specification, the description of the autonomous driving method along the driving route is omitted.

[0091] In the autonomous driving control, collision avoidance control is included. The collision avoidance control is control for avoiding the autonomous driving vehicle 3 from colliding with surrounding targets (avoidance targets). As the avoidance targets, for example, Figure 1 the construction machinery CM shown is exemplified. In the identification of the target (avoidance target), image processing technology using artificial intelligence is utilized. When a safety appliance such as a triangular cone is identified from the captured image information (or the combined information of the captured image information and the target information), the control device 38 identifies the area divided by the safety appliance as the interval IPS. When the interval IPS is identified, the control device 38 executes collision avoidance control with the interval IPS as the avoidance target.

[0092] When executing the collision avoidance control with the interval IPS as the avoidance target, the control device 38 determines whether the area around the interval IPS corresponds to the interval OAS based on the captured image information, the position information, and the map information. For example, based on the position information and the map information, it is known that the autonomous driving vehicle 3 is traveling on a two-way traffic road with one lane on each side. In addition, based on the captured image information, the ratio in the lane width direction occupied by the interval IPS is known. Therefore, it is determined whether the occupancy ratio of the interval IPS is equal to or greater than the threshold value. Thereby, it is possible to determine whether the area around the interval IPS corresponds to the interval OAS.

[0093] When the traffic signal TS is identified based on the camera image information or the comprehensive information, the determination based on the occupancy ratio of the section IPS can also be omitted. The traffic signal TS is a typical security device provided in the section OAS. Therefore, when the traffic signal TS is identified, it is possible to easily determine that the area around the section IPS corresponds to the section OAS.

[0094] When it is determined that there is a section OAS in front of the autonomous driving vehicle 3, the control device 38 determines whether the passing permission conditions for automatically passing through the section OAS are satisfied based on the camera image information. The passing permission conditions include, for example, the following conditions.

[0095] (i) Identify the traffic signal TS

[0096] (ii) The indication content of the lighting unit corresponds to "passing permission"

[0097] When it is determined that the passing permission conditions are satisfied, the autonomous driving vehicle 3 automatically passes through the section OAS. When automatically passing through the section OAS, for example, a driving route (target trajectory) from the current position of the autonomous driving vehicle 3 to an arbitrary position within the section OAS is set. Then, the control device 38 controls the driving, steering, and braking of the autonomous driving vehicle 3 so that the autonomous driving vehicle 3 travels along the set target trajectory. After passing through the section IPS, the control device 38 sets a target trajectory from the current position of the autonomous driving vehicle 3 to an arbitrary position within the traffic lane L1. Then, the control device 38 controls the driving, steering, and braking of the autonomous driving vehicle 3 so that the autonomous driving vehicle 3 travels along the set target trajectory.

[0098] When it is determined that the passing permission conditions are not satisfied, the control device 38 (processor 38a) controls the driving, steering, and braking of the autonomous driving vehicle 3 so that the autonomous driving vehicle 3 stops in front of the section OAS. Then, the control device 38 sends a support request signal REQ to the remote facility 4. The sending of the support request signal REQ is performed after the completion of the stop operation of the autonomous driving vehicle 3. The sending of the support request signal REQ may also be performed before the completion of the stop operation. That is, the support request signal REQ may be sent during the deceleration operation of the autonomous driving vehicle 3.

[0099] 2-2-2. Camera image transmission function

[0100] The control device 38 (processor 38a) sends the captured image information from the camera 31 to the remote facility 4. Among the sent captured image information, at least the captured image information in front of the autonomous vehicle 3 is included. The captured image information sent during the temporary stop in front of the traffic signal TS is the reference image information REF. The reference image information REF is sent to the remote facility 4 and stored in a predetermined area of the memory 38b (or the cache memory of the processor 38a).

[0101] The control device 38 sends the captured image information in accordance with the communication cycle between the remote facility 4 and the communication device 37. The captured image information sent to the remote facility 4 is used for remote monitoring around the autonomous vehicle 3. The communication cycle can be fixed or can vary according to the driving environment information. For example, the communication cycle during driving on an exclusive car lane can be extended, and the communication cycle during driving at an intersection where a traffic signal exists can be shortened. The communication cycle during the temporary stop in front of the traffic signal TS can be set, for example, to the middle of these communication cycles.

[0102] 3. Example of processing in the case of setting the section IPS on a two-way traffic road

[0103] 3-1. Processing on the side of the autonomous vehicle (control device)

[0104] Figure 5 is a flowchart showing the flow of the autonomous driving control process performed by the control device 38 when the section IPS is recognized. Whenever the section IPS is recognized, the Figure 5 shown routine is repeatedly executed.

[0105] In Figure 5 the shown routine, first, it is determined whether there is a section OAS in front of the autonomous vehicle 3 (step S11). Based on the captured image information, position information, and map information, it is determined whether there is a section OAS. For example, the control device 38 determines whether the autonomous vehicle 3 is driving on a two-way traffic road with a single lane on one side based on the position information and map information. In addition, the control device 38 calculates the ratio in the lane width direction occupied by the section IPS based on the captured image information. Then, the control device 38 determines whether the occupancy ratio is above a threshold value. If the occupancy ratio is above the threshold value, it is determined that there is a section OAS.

[0106] When the determination result in step S11 is negative, the processing of this routine ends. In addition, in this case, the control device 38 performs collision avoidance control that designates the section IPS as an avoidance object according to a control program different from this routine. When the determination result in step S11 is positive, it is determined whether the passage permission condition is satisfied based on the captured image information (step S12). Examples of the passage permission condition are as described above.

[0107] When the determination result in step S12 is positive, vehicle control for automatically passing through the section OAS is performed (step S13). For example, the control device 38 sets a target trajectory of the autonomous driving vehicle 3 from the current position to an arbitrary position within the section OAS. Then, the control device 38 controls the driving, steering, and braking of the autonomous driving vehicle 3 so that the autonomous driving vehicle 3 travels along the target trajectory. After passing through the section IPS, the control device 38 sets a target trajectory of the autonomous driving vehicle 3 from the current position to an arbitrary position within the traffic lane L1. Then, the control device 38 controls the driving, steering, and braking of the autonomous driving vehicle 3 so that the autonomous driving vehicle 3 travels along the set target trajectory.

[0108] When the determination result in step S12 is negative, a temporary stop is performed at the front of the section OAS (step S14). For example, the control device 38 controls the driving, steering, and braking of the autonomous driving vehicle 3 so that the autonomous driving vehicle 3 stops at the front of the section OAS.

[0109] Following the processing of step S14, a support request signal REQ is output (step S15). The support request signal REQ is sent from the control device 38 to the communication device 37. Thereafter, the support request signal REQ is sent to the communication device 41 via the network 2.

[0110] Following the processing of step S15, it is determined whether support information ASS including position information PPS is received (step S16). As described above, the position information PPS is the position information of the pixels constituting the traffic signal TS included in the reference image information REF (see Figure 3 ). The processing of step S16 is repeatedly executed until an affirmative determination result is obtained.

[0111] When the determination result in step S16 is positive, it is determined whether the passage permission condition is satisfied (step S17). That is, when the position information PPS is received, the control device 38 determines again whether the passage permission condition is satisfied. The content of the processing in step S17 is basically the same as that in step S12. However, in the processing of step S12, the determination is made based on the captured image information, while in the processing of step S17, the determination is made based on the position information PPS and the captured image information.

[0112] As described above, the position information PPS is the position information of the pixels constituting the traffic signal TS. That is, the position information PPS is information indicating in which pixel coordinates of the reference image information REF the traffic signal TS exists. Here, since the autonomous driving vehicle 3 temporarily stops, the pixel coordinates of the traffic signal TS in the captured image information obtained during this period are the same as those in the reference image information REF. Therefore, based on the position information PPS, the accuracy of identifying the traffic signal TS by the control device 38 can be improved. Thus, it is easy to satisfy the first condition (i) of the passage permission condition.

[0113] When both the first condition (i) and the second condition (ii) of the passage permission condition are satisfied, it is determined that the passage permission condition is satisfied. When the determination result in step S17 is affirmative, the process of step S13 is performed. That is, the control device 38 performs vehicle control for starting from the stop position in front of the section OAS and automatically passing through the section OAS.

[0114] Figure 6 It is a flowchart showing the flow of other autonomous driving control processes performed by the control device 38 when the section IPS is identified. Figure 6 The shown routine is the same as Figure 5 The shown routine and is repeatedly executed every time the section IPS is identified.

[0115] Figure 6 The processes of steps S21 to S25 shown are the same as the processes of steps S11 to S15 described in Figure 5 Therefore, hereinafter, the processes after step S26 will be described.

[0116] Following the process of step S25, it is determined whether support information ASS including the position information PPS and the passage permission signal STA is received (step S26). Regarding the determination of receiving the position information PPS, it is as described in Figure 5 Step S16. The passage permission signal STA is a signal permitting the start of the autonomous driving vehicle 3. The process of step S26 is repeatedly executed until an affirmative determination result is obtained.

[0117] When the determination result in step S26 is affirmative, it is determined whether the passage permission condition is satisfied (step S27). The content of the process of step S27 is basically the same as Figure 5 The process of step S17. However, in the process of step S27, after receiving the passage permission signal STA, the determination is made based on the position information PPS and the captured image information.

[0118] When the determination result in step S27 is affirmative, the process of step S23 is performed. That is, the control device 38 performs vehicle control for starting from the stop position in front of the section OAS and automatically passing through the section OAS.

[0119] 3-2. Processing on the side of the remote facility (information processing device)

[0120] Figure 7 It is a flowchart showing the flow of input information processing performed by the information processing device 42 when an operator responds to the support request signal REQ and performs remote support. Whenever the information processing device 42 receives the support request signal REQ, the Figure 7 shown routine is repeatedly executed.

[0121] In Figure 7 the shown routine, first, it is determined whether there is an operation input performed by the operator (step S31). The operator performing remote support identifies the signal lamp TS based on the reference image information REF displayed on the display device 43. In addition, according to the display control process, the reference image information REF is displayed. The operator who has identified the signal lamp TS operates the input device 44 to specify the area of interest DR so as to surround the signal lamp TS. The process of step S31 is repeatedly executed until an affirmative determination result is obtained.

[0122] When the determination result in step S31 is affirmative, position information PPS is generated (step S32). The information processing device 42 generates coordinate information (i.e., position information PPS) of the pixels constituting the signal lamp TS based on the area specified by the operator.

[0123] Following the process of step S32, support information ASS including the position information PPS is output (step S33). This support information ASS is sent from the information processing device 42 to the communication device 41. Thereafter, the support information ASS is sent to the communication device 37 via the network 2.

[0124] The operator performing remote support may also identify the case where the indication content of the lit part corresponds to "traffic permission". In this case, the operator may also operate the input device 44 to input a traffic permission signal STA. When the traffic permission signal STA is input, the information processing device 42 adds this traffic permission signal STA to the support information ASS.

[0125] Next, in step S33, it is determined whether there is an input indicating the end of an operation performed by an operator (step S34). The operator performing remote support identifies the start of the autonomous vehicle 3 or confirms the passage of the autonomous vehicle 3 through the OAS section based on the captured image information displayed on the display device 43. Alternatively, the operator performing remote support identifies the case where the reception of the support request signal REQ has ended. In either case, the operator operates the input device 44 to input an instruction to end the remote support in response to the support request signal REQ. The process of step S34 is repeatedly executed until a positive determination result is obtained.

[0126] 4. Effects

[0127] According to the remote support system according to the present embodiment described above, even when the accuracy of identifying the traffic signal TS or the like is low, the support request signal REQ is responded to, and the support information ASS including the position information PPS is provided from the remote facility 4 to the autonomous vehicle 3. Therefore, it is possible to improve the situation where the accuracy of identifying the traffic signal TS or the like is low. Therefore, it is possible to identify the real-time information indicated by the traffic signal TS in the autonomous vehicle 3 and determine the passage permission conditions. Therefore, it is possible to improve the safety when automatically passing through the OAS section.

Claims

1. A remote support system for a vehicle, comprising: a vehicle that performs autonomous driving; and a remote facility that communicates with the vehicle, wherein the remote support system of the vehicle is characterized in that the vehicle is provided with a control device that performs autonomous driving control of the vehicle based on driving environment information including captured image information in front of the vehicle, the remote facility is provided with: an input device operated by an operator; a display device that displays the captured image information; and an information processing device that performs input information processing for processing input information received by the input device and display control processing for the display device, in the autonomous driving control, the control device determines whether the vehicle's passage permission condition according to the indication content of the traffic signal in front of the vehicle is satisfied based on the driving environment information, when it is determined that the passage permission condition is not satisfied, causes the vehicle to temporarily stop in front of the traffic signal, sends a support request signal requesting support to be implemented by the operator to the remote facility, the information processing device when receiving the support request signal, in the display control processing, performs processing of reference image information for displaying the captured image information indicating that the vehicle is temporarily stopped, when receiving the position information of the pixels of the traffic signal included in the reference image information as the input information, in the input information processing, sends support information including the position information to the vehicle, in the autonomous driving control, the control device further when receiving the support information, determines again whether the passage permission condition is satisfied based on the position information, when it is determined that the passage permission condition is satisfied, releases the temporary stop state of the vehicle and performs vehicle control for driving according to the indication content of the traffic signal.

2. The remote support system according to claim 1, characterized in that when the information processing device receives the vehicle's passage permission signal as the input information, in the input information processing, sends the support information including the passage permission signal to the vehicle, in the autonomous driving control, the control device further when it is determined that the passage permission condition is satisfied after receiving the passage permission signal, releases the temporary stop state and performs the vehicle control.

3. A remote support method for a vehicle, which remotely supports the autonomous driving of the vehicle, characterized in that the control device of the vehicle performs autonomous driving control of the vehicle based on driving environment information including captured image information in front of the vehicle, the information processing device of the remote facility that communicates with the vehicle performs: input information processing for processing input information received by an input device operated by an operator; and display control processing for a display device that displays the captured image information, in the autonomous driving control, the control device determining, based on the driving environment information, whether a traffic permission condition for the vehicle according to the indication content of the traffic light in front of the vehicle is satisfied; If it is determined that the passage permission condition is not satisfied, temporarily stopping the vehicle in front of the traffic light; sending a support request signal to the remote facility requesting support by the operator, The information processing device When the assistance request signal is received, in the display control process, processing is performed for displaying reference image information indicating the captured image information during the temporary stop of the vehicle, When receiving position information of pixels constituting a traffic light included in the reference image information as the input information, in the input information processing, support information including the position information is sent to the vehicle, The control device further comprises: When the support information is received, it is determined again whether the passage permission condition is satisfied based on the position information. When it is determined that the passage permission condition is satisfied, the temporary stop state of the vehicle is released, and the vehicle is controlled so as to travel in accordance with the instruction content of the traffic light.

4. The remote support method according to claim 3, It is characterized in that When the information processing device receives a pass permission signal from the vehicle as the input information, the information processing device transmits the support information including the pass permission signal to the vehicle in the input information processing; The control device further comprises: When it is determined that the passage permission condition is satisfied after the passage permission signal is received, the temporary stop state is released and the vehicle control is performed.

Images