Image display device

Abstract

To provide an image display device which can notify a remote operator of proper information.SOLUTION: An image display device comprises: an external sensor data acquisition unit which acquires external sensor data detected at the first time by an external sensor detecting information about an external environment of a vehicle through the communication from the vehicle; an internal sensor data acquisition unit which acquires internal sensor data detected at the first time by an internal sensor detecting the travel situation of the vehicle through the communication from the vehicle; a decision unit which decides the position of the vehicle at the second time being the future time that has elapsed a prescribed time from the first time on the basis of the internal sensor data of the first time; and a display control unit which displays on a display unit a monitor image showing the periphery of the vehicle on the basis of the external sensor data of the first time. The display control unit superimposes an object indicating the position of the vehicle at the second time on a position on the monitor image corresponding to the position of the vehicle at the second time decided by the decision unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This disclosure relates to an image display device.

Background Art

[0002] Patent Document 1 discloses a device that displays a camera image to a remote operator based on camera image data received from a remotely controllable autonomous vehicle. This device causes a display unit to display a camera image with information related to the running of the vehicle superimposed thereon.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] A remote operator checks a camera image displayed on a display unit and remotely instructs an autonomous vehicle. The device described in Patent Document 1 has room for improvement from the viewpoint of notifying a remote operator of more appropriate information as judgment material. This disclosure provides an image display device capable of notifying a remote operator of appropriate information.

Means for Solving the Problems

[0005] An image display device relating to one aspect of this disclosure is connected to a display unit that displays information to a remote operator who remotely controls a vehicle. This image display device includes an external sensor data acquisition unit that acquires external sensor data detected at a first time by an external sensor that acquires information about the vehicle's external environment via communication from the vehicle; an internal sensor data acquisition unit that acquires internal sensor data detected at a first time by an internal sensor that acquires the vehicle's driving status via communication from the vehicle; a determination unit that determines the position of the vehicle at a second time, which is a future time after a predetermined time has elapsed from the first time, based on the internal sensor data at the first time; and a display control unit that causes the display unit to display a monitoring image showing the area around the vehicle based on the external sensor data at the first time. The display control unit superimposes an object indicating the position of the vehicle at the second time onto the position on the monitoring image corresponding to the position of the vehicle at the second time determined by the determination unit.

[0006] In an image display device relating to one aspect of this disclosure, the position of the vehicle at a second time point is calculated based on internal sensor data detected by the vehicle's internal sensors at a first time point. The calculated position of the vehicle at the second time point is superimposed as an object on a monitoring image displayed based on external sensor data at the first time point and provided to the remote operator. Therefore, this image display device can inform the remote operator of information that allows them to determine whether the vehicle's state at the first time point will be maintained until the second time point.

[0007] In one embodiment, the image display device includes a calculation unit that calculates the delay time of communication with the vehicle, a determination unit that determines the range of possible positions for the vehicle at a second time based on the delay time, and a display control unit that may superimpose an object indicating the range onto the position on the monitoring image corresponding to the range. In this case, the image display device can inform the remote operator that a communication delay has occurred and the effect that the delay has on the vehicle's position at a second time.

[0008] In one embodiment, the display control unit may superimpose an object indicating a range onto the position on the monitoring image corresponding to the range when the delay degree, which is defined based on at least one of the absolute value of the delay time and the variance value within a predetermined time, and in which a larger value indicates a greater delay, is greater than or equal to a threshold. In this case, the image display device may not notify the remote operator when the degree of delay is small, but may notify the remote operator when the degree of delay is large.

[0009] In one embodiment, the object indicating the vehicle's position at a second time point is a guide line object, and the object indicating a range is an object extending in the line width direction of the guide line object. The display control unit may increase the length of the range-indicating object in the line width direction as the degree of delay increases. In this case, the image display device can use the guide line object and the object extending in the line width direction to inform the remote operator of the degree of delay.

[0010] In one embodiment, the determination unit detects other vehicles present around the vehicle based on external sensor data at a first time point and determines the position of the other vehicles at a second time point. The display control unit may then superimpose an object indicating the position of the other vehicles at the second time point onto the position on the monitoring image corresponding to the position of the other vehicles at the second time point determined by the determination unit. In this case, the image display device can inform a remote operator of the position of the other vehicles at the second time point. [Effects of the Invention]

[0011] According to this disclosure, an image display device is provided that can provide appropriate information to a remote operator. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows an overview of a vehicle remote instruction system including an image display device according to one embodiment. [Figure 2] This figure shows an example of the hardware configuration of a remote control device. [Figure 3]This is a block diagram showing the functions of an image display device. [Figure 4] (A) is an example of a monitoring image with guideline objects superimposed. (B) is an example of a monitoring image with guideline objects and delay-indicating objects superimposed. [Figure 5] (A) is another example of a monitoring image with guideline objects superimposed. (B) is another example of a monitoring image with guideline objects and delay-indicating objects superimposed. [Figure 6] This is a flowchart illustrating an example of the display process of an image display device. [Figure 7] This flowchart shows another example of the display processing for an image display device. [Figure 8] (A) is an example of a monitoring image in which guideline objects are superimposed on other vehicles. (B) is an example of a monitoring image in which guideline objects and delay-indicating objects are superimposed on a specific other vehicle. [Figure 9] This is a flowchart illustrating an example of the decision-making process for an image display device. [Figure 10] (A) is another example of a monitoring image with guideline objects superimposed. (B) is another example of a monitoring image with guideline objects and delay-indicating objects superimposed. [Modes for carrying out the invention]

[0013] The embodiments will be described below with reference to the drawings. In the following description, the same or equivalent elements will be denoted by the same reference numerals, and redundant descriptions will not be repeated.

[0014] FIG. 1 is a diagram showing an overview of a vehicle remote instruction system including an image display device according to an embodiment. The image display device 1 shown in FIG. 1 is included in the vehicle remote instruction system 100. The vehicle remote instruction system 100 is a system that operates the vehicle 2 based on a remote instruction by a remote operator R. The remote instruction is an instruction of the remote operator R regarding the behavior of the vehicle 2. The vehicle 2 is not particularly limited, but as an example, it is a vehicle that travels by automatic driving. Hereinafter, it will be described assuming that the vehicle 2 is a vehicle having an automatic driving function. The remote instruction may include at least one of an instruction to start a right turn at an intersection, an instruction to start entering an intersection with a traffic signal, an instruction to start entering an intersection with poor visibility, an instruction to start a lane change, an instruction to start offset avoidance for an obstacle ahead, and an emergency evacuation. In addition, the remote instruction may include an instruction for starting the vehicle 2 that was in a parked state. The remote instruction may include an instruction for starting the vehicle 2 when a pedestrian or the like is detected around the vehicle 2 that was in a stopped state. The remote instruction may include an instruction regarding boarding and alighting of passengers on the vehicle 2 (for example, an instruction for automatic opening and closing of a door, an instruction for starting voice guidance for alighting).

[0015] [Configuration of Vehicle Remote Instruction System] As shown in FIG. 1, the vehicle remote instruction system 100 includes a remote instruction device 10. The remote instruction device 10 includes an image display device 1 and is connected to an operator interface 3. The operator interface 3 displays information to the remote operator R under the control of the image display device 1. The operator interface 3 receives a remote instruction from the remote operator R and outputs it to the remote instruction device 10. The remote instruction device 10 is communicably connected to the vehicle 2 via the network N. The network N is a wireless communication network. The remote instruction device 10 receives various information from the vehicle 2. Note that the remote instruction device 10 may be communicably connected to other vehicles via the network N.

[0016] In the vehicle remote instruction system 100, for example, in response to a remote instruction request from vehicle 2, an input of a remote instruction is requested from the remote operator R. The remote operator R inputs a remote instruction to the operator interface 3. The remote instruction device 10 transmits a remote instruction to vehicle 2 through the network N. Vehicle 2 automatically travels according to the remote instruction.

[0017] In the vehicle remote instruction system 100, the number of remote operators R is not limited, and it may be one person or two or more persons. The number of vehicles 2 that can communicate with the vehicle remote instruction system 100 is not particularly limited either. A mode in which a plurality of remote operators R alternately give remote instructions to one vehicle 2 may be adopted, or a mode in which one remote operator R gives remote instructions to two or more vehicles 2 may be adopted.

[0018] As an example, vehicle 2 has an automatic driving ECU (Electronic Control Unit) 20. The automatic driving ECU 20 is an electronic control unit having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. In the automatic driving ECU 20, for example, a program recorded in the ROM is loaded into the RAM, and various functions are realized by executing the program loaded into the RAM with the CPU. The automatic driving ECU 20 may be composed of a plurality of electronic units. The automatic driving ECU 20 is communicably connected to a GPS (Global Positioning System) receiver 21, an external sensor 22, an internal sensor 23', a map database 24, a vehicle communication unit 25, and an actuator 26.

[0019] The GPS receiver 21 is a device that receives signals from three or more GPS satellites. The GPS receiver 21 measures the position of vehicle 2 (for example, the latitude and longitude of vehicle 2) based on the received signals. The GPS receiver 21 outputs information regarding the measured position. The GPS receiver 21 also outputs the GPS time measured in the GPS time system.

[0020] Note: There seems to be a small error in the original text where it says "map database 24, a vehicle communication unit 25, and an actuator 26. " and then in the following part starts with "GPS receiver 21" without closing the previous sentence properly. I've translated it as best as possible with the given text. Also, I'm assuming the' in "internal sensor 23'" is a typo and left it as is in the translation. The external sensor 22 is a sensor that detects information about the external environment of the vehicle 2. The external sensor 22 outputs the detection result as external sensor data. The external sensor 22 includes a camera. The camera is an imaging device that captures images of the external environment of the vehicle 2. The camera outputs image data related to the external environment of the vehicle 2. Image data is information that can be used to create an image. The camera is installed, for example, behind the windshield of the vehicle 2 and captures images of the area in front of the vehicle. The camera may be a monocular camera or a stereo camera. Multiple cameras may be installed and may capture images of the left and right sides and the rear of the vehicle 2, in addition to the front.

[0021] The external sensor 22 may include a radar sensor. A radar sensor is a detection device that uses radio waves (e.g., millimeter waves) or light to detect objects around the vehicle 2. Examples of radar sensors include millimeter-wave radar or lidar (LIDAR: Light Detection and Ranging). The radar sensor detects objects by transmitting radio waves or light around the vehicle 2 and receiving the radio waves or light reflected by the objects, and outputs the result. Objects include fixed objects such as guardrails and buildings, as well as moving objects such as pedestrians, bicycles, and other vehicles. The detection result of the radar sensor can be handled as three-dimensional image data.

[0022] The internal sensor 23 is a sensor that detects the driving state of the vehicle 2. The internal sensor 23 outputs the detection result as internal sensor data. The internal sensor 23 includes a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. The vehicle speed sensor is a detector that detects the speed of the vehicle 2. As the vehicle speed sensor, a wheel speed sensor can be used that is installed on the wheels of the vehicle 2 or on a drive shaft that rotates integrally with the wheels and detects the rotational speed of each wheel.

[0023] An acceleration sensor is a detector that detects the acceleration of vehicle 2. The acceleration sensor may include, for example, a longitudinal acceleration sensor that detects the longitudinal acceleration of vehicle 2. The acceleration sensor may also include a lateral acceleration sensor that detects the lateral acceleration of vehicle 2. A yaw rate sensor is a detector that detects the yaw rate (rotational angular velocity) of vehicle 2 around the vertical axis of the center of gravity. For example, a gyro sensor can be used as the yaw rate sensor.

[0024] The map database 24 is a database that records map information. The map database 24 is formed in a recording device such as an HDD (Hard Disk Drive) installed in the vehicle 2. The map information includes road location information, road shape information (e.g., curvature information), and location information of intersections and junctions. The map information may also include traffic regulation information such as legal speed limits associated with the location information. The map information may also include landmark information used to obtain the location information of the vehicle 2. Landmarks can include road signs, road markings, traffic lights, utility poles, etc. The map database 24 may be configured as a server that can communicate with the vehicle 2.

[0025] The vehicle communication unit 25 is a communication device that controls wireless communication between the vehicle 2 and the outside world. The vehicle communication unit 25 transmits and receives various information to and from the remote control device 10 via the network N. The vehicle communication unit 25 transmits external sensor data, internal sensor data, and GPS time to the remote control device 10 for the control of the image display device 1.

[0026] The actuator 26 is a device used to control the vehicle 2. The actuator 26 includes a drive actuator, a brake actuator, and a steering actuator. The drive actuator controls the amount of air supplied to the engine (throttle opening) in response to a control signal from the automatic driving ECU 20, thereby controlling the driving force of the vehicle 2. If the vehicle 2 is a hybrid vehicle, in addition to the amount of air supplied to the engine, a control signal from the automatic driving ECU 20 is input to the motor, which is a power source, to control its driving force. If the vehicle 2 is an electric vehicle, a control signal from the automatic driving ECU 20 is input to the motor, which is a power source, to control its driving force. In these cases, the motor, which is a power source, constitutes the actuator 26.

[0027] The brake actuator controls the brake system in accordance with control signals from the autonomous driving ECU 20, and controls the braking force applied to the wheels of vehicle 2. For example, a hydraulic brake system can be used as the brake system. The steering actuator controls the drive of the assist motor that controls the steering torque within the electric power steering system in accordance with control signals from the autonomous driving ECU 20. In this way, the steering actuator controls the steering torque of vehicle 2.

[0028] The autonomous driving ECU 20 performs various functions based on the output results of the above-mentioned devices or the information held by the devices. The autonomous driving ECU 20 includes a vehicle position acquisition unit 200, an external environment recognition unit 201, a driving state recognition unit 202, a remote instruction request determination unit 203, a route generation unit 204, an autonomous driving control unit 205, and an emergency brake determination unit 206.

[0029] The vehicle position acquisition unit 200 acquires the position information of vehicle 2 based on the position information of the GPS receiver unit 21 and the map information of the map database 24. The vehicle position acquisition unit 200 may also acquire the position information of vehicle 2 using SLAM (Simultaneous Localization and Mapping) technology, utilizing the landmark information included in the map information of the map database 24 and the detection results of the external sensor 22. The vehicle position acquisition unit 200 may also recognize the lateral position of vehicle 2 relative to the lane (the position of vehicle 2 in the lane width direction) from the positional relationship between the lane markings and vehicle 2 and include it in the position information. The vehicle position acquisition unit 200 may also acquire the position information of vehicle 2 by other well-known methods.

[0030] The external environment recognition unit 201 recognizes the external environment of the vehicle 2 based on the detection results of the external sensor 22. The external environment includes the relative positions of surrounding objects with respect to the vehicle 2. The external environment may also include the relative speed and direction of movement of surrounding objects with respect to the vehicle 2. The external environment may also include the types of objects, such as other vehicles, pedestrians, and bicycles. The types of objects can be identified by well-known methods such as pattern matching. The external environment may also include the results of lane marking recognition (white line recognition) around the vehicle 2. The external environment may also include the recognition results of the lighting status of traffic lights. For example, the external environment recognition unit 201 can recognize the lighting status of traffic lights in front of the vehicle 2 based on the camera image from the external sensor 22.

[0031] The driving state recognition unit 202 recognizes the driving state of vehicle 2 based on the detection results of the internal sensor 23. The driving state includes the vehicle speed of vehicle 2, the acceleration of vehicle 2, and the yaw rate of vehicle 2. Specifically, the driving state recognition unit 202 recognizes the vehicle speed of vehicle 2 based on the vehicle speed information from the vehicle speed sensor. The driving state recognition unit 202 recognizes the acceleration of vehicle 2 based on the vehicle speed information from the acceleration sensor. The driving state recognition unit 202 recognizes the orientation of vehicle 2 based on the yaw rate information from the yaw rate sensor.

[0032] The remote instruction request determination unit 203 determines whether or not to request remote instructions from the remote operator R. The remote instruction request determination unit 203 determines whether or not to request remote instructions based on at least one of the following: the location information of the vehicle 2 acquired by the vehicle position acquisition unit 200 and the map information of the map database 24, the external environment recognized by the external environment recognition unit 201, and the route generated by the route generation unit 204, which will be described later.

[0033] The remote instruction request determination unit 203 determines that a remote instruction should be requested when the vehicle 2 is in a situation where a pre-set remote instruction should be requested. For example, the remote instruction request determination unit 203 determines that a remote instruction should be requested when the vehicle 2 is in a situation where it is turning right at an intersection. The remote instruction request determination unit 203 may also determine that a remote instruction should be requested when the vehicle 2 is entering a signalized intersection or an intersection with poor visibility. The remote instruction request determination unit 203 may also determine that a remote instruction should be requested when the vehicle 2 is in a situation where it is beginning to change lanes in order to reach its destination. The remote instruction request determination unit 203 may also determine that a remote instruction should be requested when there is an obstacle in front of the vehicle 2 that should be offset to avoid.

[0034] The remote instruction request determination unit 203 can recognize, for example, from the location information, map information, and path of the vehicle 2, that the vehicle 2 is in a situation where it is turning right at an intersection, that the vehicle 2 is entering an intersection with traffic lights, or that the vehicle 2 is beginning to change lanes. The remote instruction request determination unit 203 can also recognize, based on the external environment of the vehicle 2, that there is an obstacle in front of the vehicle 2 that should be offset to avoid.

[0035] The remote instruction request determination unit 203, if it determines that a remote instruction should be requested, requests a remote instruction from the remote instruction device 10 by the remote operator R. The remote instruction request includes, for example, identification information of the vehicle 2. The remote instruction request determination unit 203 may request a remote instruction with sufficient lead time in advance. The remote instruction request determination unit 203 may also determine that a remote instruction should be requested when the distance between the vehicle 2 and the intersection or other location to which the remote instruction is to be made falls below a certain distance. Instead of distance, the remaining time to reach the destination may be used.

[0036] If the remote instruction request determination unit 203 determines that a remote instruction should be requested, it transmits information on the vehicle's driving status to the remote instruction device 10. The information on the vehicle's driving status includes information that allows the remote operator R to recognize the status of the vehicle.

[0037] The vehicle 2 driving status information includes the vehicle 2's position information, external sensor data detected by the external sensor 22, and internal sensor data detected by the internal sensor 23. The external sensor data includes camera images, laser radar detection results, etc. The internal sensor data includes the vehicle speed of the vehicle 2, the yaw rate of the vehicle 2, the steering angle of the vehicle 2, etc. The vehicle 2 driving status information may also include information about the occupants (presence or absence of occupants, number of occupants). The vehicle 2 driving status information may also include information about the course in response to selectable remote instructions from the remote operator R.

[0038] Regardless of whether or not it has determined whether or not to request remote instructions, the autonomous driving ECU 20 may transmit information about the vehicle 2's driving status to the remote instruction device 10 at a predetermined timing. The predetermined timing may be at regular intervals, at each road node on the map, or at each event such as when approaching a pedestrian crossing or an intersection.

[0039] The trajectory generation unit 204 generates a trajectory to be used for the autonomous driving of vehicle 2. The trajectory generation unit 204 generates the autonomous driving trajectory based on a pre-set driving route, map information, the location information of vehicle 2, the external environment of vehicle 2, and the driving state of vehicle 2.

[0040] The driving route is the route that vehicle 2 travels in autonomous driving. The route generation unit 204 determines the autonomous driving route based, for example, on the destination, map information, and the location information of vehicle 2. The driving route may be set by a well-known navigation system. The destination may be set by the occupants of vehicle 2, or it may be automatically suggested by the autonomous driving ECU 20 or the navigation system, etc.

[0041] The path includes the route the vehicle will travel in autonomous driving mode and the vehicle speed profile in autonomous driving mode. The path is the trajectory that the vehicle is scheduled to travel along the route while in autonomous driving mode. The path can be, for example, data on the change in steering angle of vehicle 2 according to its position on the route (steering angle profile). The position on the route is, for example, a set longitudinal position set at predetermined intervals (e.g., 1m) in the direction of travel along the route. The steering angle profile is data in which a target steering angle is associated with each set longitudinal position.

[0042] The route generation unit 204 generates a path for vehicle 2 based on, for example, the driving route, map information, the external environment of vehicle 2, and the driving state of vehicle 2. The route generation unit 204 generates a path such that vehicle 2 passes through the center of the lanes included in the driving route (the center in the direction of the lane width).

[0043] A vehicle speed profile is data associated with a target vehicle speed for each set longitudinal position. The set longitudinal position may be set based on the vehicle's travel time rather than distance. The set longitudinal position may also be set as the vehicle's destination after 1 second, the vehicle's destination after 2 seconds, etc.

[0044] The route generation unit 204 generates a vehicle speed profile based, for example, on the route and traffic regulation information such as legal speed limits included in the map information. Instead of legal speed limits, a pre-set speed for a location or section on the map may be used. The route generation unit 204 generates an automated driving route from the route and vehicle speed profile. The method of route generation in the route generation unit 204 is not limited to the above, and well-known methods related to automated driving can be adopted. The same applies to the content of the route.

[0045] When the remote instruction request determination unit 203 requests a remote instruction from the remote instruction device 10, the route generation unit 204 pre-generates a route corresponding to the remote instruction. The content of the remote instruction is predetermined according to the status of the vehicle 2. For example, the content of the remote instruction when turning right at an intersection includes a remote instruction to start turning right (start moving) and a remote instruction to wait. The content of the remote instruction when turning right at an intersection may also include a remote instruction to abandon the right turn and go straight, or a remote instruction to take emergency refuge.

[0046] The route generation unit 204 generates a route for vehicle 2 to turn right at an intersection, for example, in a situation where vehicle 2 is turning right at an intersection, in response to a remote instruction to start a right turn. The route generation unit 204 may update the route in response to changes in the external environment until it receives a remote instruction. In addition, if there is a remote instruction to switch from turning right at an intersection to going straight at an intersection, the route generation unit 204 may pre-generate a route for going straight at an intersection.

[0047] The route generation unit 204 may pre-generate an emergency evacuation route if there is a remote instruction for emergency evacuation. The emergency evacuation route is generated to stop vehicle 2 in one of the evacuation spaces pre-set on the map. The route generation unit 204 recognizes, for example, the presence or absence of obstacles in each evacuation space based on the external environment and generates an emergency evacuation route to stop in an available evacuation space. Note that the route generation unit 204 does not necessarily need to pre-generate a route; it may generate a route corresponding to a remote instruction after receiving the remote instruction.

[0048] The automatic driving control unit 205 performs automatic driving of the vehicle 2. The automatic driving control unit 205 performs automatic driving of the vehicle 2 based on, for example, the external environment of the vehicle 2, the driving state of the vehicle 2, and the path generated by the path generation unit 204. The automatic driving control unit 205 performs automatic driving of the vehicle 2 by transmitting control signals to the actuator 26.

[0049] When the remote instruction request determination unit 203 requests a remote instruction from the remote instruction device 10, the automatic driving control unit 205 waits for the remote instruction to be received from the remote instruction device 10. If the vehicle 2 has stopped and a remote instruction is requested, the automatic driving control unit 205 maintains the stopped state until the remote instruction is received.

[0050] If a licensed driver is on board and the automatic driving control unit 205 does not receive remote instructions after a preset waiting period has elapsed, it may request the driver to make a decision or take manual control. If the automatic driving control unit 205 does not receive remote instructions after the waiting period has elapsed and neither the driver making a decision nor taking manual control is possible, it may automatically perform an emergency evacuation.

[0051] The emergency brake determination unit 206 determines whether emergency braking is necessary based on the external environment of the vehicle 2 or the detection results of the external sensor 22 (camera images and / or object information from the radar sensor). For example, the emergency brake determination unit 206 determines that emergency braking is necessary when the time to collision (TTC) between the vehicle 2 and an obstacle in front of the vehicle 2 falls below the TTC threshold. The TTC threshold is a preset threshold value. Alternatively, the time headway (THW) or the distance between the vehicle 2 and the obstacle may be used instead of the collision time.

[0052] The emergency brake determination unit 206, if it determines that emergency braking is necessary, sends a control signal to the brake actuator to execute emergency braking. The emergency brake determination is performed independently of the autonomous driving function. The emergency brake determination unit 206 may be formed in an electronic unit different from the autonomous driving ECU 20. Emergency braking may be performed in a so-called PCS (Pre-Crash Safety) system.

[0053] [Configuration of the remote control device] Figure 2 shows an example of the hardware configuration of a remote control device. As shown in Figure 2, the remote control device 10 is configured as a general-purpose computer equipped with a processor 10a, a recording unit 10b, a communication unit 10c, and a user interface 10d.

[0054] The processor 10a controls the remote control device 10 by running various operating systems. The processor 10a is an arithmetic unit such as a CPU, which includes a control unit, arithmetic unit, registers, etc. The processor 10a manages the recording unit 10b, the communication unit 10c, and the user interface 10d. The recording unit 10b includes at least one of memory and storage. Memory is a recording medium such as ROM or RAM. Storage is a recording medium such as HDD.

[0055] The communication unit 10c is a communication device for communication via the network N. The communication unit 10c can utilize network devices, network controllers, network cards, etc. The user interface 10d is an input / output unit for users such as administrators. The user interface 10d includes output devices such as displays and speakers, and input devices such as touch panels. The remote control device 10 does not necessarily need to be installed in a facility; it may be mounted on a mobile device such as a vehicle.

[0056] [Configuration and Functions of Image Display Devices] Since the image display device 1 is provided with the remote control device 10, its hardware is identical to that of the remote control device 10. The hardware of the image display device 1 is not limited to being identical or common to that of the remote control device 10, and may be separate from the remote control device 10 and have the configuration shown in Figure 2. Figure 3 is a block diagram showing the functions of the image display device. As shown in Figure 3, the image display device 1 includes a communication unit 11 as a communication function. The functions of the communication unit 11 are realized by the processor 10a and the communication unit 10c.

[0057] The communication unit 11 is connected to the vehicle communication unit 25 of vehicle 2 via network N. The vehicle communication unit 25 of vehicle 2 has an external sensor data transmission unit 250, an internal sensor data transmission unit 251, and a time transmission unit 252 for transmitting external sensor data, internal sensor data, and GPS time. The communication unit 11 has an external sensor data receiving unit 110 (an example of an external sensor data acquisition unit), an internal sensor data receiving unit 111 (an example of an internal sensor data acquisition unit), and a time receiving unit 112 for receiving external sensor data, internal sensor data, and GPS time. The external sensor data receiving unit 110 acquires external sensor data detected at a first time by the external sensor 22 of vehicle 2 from the external sensor data transmission unit 250 of vehicle 2 via network N. The internal sensor data receiving unit 111 acquires internal sensor data detected at a first time by the internal sensor 23 of vehicle 2 from the internal sensor data transmission unit 251 of vehicle 2 via network N. The first time is the detection time of the external sensor 22 and the internal sensor 23. The time receiving unit 112 acquires the GPS time from the time transmitting unit 252 of the vehicle 2 via the network N.

[0058] In addition to the communication unit 11, the image display device 1 includes a delay time calculation unit 12 (an example of a calculation unit), an assist information generation unit 13 (an example of a determination unit), and a display control unit 14.

[0059] The delay time calculation unit 12 calculates the delay time of communication with the vehicle 2. The delay time calculation unit 12 compares the GPS time obtained by the time reception unit 112 with the GPS time obtained by the GPS reception unit (not shown) of the image display device 1 to calculate the communication delay time.

[0060] The assist information generation unit 13 generates assist information, which is information that assists the decision-making of the remote operator R. The assist information generation unit 13 determines the position of vehicle 2 at a second time, which is a future time that has elapsed a predetermined time from the first time. A future time is a time that is ahead of the current time. When the processing time of the image display device 1 is used as the reference point, the first time is a past time that is close to the current time, and the second time is a future time that is ahead of the current time. Multiple second time settings may be set. The assist information generation unit 13 determines the position of vehicle 2 at the second time based on the sensor data of the first time acquired by the internal sensor data receiving unit 111. As an example, the assist information generation unit 13 assumes that there is no change in the remote instructions from the remote operator R from the first time to the second time, and determines the position of vehicle 2 at the second time based on the sensor data of the first time. For example, if the vehicle speed at the first time is 60 km / h, the assist information generation unit 13 assumes that the vehicle speed will be maintained at 60 km / h until the second time. For example, if the assist information generation unit 13 is decelerating at the first time point, it assumes that the vehicle will continue to decelerate at the same acceleration until the second time point. In this way, the assist information generation unit 13 determines the future position of the vehicle 2 assuming there is no change in the vehicle's driving state.

[0061] The assist information generation unit 13 may determine the possible range of the vehicle 2's position at the second time step based on the delay time calculated by the delay time calculation unit 12. For example, if the vehicle speed is 60 km / h and the delay time is 0.1 seconds, an error of approximately 1.7 m will occur. The assist information generation unit 13 may determine the maximum delay position by subtracting the distance of the error from the position of the vehicle 2 at the second time step determined based on the sensor data at the first time step. The assist information generation unit 13 may determine the range from the maximum delay position to the position of the vehicle 2 at the second time step as the possible range. The assist information generation unit 13 may determine the possible range by utilizing the position of the vehicle 2 at the second time step, the error distance, and a normal distribution.

[0062] The display control unit 14 displays a monitoring image showing the area around the vehicle 2 on the operator interface 3 based on the external sensor data at a first time point acquired by the external sensor data receiving unit 110. The operator interface 3 includes a display device 30 (an example of a display unit). The display control unit 14 is connected to the display device 30. The monitoring image is an image displayed on the screen of the display device 30, and can be any image showing the area around the vehicle 2. For example, the monitoring image may be a bird's-eye view of the vehicle 2 from directly above or from an oblique angle, or it may be the camera image or radar sensor image acquired by the external sensor 22. In addition to the display device 30, the operator interface 3 may also include an instruction receiving unit 31 for operation by the remote operator R.

[0063] The display control unit 14 superimposes an object indicating the position of vehicle 2 at the second time step onto the position on the monitoring image corresponding to the position of vehicle 2 at the second time step determined by the assist information generation unit 13. The object indicating the position of vehicle 2 at the second time step is, for example, a guide line object. For example, the guide line object can be a solid line, dashed line, dotted line, etc., and can be assigned various colors. The object indicating the position of vehicle 2 at the second time step is not limited to these. For example, the object indicating the position of vehicle 2 at the second time step may be a circle or a rectangle.

[0064] Figure 4(A) is an example of a monitoring image with guideline objects superimposed. In the monitoring image G1 of Figure 4(A), the vehicle object OV1, which represents vehicle 2, is located in the center of the screen. The assist information generation unit 13 generates, as an example, the position of vehicle 2 at three second time points: 1 second, 2 seconds, and 3 seconds after the current time. The display control unit 14 superimposes the respective guideline objects onto the monitoring image G1 based on the positions of vehicle 2 at the three second time points determined by the assist information generation unit 13. For example, the display control unit 14 superimposes the line object OL1, which surrounds vehicle 2 and indicates its position at 1 second later, onto the monitoring image G1. The display control unit 14 can superimpose an object M1, which indicates information related to the second time point (e.g., elapsed time from the current time), at a screen position corresponding to the line object OL1. Similarly, the display control unit 14 superimposes the line object OL2, which indicates the position of vehicle 2 at 2 seconds later, and the corresponding object M2 related to the second time point. The display control unit 14 superimposes a line object OL3 indicating the position of vehicle 2 three seconds later and a corresponding object M3 related to the second time point.

[0065] The display control unit 14 may superimpose objects related to delay time onto the monitoring image. For example, the display control unit 14 may superimpose the range of possible positions determined by the assist information generation unit 13 as objects onto the monitoring image. The objects indicating the range of possible positions may be highly transparent figures or may be colored in various ways. Figure 4(B) is an example of a monitoring image in which guideline objects and delay-indicating objects are superimposed. In the monitoring image G2 of Figure 4(B), the vehicle object OV1, which represents vehicle 2, is located in the center of the screen. The following explanation will use the case in which delay-indicating objects are added to the monitoring image of Figure 4(A) as an example. As an example, the assist information generation unit 13 determines the range of possible positions for three second time points: 1 second, 2 seconds, and 3 seconds after the current time. The display control unit 14 superimposes each object onto the monitoring image G2 based on the range of possible positions corresponding to the three second time points from the assist information generation unit 13. For example, the display control unit 14 superimposes object B1, which extends in the line width direction of line object OL1, to show the range of possible positions for vehicle 2 after 1 second. Similarly, the display control unit 14 superimposes object B2, which extends in the line width direction of line object OL2, to show the range of possible positions for vehicle 2 after 2 seconds. The display control unit 14 superimposes object B3, which extends in the line width direction of line object OL3, to show the range of possible positions for vehicle 2 after 3 seconds.

[0066] The display control unit 14 may indicate the magnitude of the delay by the line width length of objects B1 to B3. For example, the display control unit 14 determines the delay degree using the delay time calculated by the assist information generation unit 13. The delay degree is a value that indicates a larger delay as the delay time increases. The delay degree may also be the absolute value of the delay time. The display control unit 14 may also determine the delay degree using the variance value of the delay time within a predetermined period calculated by the assist information generation unit 13. In this case, the delay degree is a value that indicates a larger delay as the variance value increases. The delay degree may also be the variance value itself. Alternatively, the delay degree may be a weighted sum or weighted average of the delay degree of the delay time and the delay degree of the variance value. Alternatively, the delay degree may be determined based on the relationship between the absolute value of the delay time and the variance value. Specifically, the degree of delay is determined to be small if the absolute value of the delay time is smaller than a predetermined threshold, regardless of the variance of the delay time (even if the variance of the delay time is large). On the other hand, if the absolute value of the delay time is greater than or equal to the predetermined threshold, the degree of delay may be determined to be larger the larger the variance of the delay time, and smaller the degree of delay may be determined to be smaller the smaller the variance of the delay time. In this way, the degree of delay is defined based on at least one of the absolute value of the delay time and the variance within the predetermined time. The display control unit 14 can express the magnitude of the delay by changing the length of the line width direction of objects B1 to B3 to be longer as the degree of delay increases.

[0067] The display control unit 14 may switch between the monitoring image G1 in Figure 4(A) and the monitoring image G2 in Figure 4(B) depending on the conditions. For example, the display control unit 14 displays the monitoring image G1 in Figure 4(A) on the display device 30 when the delay is less than a threshold, and displays the monitoring image G2 in Figure 4(B) on the display device 30 when the delay is greater than or equal to the threshold. The threshold is a value set in advance for determining the delay. For example, the threshold may be determined by the interval of the guide lines displayed on the monitoring image and the set ratio. For example, if the delay time is the delay, and the interval of the guide lines is 1 second and the set ratio is 0.5, then the threshold is set to 0.5 seconds. In this case, the display control unit 14 displays the monitoring image G1 when the delay time is less than 0.5 seconds, and displays the monitoring image G2 when the delay time is 0.5 seconds or more.

[0068] Figure 5(A) shows another example of a monitoring image with guideline objects superimposed. As described above, guideline objects can be represented in various forms. In the monitoring image G3 of Figure 5(A), a vehicle object OV1 representing vehicle 2 and its planned route PA are shown. For such a display, the image display device 1 obtains the planned route from vehicle 2 in advance via the communication unit 11. The display control unit 14 superimposes each guideline object onto the planned route PA of the monitoring image G1 based on the position of vehicle 2 at five second time points determined by the assist information generation unit 13. For example, the display control unit 14 superimposes a line object OL1 indicating the position of vehicle 2 one second later onto the monitoring image G1. The display control unit 14 can superimpose an object M1 indicating information related to the second time point (e.g., elapsed time from the current time) at the screen position corresponding to the line object OL1. In this case, the display control unit 14 may refer to map information for the shape of the object. Similarly, the display control unit 14 superimposes the line object OL2 indicating the position of vehicle 2 after 2 seconds and the corresponding object M2 associated with the second time point. The display control unit 14 superimposes the line object OL3 indicating the position of vehicle 2 after 3 seconds and the corresponding object M3 associated with the second time point. The display control unit 14 superimposes the line object OL4 indicating the position of vehicle 2 after 4 seconds and the corresponding object M4 associated with the second time point. The display control unit 14 superimposes the line object OL5 indicating the position of vehicle 2 after 5 seconds and the corresponding object M5 associated with the second time point.

[0069] Figure 5(B) shows another example of a monitoring image in which guideline objects and delay-indicating objects are superimposed. In Figure 5(B), as explained in Figure 4(B), delay-indicating objects B1-B5 are added to the monitoring image G3 in Figure 5(A). The method for setting objects B1-B5 and switching between monitoring images is the same as described above.

[0070] [Operation of the image display device] Figure 6 is a flowchart illustrating an example of the display process of an image display device. The flowchart shown in Figure 6 begins when an instruction to start the display function of the image display device 1 is given.

[0071] First, as part of the data acquisition process (S10), the external sensor data receiving unit 110 acquires external sensor data detected at the first time from the vehicle 2. The internal sensor data receiving unit 111 acquires internal sensor data at the first time from the vehicle 2.

[0072] Next, as part of the assist information generation process (S12), the assist information generation unit 13 determines the position of vehicle 2 at the second time step, assuming that there has been no change in the remote instructions from the remote operator R from the first time step to the second time step.

[0073] Next, as a display process (S14), the display control unit 14 displays a monitoring image showing the surroundings of the vehicle 2 on the display device 30 based on the external sensor data for the first time step, and superimposes an object indicating the position of the vehicle 2 at the second time step onto the position on the monitoring image corresponding to the position of the vehicle 2 at the second time step determined by the assist information generation unit 13.

[0074] When the display process (S14) is completed, the flowchart shown in Figure 6 is terminated. By executing the flowchart shown in Figure 6, the image display device 1 can appropriately inform the remote operator R of the future position of vehicle 2. After the flowchart is terminated, the image display device 1 restarts the flowchart from the beginning until an instruction to terminate the display function is given.

[0075] Figure 7 is a flowchart illustrating another example of the display process of an image display device. In the flowchart shown in Figure 7, the image display device 1 notifies the remote operator R of the delay time according to the conditions. The flowchart shown in Figure 7 starts when an instruction to start the display function of the image display device 1 is given.

[0076] First, as part of the data acquisition process (S20), the external sensor data receiving unit 110 acquires external sensor data detected at the first time from the vehicle 2. The internal sensor data receiving unit 111 acquires internal sensor data at the first time from the vehicle 2.

[0077] Next, as part of the GPS time acquisition process (S22), the time receiving unit 112 acquires the GPS time from the vehicle 2. The delay time calculation unit 12 also acquires its own GPS time from the GPS receiving unit (not shown) of the image display device 1.

[0078] Next, as part of the delay calculation process (S24), the delay time calculation unit 12 calculates the delay by comparing the GPS time of vehicle 2 with its own GPS time. As an example, the delay time calculation unit 12 uses the delay time, which is the difference between the GPS time of vehicle 2 and its own GPS time, as the delay. The delay time calculation unit 12 may also calculate the difference between the GPS time of vehicle 2 and its own GPS time for a predetermined period of time and use the variance of that difference as the delay.

[0079] Next, as a determination process (S26), the assist information generation unit 13 determines whether the delay is greater than or equal to a threshold. If the delay is not greater than or equal to a threshold, the assist information generation unit 13, as an assist information generation process (S28), determines the position of vehicle 2 at the second time step, assuming that there has been no change in the remote instructions from the remote operator R from the first time step to the second time step. If the delay is greater than or equal to a threshold, the assist information generation unit 13, as an assist information generation process (S30), determines the position of vehicle 2 at the second time step, assuming that there has been no change in the remote instructions from the remote operator R from the first time step to the second time step, and also determines the range of possible positions for vehicle 2 based on the delay time.

[0080] When the assist information generation process (S28 or S30) is completed, as a display process (S32), the display control unit 14 displays a monitoring image showing the area around the vehicle 2 on the display device 30 based on the external sensor data for the first time step, and superimposes an object indicating the position of the vehicle 2 at the second time step onto the position on the monitoring image corresponding to the position of the vehicle 2 at the second time step determined by the assist information generation unit 13. If the range of possible locations for the vehicle 2 has been determined by the assist information generation unit 13, the display control unit 14 further superimposes an object indicating the range of possible locations for the vehicle 2 onto the monitoring image.

[0081] When the display process (S32) is completed, the flowchart shown in Figure 7 ends. By executing the flowchart shown in Figure 7, the image display device 1 can notify the remote operator R of the delay time according to the conditions. After the flowchart ends, the image display device 1 restarts the flowchart from the beginning until an instruction to terminate the display function is given.

[0082] [Summary of Embodiments] In one embodiment of the image display device 1, the position of the vehicle 2 at a second time point is calculated based on internal sensor data detected by the vehicle 2's internal sensor 23 at a first time point, assuming there is no change in remote instructions from the remote operator R from the first time point to the second time point. The calculated position of the vehicle 2 at the second time point is superimposed as an object on the monitoring image displayed based on the external sensor data at the first time point and provided to the remote operator R. Therefore, the image display device 1 can inform the remote operator R of information that allows them to determine whether to maintain the vehicle state at the first time point until the second time point.

[0083] The image display device 1 can notify the remote operator R of the communication delay and the effect of that delay on the vehicle's position at the second time step. The image display device 1 can notify the remote operator R if the delay is small, but can notify the remote operator if the delay is large. The image display device 1 can notify the remote operator R of the degree of delay using a guide line object and an object extending in the direction of the line width.

[0084] While embodiments of this disclosure have been described above, this disclosure is not limited to the embodiments described above. This disclosure can be implemented in various forms, including the embodiments described above, with various modifications and improvements based on the knowledge of those skilled in the art.

[0085] [Differentiation] [Display of other vehicles] The image display device 1 may display information about other vehicles present around vehicle 2 on the display device 30. The image display device 1 recognizes other vehicles based on external sensor data detected by the external sensor 22 and displays the assist information displayed on vehicle 2 to the other vehicles.

[0086] Figure 8(A) is an example of a monitoring image with guideline objects superimposed on it for other vehicles. The monitoring image G5 in Figure 8(A) is an overhead view that includes vehicle object OV1 representing vehicle 2, as well as vehicle objects OV2 and OV3 representing other vehicles in the vicinity of vehicle 2. The assist information generation unit 13 generates, as an example, the positions of other vehicles at three second time points: 1 second, 2 seconds, and 3 seconds after the current time. The display control unit 14 superimposes the respective guideline objects onto the monitoring image G5 based on the positions of other vehicles at the three second time points determined by the assist information generation unit 13. For example, the display control unit 14 superimposes a line object surrounding another vehicle on the monitoring image G5 to show the position of the other vehicle at 1 second later. The display control unit 14 can superimpose an object showing information related to the second time point (e.g., elapsed time from the current time) at the screen position corresponding to the line object. Similarly, the display control unit 14 superimposes a line object showing the position of the other vehicle at 2 seconds later and the corresponding object related to the second time point. The display control unit 14 overlays a line object indicating the position of other vehicles 3 seconds later and an object related to the corresponding second time point.

[0087] In Figure 8(A), assist information is displayed for all other vehicles surrounding vehicle 2, but assist information may be displayed only for specific other vehicles. Also, in Figure 8(A), objects extending in the line width direction of the line object may be displayed to show the information while taking delay time into consideration.

[0088] Figure 8(B) is an example of a monitoring image in which a guideline object and a delay indicator object are superimposed for a specific other vehicle. In monitoring image G6 of Figure 8(B), vehicle object OV2 is selected as the vehicle to be displayed, and assist information is displayed only for vehicle object OV2. The image display device 1 selects the vehicle to be displayed by executing the flowchart in Figure 9 below.

[0089] Figure 9 is a flowchart showing an example of the decision process for an image display device. The flowchart shown in Figure 9 starts when an instruction to start the display function of the image display device 1 is given.

[0090] First, as part of the data acquisition process (S40), the external sensor data receiving unit 110 acquires the external sensor data detected at the first time by the external sensor 22 from the vehicle 2.

[0091] Next, in the evaluation value calculation process (S42), the image display device 1 recognizes the position of other vehicles based on external sensor data. Then, the image display device 1 calculates an evaluation value for each recognized other vehicle. The evaluation value is an indicator of safety, and as an example, it is a value that indicates whether or not it has a significant impact on the behavior of vehicle 2. The larger the evaluation value, the greater the impact on the behavior of vehicle 2. In other words, the larger the evaluation value, the lower the safety. For example, the image display device 1 calculates the evaluation value by weighting the relative speed, relative yaw angle, relative distance, etc., between vehicle 2 and the target other vehicle. The weighted sum may include a term corresponding to the type of vehicle of the target other vehicle.

[0092] Next, as part of the display target determination process (S44), the image display device 1 determines which other vehicles to display based on the evaluation value calculated for each other vehicle. The image display device 1 selects the other vehicle with the highest evaluation value.

[0093] When the display target determination process (S44) is completed, the flowchart shown in Figure 9 ends. After the flowchart shown in Figure 9 ends, assist information is generated and displayed for the other vehicles to be displayed. The generation and display of assist information is the same as in the flowcharts of Figures 6 and 7. By executing the flowchart shown in Figure 9, the image display device 1 can select a vehicle with low safety from among multiple other vehicles and display assist information for it. After the flowchart ends, the image display device 1 restarts the flowchart from the beginning until an instruction to terminate the display function is given.

[0094] [Example of display] In this embodiment, the display control unit 14 displays an overhead image on the display device 30, but it may also display the camera image itself. Figure 10(A) is another example of a monitoring image with guideline objects superimposed. As shown in Figure 10(A), the monitoring image G7 is a camera image with line objects OL1 to OL4 indicating the future position of the vehicle 2 and objects M1 to M4 related to the corresponding time superimposed. Figure 10(B) is another example of a monitoring image with guideline objects and objects indicating delay superimposed. As shown in Figure 10(B), the monitoring image G8 is a camera image with line objects OL1 to OL4 indicating the future position of the vehicle 2, objects M1 to M4 related to the corresponding time, and objects B1 to B4 extending in the line width direction of the line objects OL1 to OL according to the delay time superimposed. In this way, the image display device 1 may superimpose assist information onto the camera image.

[0095] In the modified example relating to the display of other vehicles, the display control unit 14 displayed a line object indicating the future position of the other vehicle, but is not limited to this. For example, the display control unit 14 may display a line object indicating the TTC between vehicle 2 and the target other vehicle.

[0096] [Variations in the structure] Vehicle 2 can be any vehicle capable of remote control and is not limited to autonomous vehicles. Vehicle 2 does not need to have a map database. If the display of delay time is omitted, the image display device 1 does not need to have a time receiving unit 112 and a delay time calculation unit 12. Also, the image display device 1 may use time obtained from an NTP (Network Time Protocol) server instead of GPS time. [Explanation of Symbols]

[0097] 1…Image display device, 2…Vehicle, 14…Display control unit, 22…External sensor, 23…Internal sensor, R…Remote operator.

Claims

1. An image display device connected to a display unit that displays information to a remote operator who remotely controls a vehicle, An external sensor data acquisition unit acquires images captured by a camera that captures the area in front of the vehicle via communication from the vehicle, An internal sensor data acquisition unit acquires internal sensor data detected by an internal sensor that detects the driving status of the vehicle from the vehicle via communication. A display control unit, based on the internal sensor data, combines an object indicating the destination position, which is the position the vehicle will reach in the future when the vehicle is in motion, and an object indicating the range of possible destination positions calculated based on the delay time of communication with the vehicle, with the captured image and displays them on the display unit. An image display device equipped with the following features.

2. The image display device according to claim 1, wherein the display control unit displays an object indicating the destination and an object indicating the range of possible destinations, together with a strip-shaped object extending in front of the vehicle.

3. The image display device according to claim 1 or 2, wherein the display control unit causes an object indicating the destination and an object indicating the range of possible destinations to be along at least the route that the vehicle can travel.

4. The image display device according to claim 3, wherein the display control unit causes an object indicating the range of possible destinations to extend along the path.

5. The display control unit, As an object indicating the destination, a guide line object that forms a rectangle in plan view is displayed. The image display device according to any one of claims 1 to 4, wherein an object extending in the line width direction of the guide line object is displayed as an object indicating the range of possible destination positions.

6. The image display device according to claim 5, wherein the display control unit changes the length in the line width direction of the object indicating the range of possible destination positions to be increased as the delay time increases.

7. The image display device according to any one of claims 1 to 6, wherein the display control unit superimposes an object indicating the range of possible destinations on the image corresponding to the range of possible destinations when the delay time is greater than or equal to a threshold.

8. The image display device according to any one of claims 1 to 6, wherein the display control unit superimposes an object indicating the range of possible destinations on the image corresponding to the range of possible destinations when the delay degree, defined based on the absolute value and variance of the delay time, is greater than or equal to a threshold.

9. A determination unit that detects other vehicles present in the vicinity of the vehicle based on data at a first time obtained by the external sensor data acquisition unit, and determines the position of the other vehicles at a future time after a predetermined time has elapsed from the first time, The image display device according to any one of claims 1 to 8, wherein the display control unit overlays an object indicating the position of another vehicle at the future time determined by the determination unit onto the captured image.

Images