Remote Indicator Device, Display Control Method, Display Control Program

Abstract

To provide a technique that enables an operator to drive or assist a vehicle on the assumption that a communication is interrupted while suppressing a decline in usability regarding a technology for controlling display of a display device installed on a remote instruction device which communicates with the vehicle so as to drive or assist the vehicle remotely.SOLUTION: A remote instruction device according to the present disclosure is configured to perform processing for: displaying a travel video on a display device; predicting an emergency stop position of a vehicle when communication with the vehicle is interrupted; and displaying an object indicating the predicted emergency stop position on the display device by superimposing the travel video thereon. Here, the processing for displaying the object indicating the predicted emergency stop position on the display device by superimposing the travel video thereon includes: calculating a degree of reliability of communication with the vehicle; and changing an external appearance of the object indicating the emergency stop position such that the higher the degree of reliability, the lower the attractiveness.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a technique for controlling the display of a display device provided in a remote instruction device for remotely driving or assisting a vehicle.

Background Art

[0002] In recent years, a technique of superimposing various information on an image related to a vehicle displayed on a display device provided in a remote instruction device has been considered.

[0003] For example, Patent Document 1 discloses a display device including a determination unit that determines the position of a vehicle at a future time, a display unit that displays a monitoring image showing the periphery of the vehicle, and a display control unit that superimposes an object showing the determined position of the vehicle at the future time on the monitoring image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, when the communication between the vehicle and the remote instruction device is interrupted while the vehicle is being driven or assisted by the remote instruction device, generally the vehicle makes an emergency stop to ensure safety. However, depending on the driving environment of the vehicle, there is a risk that it may not be possible to make a safe emergency stop.

[0006] Therefore, the inventors of the present disclosure are considering superimposing and displaying information that enables an operator to drive or assist a vehicle assuming a situation where communication has been interrupted on the driving video. Considering that this information is related to the safety of the vehicle, it is desirable that the information be displayed sufficiently and clearly. However, if the information is always displayed sufficiently and clearly, the operator has to be aware of the information and drive or assist the vehicle even in a situation where the possibility of communication interruption is low. Consequently, there is a risk that the usability of the remote instruction device will deteriorate, such as neglecting other safety checks.

[0007] One object of the present disclosure is to provide a technology that enables an operator to drive or assist a vehicle assuming a situation where communication has been interrupted while suppressing a deterioration in usability in view of the above problems.

Means for Solving the Problems

[0008] The first disclosure relates to a remote instruction device that communicates with a vehicle to remotely drive or assist the vehicle.

[0009] The remote instruction device according to the first disclosure includes a display device and one or more processors. The one or more processors are configured to execute a process of acquiring a driving video and driving state information of the vehicle, a process of displaying the driving video on the display device, a process of predicting an emergency stop position of the vehicle when communication with the vehicle is interrupted based on the driving state information, and a process of superimposing an object indicating the predicted emergency stop position on the driving video and displaying it on the display device. The process of superimposing an object indicating the emergency stop position on the driving video and displaying it on the display device includes calculating the reliability of communication with the vehicle and changing the appearance of the object indicating the emergency stop position so that the higher the reliability, the lower the conspicuity.

[0010] The second disclosure relates to a display control method for controlling the display of a display device provided in a remote instruction device that communicates with a vehicle to remotely drive or assist the vehicle.

[0011] The display control method according to the second disclosure includes obtaining a running video of a vehicle and running state information, displaying the running video on a display device, predicting an emergency stop position of the vehicle when communication with the vehicle is interrupted based on the running state information, and superimposing an object indicating the predicted emergency stop position on the running video and displaying it on the display device. Displaying an object indicating the emergency stop position by superimposing it on the running video on the display device includes calculating the reliability of communication with the vehicle and changing the appearance of the object indicating the emergency stop position so that the higher the reliability, the lower the conspicuity.

[0012] The third disclosure relates to a display control program that causes a computer to execute control of the display of a display device provided in a remote instruction device that communicates with a vehicle to remotely drive or assist the vehicle.

[0013] The display control program according to the third disclosure is configured to cause a computer to execute a process of obtaining a running video of a vehicle and running state information, a process of displaying the running video on a display device, a process of predicting an emergency stop position of the vehicle when communication with the vehicle is interrupted based on the running state information, and a process of superimposing an object indicating the predicted emergency stop position on the running video and displaying it on the display device. The process of superimposing an object indicating the emergency stop position on the running video and displaying it on the display device includes calculating the reliability of communication with the vehicle and changing the appearance of the object indicating the emergency stop position so that the higher the reliability, the lower the conspicuity.

Advantages of the Invention

[0014] According to the present disclosure, an object indicating an emergency stop position of a vehicle when communication with the vehicle is interrupted is superimposed on a driving video and displayed on a display device. Thereby, the operator can drive or support the vehicle while ensuring safety assuming the case where communication is interrupted. Further, according to the present disclosure, the appearance of the object indicating the emergency stop position is changed so that the higher the communication reliability, the lower the visibility. Thereby, when the communication reliability can be sufficiently ensured and the possibility of communication interruption is low, the operator's excessive awareness of the object indicating the emergency stop position can be suppressed. Consequently, it is possible to suppress a decrease in usability due to displaying the object indicating the emergency stop position on the driving video.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0016] Hereinafter, the present embodiment will be described with reference to the drawings.

[0017] 1. Configuration FIG. 1 is a block diagram showing the configuration of a remote instruction system 10 according to the present embodiment. The remote instruction system 10 according to the present embodiment provides a function of remotely driving or remotely supporting a vehicle 200 (hereinafter simply referred to as a “remote instruction function”).

[0018] Note that the remote driving function typically refers to a function of remotely performing at least one of the operations related to the running, turning, and stopping of the vehicle 200. For example, the remote instruction system 10 provides a function of remotely performing the accelerator, brake, and steering operations of the vehicle 200. In this case, the vehicle 200 may be a vehicle that runs only by remote driving, or may be a manual driving vehicle or an autonomous driving vehicle that can run by remote driving as needed. Also, the remote assistance function typically refers to a function of providing information related to driving judgment and recognition judgment to the vehicle 200. For example, the remote instruction system 10 provides a function of receiving a remote assistance request related to the permission of progress or recognition assistance from the vehicle 200 and accepting the judgment by the operator. In this case, the vehicle 200 is typically an autonomous driving vehicle that issues a remote assistance request as needed.

[0019] The remote instruction system 10 includes a remote instruction device 100. The remote instruction device 100 communicates with the vehicle 200 via the communication network 300. Here, the communication network 300 is composed of, for example, a mobile communication network composed of a plurality of base stations and the Internet. The operator of the vehicle 200 can remotely drive or assist the vehicle 200 by operating the remote instruction device 100. Hereinafter, the configurations of the remote instruction device 100 and the vehicle 200 will be described.

[0020] The remote instruction device 100 includes a control device 110, an HMI 120, and a communication device 130.

[0021] The HMI 120 is a device that provides HMI functions to the operator of the vehicle 200. The HMI 120 includes a display device 121, a speaker 122, and an operation input device 123. Here, the operation input device 123 is a device that is operated by the operator and receives various inputs. For example, the operation input device 123 related to the function of remotely driving is a device that receives the driving operation by the operator. In this case, examples of the operation input device 123 include an accelerator pedal, a brake pedal, a steering wheel, etc. Also, the operation input device 123 related to the function of remotely assisting is a device that receives the judgment by the operator. In this case, examples of the operation input device 123 include an operation panel, a keyboard, a switch, etc.

[0022] In the remote instruction device 100, it is assumed that the operator recognizes the driving environment of the vehicle 200 from the display of the display device 121 and the sound of the speaker 122, and operates the operation input device 123.

[0023] The control device 110 is connected so as to be able to transmit and receive information to and from the HMI 120 and the communication device 130. For example, the control device 110 is electrically connected to these devices by a cable.

[0024] The control device 110 is a computer that executes processing related to the remote instruction function. By executing the processing, the control device 110 transmits information on various inputs received by the operation input device 123 (hereinafter simply referred to as "operation input information") to the vehicle 200 via the communication device 130. Also, the control device 110 controls the display device 121 and the speaker 122 based on the information related to the driving environment (hereinafter simply referred to as "driving environment information") received from the vehicle 200 via the communication device 130.

[0025] The communication device 130 communicates with devices external to the remote instruction device 100 to transmit and receive information. At least, the communication device 130 includes a device that communicates with the vehicle 200 via the communication network 300. For example, the communication device 130 includes a device that connects to the Internet to transmit and receive information.

[0026] Vehicle 200 includes an ECU (Electronic Control Unit) 210, sensors 220, a communication device 230, and an actuator 240.

[0027] The sensors 220 are a type of sensors that detect the driving environment of the vehicle 200. The sensors 220 include a camera 221, a driving state detection sensor 222, and a GNSS sensor 223.

[0028] The camera 221 images the surrounding environment of the vehicle 200 including at least the front, and outputs a driving video. The driving state detection sensor 222 detects the driving state (vehicle speed, acceleration, yaw rate, etc.) of the vehicle 200, and outputs driving state information. Examples of the driving state detection sensor 222 include a wheel speed sensor, an acceleration sensor, a gyro sensor, an IMU (Internal Measurement Unit), etc. The GNSS (Global Navigation Satellite System) sensor 223 measures the position of the vehicle 200 by receiving signals from satellites, and outputs GNSS position information.

[0029] The ECU 210 is connected so as to be able to communicate with the sensors 220, the communication device 230, and the actuator. Typically, the ECU 210 is connected to these devices via a vehicle-mounted network composed of CAN (Control Area Network) or the like.

[0030] The ECU 210 transmits the detection information of the sensors 220 as driving environment information to the remote instruction device 100 via the communication device 230.

[0031] The ECU 210 also executes processes related to the control of the vehicle 200 and generates control signals. At least, the ECU 210 executes a process of controlling the vehicle 200 according to the operation input information received from the remote instruction device 100 via the communication device 230. For example, regarding the function of remote driving, the ECU 210 generates control signals related to acceleration, braking, and steering according to the operation amounts (such as accelerator opening, brake pedal depression amount, steering angle, etc.) of the driving operations received as operation input information. Also, for example, regarding the function of remote assistance, the ECU 210 performs automatic driving of the vehicle 200 according to the operator's judgment received as operation input information.

[0032] Furthermore, the ECU 210 is configured to execute a process of causing the vehicle 200 to make an emergency stop when the communication with the remote instruction device 100 is interrupted while the vehicle 200 is being driven or assisted by the remote instruction device 100. For example, when the communication with the remote instruction device 100 is interrupted, the ECU 210 causes the vehicle 200 to make an emergency stop at a predetermined deceleration.

[0033] The communication device 230 communicates with devices external to the vehicle 200 to transmit and receive information. At least, the communication device 230 includes a device that communicates with the remote instruction device 100 via the communication network 300. For example, the communication device 230 includes a device that communicates with a base station located around the vehicle 200 to transmit and receive information.

[0034] The actuator 240 operates according to the control signal generated by the ECU 210. By operating according to the control signal obtained by the actuator 240 from the ECU 210, the driving or assistance of the vehicle 200 by the remote instruction device 100 is realized.

[0035] Hereinafter, with reference to FIG. 2, the configuration of the control device 110 provided in the remote instruction device 100 will be described. FIG. 2 is a block diagram showing the configuration of the control device 110.

[0036] The control device 110 includes one or more processors 111 (hereinafter simply referred to as "processor 111") and one or more storage devices 112 (hereinafter simply referred to as "storage device 112"). The processor 111 executes various processes. The processor 111 can be composed of a CPU (Central Processing Unit) including, for example, an arithmetic unit, registers, and the like. The storage device 112 is coupled to the processor 111 and stores various information necessary for the execution of the processes of the processor 111. The storage device 112 is, for example, a recording medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive).

[0037] The storage device 112 stores a plurality of instructions 114, driving environment information D10, operation input information D20, and communication state information D30.

[0038] The plurality of instructions 114 are provided by a computer program 113. By operating the processor 111 in accordance with the plurality of instructions 114, various processes by the control device 110 are realized. Note that the computer program 113 may be recorded on a computer-readable recording medium.

[0039] The driving environment information D10 is received from the vehicle 200. The operation input information D20 is acquired from the operation input device 123. The driving environment information D10 includes a driving video D11, driving state information D12, and GNSS position information D13. Further, the driving environment information D10 may include a predetermined deceleration when the vehicle 200 is to be emergently stopped, specification information (installation position, installation angle, field of view angle, etc.) of the camera 221 provided in the vehicle 200, and the like.

[0040] The communication status information D30 is information indicating the communication status between the remote instruction device 100 and the vehicle 200. Examples of the communication status information D30 include the throughput and latency of communication measured at any time, map information in which the communication status (base station installation status and communication intensity for each communication carrier) at each position on the map is associated, maintenance and failure information for each communication carrier, and the like. The communication status information D30 is obtained, for example, by executing the processing of the processor 111. Alternatively, the remote instruction device 100 may receive it from various servers via the communication network 300.

[0041] 2. Stop position display processing The control device 110 (more specifically, the processor 111) executes a process of displaying at least the traveling video D11 on the display device 121 in the process related to the control of the display device 121. Further, the processor 111 executes a process (stop position display process) of superimposing an object indicating the emergency stop position of the vehicle 200 when the communication with the vehicle 200 is interrupted on the traveling video D11 and displaying it on the display device 121.

[0042] Hereinafter, the stop position display process executed by the processor 111 will be described.

[0043] First, the processor 111 predicts the emergency stop position of the vehicle 200 when the communication with the vehicle 200 is interrupted based on the traveling state information D12. For example, the processor 111 predicts the emergency stop position of the vehicle 200 from the vehicle speed and steering angle of the vehicle 200 and a predetermined deceleration when the vehicle 200 is emergently stopped. The emergency stop position is predicted, for example, as a relative position from the vehicle 200 in a spatial coordinate system.

[0044] Next, the processor 111 superimposes an object indicating the predicted emergency stop position on the traveling video D11 and displays it on the display device 121. For example, the processor 111 converts the predicted emergency stop position based on the specifications information of the camera 221 into a position on the traveling video D11 and displays a predetermined object at that position.

[0045] FIG. 3 shows an example of the display of the display device 121 when the processor 111 executes the stop position display process in the present embodiment. As shown in FIG. 3, an object 1 (hereinafter simply referred to as "object 1") indicating the emergency stop position of the vehicle 200 is superimposed on the driving video D11 and displayed on the display device 121. Such a display is one type of AR (Augmented Reality) display. By superimposing the object 1 on the driving video D11 in this way, the operator of the vehicle 200 can drive or assist the vehicle 200 while ensuring safety assuming a situation where communication is interrupted. For example, the operator can drive or assist the vehicle 200 so that the object 1 does not deviate from the road.

[0046] In FIG. 3, the object 1 is composed of a plurality of lines, but other forms may be adopted. For example, the object 1 may be a circular or rectangular area, a three-dimensional shape imitating the vehicle 200, or the like. In the following description, it is assumed that the object 1 is composed of a plurality of lines.

[0047] In the stop position display process, the processor 111 further calculates the reliability of communication (hereinafter simply referred to as "communication reliability") between the remote instruction device 100 and the vehicle 200. Then, the processor 111 changes the appearance of the object 1 so that the higher the calculated communication reliability, the lower the conspicuity.

[0048] Here, the calculation of communication reliability can be performed as follows based on the communication state information D30. One is to use the throughput and latency of the communication measured at any time as indicators. For example, the processor 111 is configured to calculate a higher communication reliability as the throughput of the communication is larger and the latency is smaller. The other is to utilize map information associated with the communication state. For example, the processor 111 is configured to calculate the communication reliability by identifying the position of the vehicle 200 from the GNSS position information D13 and obtaining the communication state of the position from the map information. In addition, the processor 111 may be configured to correct the communication reliability based on the time zone, maintenance for each communication carrier, failure information, etc.

[0049] Also, changing the appearance of object 1 so that the higher the communication reliability, the lower the conspicuity can be done, for example, as shown in FIG. 4. FIG. 4 shows three patterns (A), (B), and (C) regarding the change in the appearance of object 1 according to the communication reliability.

[0050] FIG. 4(A) is the case of changing the color of object 1 according to the communication reliability. It is known that the conspicuity of colors is high in the order of yellow, red, blue, purple, and violet when the background is gray. Also, object 1 is usually displayed on the road portion of the driving video D11, and the road portion generally has a strong gray color tone. Therefore, by changing the color of object 1 as shown in FIG. 4(A), the higher the communication reliability, the lower the conspicuity can be made. Note that the processor 111 may be configured to change the color pattern for the communication reliability according to the color of the display portion of object 1 in the driving video D11.

[0051] FIG. 4(B) is the case of changing the line thickness of object 1 according to the communication reliability. As shown in FIG. 4(B), by making the line thickness thinner as the communication reliability is higher, the appearance of object 1 can be made less conspicuous as the communication reliability is higher.

[0052] (C) in FIG. 4 is a case where the line type of object 1 is changed according to the communication reliability. Generally, the more solid line parts a line has, the higher its visibility. Therefore, by changing the line type of object 1 as shown in (C) of FIG. 4, the higher the communication reliability, the lower the visibility can be made.

[0053] Note that these changes according to the reliability may be combined. For example, the processor 111 may be configured to change both the thickness and the line type of the line of object 1 according to the reliability. FIG. 5 shows Example (A) when the communication reliability is low and Example (B) when the communication reliability is high in this case.

[0054] By changing the appearance of object 1 in this way so that the lower the communication reliability, the higher the visibility, when the communication reliability is sufficiently ensured and the possibility of communication interruption is low, the excessive awareness of the operator regarding object 1 can be suppressed. Subsequently, the decrease in usability due to displaying object 1 in the driving video D11 can be suppressed. On the other hand, when the communication reliability is low and there is a risk of communication interruption, object 1 can be displayed sufficiently and clearly in the driving video D11. Subsequently, when there is a risk of communication interruption, the operator can be prompted to be aware of object 1 for driving or assistance, and the safety of the remote instruction function can be improved.

[0055] Note that making the emergency stop position completely unrecognizable is not desirable from the perspective of ensuring safety. Therefore, even if the communication reliability is sufficiently high, it is desirable not to erase the display of object 1.

[0056] FIG. 6 is a flowchart showing an example of the stop position display process executed by the processor 111. The flowchart shown in FIG. 6 may be repeatedly executed at a predetermined cycle.

[0057] In step S100, the processor 111 acquires various information stored in the storage device 112.

[0058] In step S200, based on the driving state information D12, the processor 111 predicts the emergency stop position of the vehicle 200 when the communication between the remote instruction device 100 and the vehicle 200 is interrupted.

[0059] In step S300, based on the communication state information D30, the processor 111 calculates the reliability of the communication (communication reliability) between the remote instruction device 100 and the vehicle 200.

[0060] In step S400, the processor 111 determines the appearance of object 1 according to the communication reliability calculated in step S300. Here, the processor 111 determines the appearance of object 1 such that the higher the communication reliability, the lower the attractiveness.

[0061] In step S500, the processor 111 superimposes object 1 indicating the emergency stop position predicted in step S200 on the driving video D11 with the appearance determined in step S400 and displays it on the display device 121.

[0062] In this way, the processor 111 executes the stop position display process. Also, by the processor 111 executing the stop position display process in this way, a display control method for controlling the display of the display device 121 is realized.

Explanation of Signs

[0063] 1 Object indicating the emergency stop position, 10 Remote instruction system, 100 Remote instruction device, 110 Control device, 111 Processor, 112 Storage device, 113 Computer program, 121 Display device, 200 Vehicle, D11 Driving video, D12 Driving state information

Claims

1. A remote instruction device that communicates with a vehicle to remotely drive or assist the vehicle, comprising: a display device; one or more processors; and is configured to: The one or more processors are configured to: acquire the running video and running state information of the vehicle; display the running video on the display device; predict the emergency stop position of the vehicle when communication with the vehicle is interrupted based on the running state information; superimpose an object indicating the predicted emergency stop position on the running video and display it on the display device; and are configured to execute the above operations. The process of superimposing the object on the running video and displaying it on the display device includes: calculating the reliability of communication with the vehicle; changing the appearance of the object so that the higher the reliability, the lower the visibility; and includes the above operations. A remote instruction device characterized by the above.

2. The remote instruction device according to Claim 1, wherein changing the appearance of the object so that the higher the reliability, the lower the visibility includes changing the color of the object to a color with lower visibility with respect to the display portion of the object in the running video as the reliability increases. A remote instruction device characterized by the above.

3. The remote instruction device according to Claim 1 or 2, wherein the object is composed of a plurality of lines, and changing the appearance of the object so that the higher the reliability, the lower the visibility includes at least one of making the plurality of lines thinner as the reliability increases and changing the plurality of lines to a line type with lower visibility as the reliability increases. A remote instruction device characterized by the above.

4. A display control method for controlling the display of a display device provided in a remote instruction device that communicates with a vehicle to remotely drive or assist the vehicle, the method including: acquiring the running video and running state information of the vehicle; displaying the running video on the display device; predicting the emergency stop position of the vehicle when communication with the vehicle is interrupted based on the running state information; superimposing an object indicating the predicted emergency stop position on the running video and displaying it on the display device; and includes the above operations. The operation of superimposing the object on the running video and displaying it on the display device includes: calculating the reliability of communication with the vehicle; changing the appearance of the object so that the higher the reliability, the lower the visibility; and includes the above operations. A display control method characterized by the above.

5. A display control program that causes a computer to execute control of the display of a display device provided in a remote instruction device that communicates with a vehicle to remotely drive or assist the vehicle, comprising: a process of acquiring a running video and running state information of the vehicle; a process of displaying the running video on the display device; a process of predicting an emergency stop position of the vehicle when communication with the vehicle is interrupted based on the running state information; a process of superimposing an object indicating the predicted emergency stop position on the running video and displaying the same on the display device; configured to cause the computer to execute; the process of superimposing the object on the running video and displaying the same on the display device includes: calculating a reliability of communication with the vehicle; changing an appearance of the object so that the higher the reliability, the lower the visibility; including A display control program characterized by the above.

Images