Remote Control System

Abstract

To provide a remote operation system for enabling an occupant of a vehicle to obtain even the intention of a remote operator of a remote operation device.SOLUTION: In a remote operation system, a peripheral image of a vehicle photographed by a camera of the vehicle is displayed on a display of a remote operation device. A viewpoint detection section detects in which place of the peripheral image the viewpoint of a remote operator is present. A viewpoint sign 90 of the remote operator is displayed by overlapping with the peripheral image 70 (operation image 80) on a display of the vehicle. Thus, the occupant of the vehicle can confirm in which place of the peripheral image 70 the viewpoint of the remote operator is present by viewing the display, so as to confirm the intention of the operation of the remote operator.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present disclosure relates to a remote control system. [Background technology]

[0002] In a remote control system in which a vehicle is driven by an operator operating a remote control device installed outside the vehicle, since the vehicle operator is outside the vehicle, there is a risk that vehicle occupants may feel uneasy about driving the vehicle.

[0003] Several proposals have been made to alleviate such passenger anxiety.

[0004] For example, Patent Document 1 proposes displaying an image of an operator operating a remote control device on a monitor of the vehicle, enabling the vehicle occupant to visually confirm the operator while operating the device, thereby eliminating the above-mentioned concerns.

[0005] Also, Patent Document 2 proposes a configuration in which the operator's operation information is superimposed on an image that is substantially the same as the image of the vehicle's surroundings that is displayed on the remote control device for the operator and displayed on the vehicle monitor. This allows the vehicle occupant to understand the operation content of the operator, thereby eliminating the above-mentioned anxiety. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2017-158149 A [Patent Document 2] JP 2020-166605 A Summary of the Invention [Problem to be solved by the invention]

[0007] In the technology described in Patent Document 1, although the vehicle occupants can visually recognize the operator, it is difficult for them to predict the operation content of the operator, i.e., the behavior of the vehicle, and there is room for improvement in alleviating the anxiety of the occupants.

[0008] In addition, in the technology described in Patent Document 2, the vehicle occupant can understand the operation contents (vehicle behavior) of the operator, but it is difficult to understand the intention of the operator regarding the operation contents. In other words, there is room for improvement in terms of eliminating the anxiety of the occupant.

[0009] The present disclosure has been made in consideration of the above-mentioned circumstances, and has an object to provide a remote control system that allows a vehicle occupant to understand the intention of a remote operator of a remote control device. [Means for solving the problem]

[0010] The remote control system of the present disclosure is a remote control system in which a remote operator drives a vehicle by operating a remote control device provided outside the vehicle, and includes an imaging unit provided in the vehicle and capturing an image of the vehicle's surroundings, a first display unit provided in the remote control device and displaying the surrounding image transmitted from the vehicle, a viewpoint detection unit that detects the viewpoint of the remote operator relative to the surrounding image displayed on the first display unit, and a second display unit provided in the vehicle and displaying a viewpoint symbol indicating the viewpoint superimposed on the surrounding image.

[0011] In the present disclosure, in a remote control system, an image of the vehicle's surroundings captured by an imaging unit of the vehicle is displayed on a first display unit of the remote control device, and a viewpoint detection unit detects where in the surrounding image the viewpoint of the remote operator of the remote control device is located (information regarding the viewpoint), and a viewpoint symbol indicating the viewpoint of the remote operator is superimposed on the surrounding image and displayed on a second display unit of the vehicle.

[0012] Therefore, by looking at the second display unit of the vehicle, the vehicle occupant can confirm where the remote operator's viewpoint (viewpoint symbol) is in the surroundings (image) of the vehicle, and can confirm the remote operator's intention of operation. Therefore, it is possible to suppress the anxiety of the vehicle occupant due to the fact that the remote operator (driver) is not on board. Effect of the Invention

[0013] The present disclosure has an effect of allowing a vehicle occupant to understand the operation intention of a remote operator of a remote control device. [Brief description of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing a schematic configuration of a vehicle control system configured to include a vehicle according to the present disclosure. [Diagram 2] FIG. 2 is a block diagram showing a hardware configuration of a vehicle. [Diagram 3] 2 is a block diagram showing an example of a functional configuration of a vehicle control device provided in a vehicle. FIG. [Figure 4] FIG. 2 is a block diagram showing a hardware configuration of the remote control device. [Diagram 5] 2 is a block diagram showing an example of a functional configuration of a remote control device included in the remote operation device. FIG. [Figure 6] 4 is a flowchart showing a flow of processing executed by a vehicle control device. [Figure 7] 4 is a flowchart showing a flow of processing executed by a vehicle control device. [Figure 8] 5 is a flowchart showing a flow of processing executed by the remote control device. [Figure 9] 5 is a flowchart showing a flow of processing executed by the remote control device. [Figure 10] 4A and 4B are diagrams illustrating operation images displayed on the remote control device when the vehicle is traveling straight. [Figure 11] 1 is a diagram showing a schematic diagram of a guide image displayed on a vehicle when the vehicle is traveling straight ahead; [Figure 12]1 is a diagram showing a schematic diagram of a guide image displayed on a vehicle when the vehicle is traveling straight and increasing in speed; [Figure 13] 10 is a diagram showing a schematic diagram of an operation image displayed on the remote control device when turning left; FIG. [Figure 14] FIG. 2 is a diagram showing a schematic diagram of a guide image displayed on a vehicle when making a left turn; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, a remote control system 10 according to an embodiment of the present invention will be described with reference to FIGS.

[0016] As shown in FIG. 1, a remote control system 10 according to this embodiment includes a vehicle 12 and a remote control device 14.

[0017] The vehicle 12 is equipped with a vehicle control device 20. The remote operation device 14 is configured to include a remote control device 40. In the above remote operation system 10, the vehicle control device 20 of the vehicle 12 and the remote control device 40 of the remote operation device 14 are connected to each other so as to be able to communicate with each other via a network N1. This network N1 is, for example, a wired and wireless communication network using a public line such as the Internet.

[0018] (Vehicle configuration)

[0019] The vehicle 12 is configured to be capable of performing manual driving based on driving operations by a driver who is an occupant of the vehicle 12, and remote driving based on driving operations by a remote operator (remote driver) using the remote control device 14. The vehicle 12 is normally set in a manual driving mode.

[0020] 2 is a block diagram showing a hardware configuration of devices mounted on the vehicle 12. In addition to the vehicle control device 20 described above, the vehicle 12 includes an external sensor 24, an internal sensor 26, an input device 28, an actuator 30, and a user I / F (Interface) 32.

[0021] 2, the vehicle control device 20 includes a CPU (Central Processing Unit) 20A, a ROM (Read Only Memory) 20B, a RAM (Random Access Memory) 20C, a storage 20D, a communication I / F (Interface) 20E, and an input / output I / F 20F. The CPU 20A, the ROM 20B, the RAM 20C, the storage 20D, the communication I / F 20E, and the input / output I / F 20F are connected to each other so as to be able to communicate with each other via a bus 20G.

[0022] The CPU 20A is a central processing unit, and executes various programs and controls each part. That is, the CPU 20A reads out the programs from the ROM 20B and executes the programs using the RAM 20C as a working area. In this embodiment, the programs are stored in the ROM 20B. By the CPU 20A executing the programs, the vehicle control device 20 functions as a surrounding information acquisition unit 200, a behavior information acquisition unit 210, an operation reception unit 220, a driving control unit 230, an authorization unit 240, a transmission unit 250, a reception unit 260, and a guidance image generation unit 270 shown in FIG. 3.

[0023] As shown in Fig. 2, the ROM 20B stores various programs and various data. The RAM 20C temporarily stores programs or data as a working area. The storage 20D is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including an operating system and various data. The communication I / F 20E includes an interface for connecting to the network N1 to communicate with the remote control device 40. For example, a communication standard such as LTE or Wi-Fi (registered trademark) is used as the interface.

[0024] The communication I / F 20E of this embodiment transmits an image captured by the camera 24A (a surrounding image 70, described later) to a remote control device 14 outside the vehicle 12, and receives an operation image 80 and viewpoint information, described later, from the remote control device 14. The communication I / F 20E also receives remote operation information, which is operation information for operating the vehicle 12, from the remote control device 14, via the network N1. The camera 24A is equipped with a microphone (not shown), and sound data recorded together with the captured image is transmitted and received together with the captured image data.

[0025] The input / output I / F 20F is an interface for communicating with each device mounted on the vehicle 12. The vehicle control device 20 of this embodiment is connected to an external sensor 24, an internal sensor 26, an input device 28, an actuator 30, and a user I / F 32 via the input / output I / F 20F. Note that the external sensor 24, the internal sensor 26, the input device 28, the actuator 30, and the user I / F 32 may be directly connected to the bus 20G.

[0026] The external sensor 24 is a group of sensors that detects peripheral information about the periphery of the vehicle 12. The external sensor 24 includes a camera 24A that captures an image of a predetermined range. The camera 24A corresponds to an "imaging unit."

[0027] The internal sensors 26 are a group of sensors that detect the traveling state of the vehicle 12. The internal sensors 26 include a steering angle sensor 26A that detects the amount of operation of the steering wheel, and a vehicle speed sensor 26B that detects the speed of the vehicle 12 (vehicle speed).

[0028] The input device 28 is a group of switches to be operated by a passenger in the vehicle 12. The input device 28 includes a steering wheel 28A as a switch for steering the wheels of the vehicle 12, an accelerator pedal 28B as a switch for accelerating the vehicle 12, and a brake pedal 28C as a switch for decelerating the vehicle 12.

[0029] The actuators 30 include a steering wheel actuator that drives the steered wheels of the vehicle 12 , an accelerator actuator that controls the acceleration of the vehicle 12 , and a brake actuator that controls the deceleration of the vehicle 12 .

[0030] The user I / F 32 includes a display 32A and a speaker (not shown) which is an audio output unit. The display 32A is disposed, for example, on an instrument panel of the vehicle 12. The speakers are disposed, for example, on each of the left and right side doors of the vehicle 12. The display 32A corresponds to a "second display unit."

[0031] Fig. 3 is a block diagram showing an example of a functional configuration of the vehicle control device 20. As shown in Fig. 3, the vehicle control device 20 has a surrounding information acquisition unit 200, a behavior information acquisition unit 210, an operation reception unit 220, a driving control unit 230, an authority granting unit 240, a transmission unit 250, a reception unit 260, and a guidance image generation unit 270. Each functional configuration is realized by the CPU 20A reading out a program stored in the ROM 20B and executing the program.

[0032] The surrounding information acquisition unit 200 has a function of acquiring surrounding information of the surroundings of the vehicle 12. The surrounding information acquisition unit 200 acquires the surrounding information of the vehicle 12 from the external sensor 24 via the input / output I / F 20F. This surrounding information includes a surrounding image of the vehicle 12 captured by the camera 24A (hereinafter referred to as a "surrounding image").

[0033] The behavior information acquisition unit 210 has a function of acquiring behavior information related to the behavior of the vehicle, such as the steering angle and vehicle speed of the vehicle 12. The behavior information acquisition unit 210 has a function of acquiring the behavior information of the vehicle 12 from the internal sensor 26 via the input / output I / F 20F.

[0034] The operation receiving unit 220 has a function of receiving signals output from each input device 28 when manual driving is performed based on the operation of an occupant of the vehicle 12. The operation receiving unit 220 generates vehicle operation information, which is operation information for controlling the actuator 30, based on the signals received from each input device 28.

[0035] The driving control unit 230 has a function of controlling remote driving based on remote operation information received (acquired) from the remote operation device 14, and manual driving based on vehicle operation information accepted from the operation acceptance unit 220.

[0036] The authority granting unit 240 has a function of granting operation authority, which is the authority to operate the vehicle 12 equipped with the vehicle control device 20, to a remote operator who operates the remote operation device 14. When operation authority is granted to a remote operator, the vehicle control device 20 transmits an authority granting signal to the remote operation device 14 operated by the remote operator. By transferring the operation authority from the vehicle 12 to the remote operator, the vehicle 12 runs based on the remote operation information received from the remote operation device 14. In other words, remote driving is performed by the remote operator.

[0037] Furthermore, when the remote driving by the remote operation device 14 is to be ended, the authority granting unit 240 transmits an authority transfer signal to the remote operation device 14 to notify the remote operation device 14 that the operation authority will be transferred to the vehicle 12. Upon receiving the authority transfer signal, the remote operation device 14 executes an end process to end the remote driving.

[0038] The transmission unit 250 has the function of acquiring specification information (dimensions, etc.) of the vehicle 12 from the storage 20D during remote driving, adding the specification information to the surrounding images and behavior information of the vehicle 12 to generate vehicle information, and transmitting it to the remote control device 40.

[0039] The receiving unit 260 has a function of receiving an operation image and viewpoint information transmitted from the remote control device 40 during remote driving.

[0040] The guide image generation unit 270 has the function of generating a guide image 100 (see Figures 11, 12, and 14) by superimposing a viewpoint symbol 90 (black circle portion) indicating the operator's viewpoint on the operation image 80 based on the viewpoint information, and displaying the guide image 100 on the display 32A.

[0041] (Remote Control Device)

[0042] 4 is a block diagram showing a hardware configuration of devices mounted on the remote operation device 14 of this embodiment. In addition to the above-mentioned remote control device 40, the remote operation device 14 includes a display 42, a speaker 44, and an input device 48. The display 42 corresponds to the "first display unit."

[0043] The remote control device 40 includes a CPU 40A, a ROM 40B, a RAM 40C, a storage 40D, a communication I / F 40E, and an input / output I / F 40F. The CPU 40A, the ROM 40B, the RAM 40C, the storage 40D, the communication I / F 40E, and the input / output I / F 40F are connected to each other so as to be able to communicate with each other via a bus 40G. The functions of the CPU 40A, the ROM 40B, the RAM 40C, the storage 40D, the communication I / F 40E, and the input / output I / F 40F are similar to those of the CPU 20A, the ROM 20B, the RAM 20C, the storage 20D, the communication I / F 20E, and the input / output I / F 20F of the vehicle control device 20 described above.

[0044] The CPU 40A reads out a program from the ROM 40B and executes the program using the RAM 40C as a working area. In this embodiment, the program is stored in the ROM 40B. By the CPU 40A executing the program, the remote control device 40 functions as a vehicle information acquisition unit 400, an operation image generation unit 410, a viewpoint detection unit 420, an operation information generation unit 430, and an operation switching unit 440 shown in FIG.

[0045] A display 42, a speaker 44, a camera 46, and an input device 48 are connected to the remote control device 40 of this embodiment via the input / output I / F 40F. The display 42, the speaker 44, the camera 46, and the input device 48 may be directly connected to the bus 40G.

[0046] The display 42 is a liquid crystal display for displaying images of the surroundings of the vehicle 12 captured by the camera 24A of the vehicle 12 and various information related to the vehicle 12. The speaker 44 outputs sounds recorded together with the captured images by a microphone attached to the camera 24A of the vehicle 12.

[0047] Camera 46 constitutes a gaze detection device that detects the gaze of the remote operator, and captures an image of the face of the remote operator seated in front of display 42 from the front, and outputs the captured image to remote control device 40.

[0048] The input device 48 is a controller operated by a remote operator using the remote operation device 14. The input device 48 includes a steering wheel 48A as a switch for steering the steering wheels of the vehicle 12, an accelerator pedal 48B as a switch for accelerating the vehicle 12, and a brake pedal 48C as a switch for decelerating the vehicle 12. Note that the form of each input device 48 is not limited to this. For example, a lever switch may be provided instead of the steering wheel 48A. Also, for example, a push button switch or a lever switch may be provided instead of the pedal switches of the accelerator pedal 48B and the brake pedal 48C.

[0049] Fig. 5 is a block diagram showing an example of a functional configuration of the remote control device 40. As shown in Fig. 5, the remote control device 40 has a vehicle information acquisition unit 400, an operation image generation unit 410, a viewpoint detection unit 420, an operation information generation unit 430, and an operation switching unit 440.

[0050] The vehicle information acquisition unit 400 has a function of acquiring images and recorded sounds from the camera 24A transmitted from the vehicle control device 20, as well as vehicle information such as steering angle, vehicle speed, and the like.

[0051] 10, the operation image generating unit 410 has a function of creating an auxiliary image 60 for assisting the operation of the operator based on the acquired vehicle information, i.e., behavior information such as specification information, steering angle, and vehicle speed of the vehicle 12, and generating an operation image 80 by superimposing the auxiliary image 60 on the acquired captured image (surrounding image 70 of the vehicle 12). In addition, the operation image generating unit 410 has a function of displaying the operation image 80 on the display 42 and transmitting the operation image 80 to the vehicle control device 20.

[0052] The viewpoint detection unit 420 has a function of detecting the line of sight of the remote operator based on the image captured by the camera 46, and detecting the viewpoint (gazing point) of the remote operator on the peripheral image displayed on the display 42.

[0053] The operation information generating unit 430 has a function of receiving signals output from each input device 48 when remote driving based on the operation of a remote operator is performed. The operation information generating unit 430 also has a function of generating remote operation information based on the signals received from each input device 48 and transmitting the information to the vehicle control device 20.

[0054] The operation switching unit 440 has a function of controlling switching regarding remote driving of the vehicle 12. Specifically, when the operation switching unit 440 receives an authority granting signal from the vehicle 12, it executes processing for remote driving of the vehicle 12. In addition, when the operation switching unit 440 receives an authority transfer signal from the vehicle 12, it executes termination processing for terminating the remote driving of the vehicle 12.

[0055] (Flow of Control)

[0056] Next, the flow of control in the vehicle 12 and the remote control device 14 constituting the remote control system 10 will be described with reference to FIGS.

[0057] (Processing in the vehicle control device)

[0058] Next, the flow of processing executed in the vehicle control device 20 of the vehicle 12 will be described with reference to the flowcharts of Figures 6 and 7. Note that in the description of this processing, a general remote operation, that is, the point where remote operation information is transmitted from the remote control device 40 to the vehicle control device 20 by operating the input device 48 of the remote operation device 14, and the actuator 30 of the vehicle 12 is controlled based on this, will be omitted. In other words, the control of generating an operation image 80 in the remote operation device 14 and displaying it on the display 42, and the control of generating a guide image 100 in the vehicle 12 and displaying it on the display 32A will be described.

[0059] In step S10, the CPU 20A determines whether or not an occupant of the vehicle 12 has selected remote driving. Specifically, it determines whether or not a remote switching signal has been received. If this determination is positive, the process proceeds to the next step S12, and if this determination is negative, the process in step S10 is repeated.

[0060] When the process proceeds to step S12, the CPU 20A transmits an authorization signal to the remote control device 40.

[0061] In step S14, the CPU 20A determines whether or not an authority acceptance signal has been received from the remote control device 40. If this determination is affirmative, the process proceeds to the next step S16, and if this determination is negative, the process in step S14 is repeated.

[0062] In step S16, the CPU 20A generates vehicle information including the surrounding image captured by the camera 24A, behavior information including the steering angle and vehicle speed detected by the steering angle sensor 26A and the vehicle speed sensor 26B at the timing of capturing the image, and specification information (dimensions, etc.) of the vehicle 12 stored in the storage 20D. Note that the specification information of the vehicle 12 is generated only the first time and is omitted from the next time onwards.

[0063] Next, in step S18, the CPU 20A transmits the generated vehicle information to the remote control device 40.

[0064] In step S20, the CPU 20A determines whether or not an operation image has been received from the remote control device 40. If this determination is affirmative, the process proceeds to the next step S22, and if this determination is negative, the process in step S20 is repeated.

[0065] In step S22, CPU 20A determines whether or not viewpoint information has been received from remote control device 40. If this determination is affirmative, the process proceeds to the next step S24, and if this determination is negative, the process in step S22 is repeated.

[0066] In step S24, CPU 20A generates guide image 100 by superimposing viewpoint symbol 90 on operation image 80 based on viewpoint information, as shown in FIG.

[0067] In step S26, CPU 20A controls so as to display a guide image on display 32A.

[0068] Furthermore, in step S28, the CPU 20A judges whether or not the occupant of the vehicle 12 has selected manual driving. Specifically, it judges whether or not a manual switching signal has been received. If this judgment is affirmative, the process proceeds to step S30, and if this judgment is negative, the process from step S16 onwards is repeated.

[0069] If the process proceeds to step S30, the CPU 20A transmits an authority transfer signal to the remote control device 14, and ends the series of processes.

[0070] (Processing by remote control device)

[0071] Next, the flow of processing executed in the remote control device 14 will be described with reference to the flowcharts of FIGS.

[0072] 8, the CPU 40A determines whether or not an authorization signal has been received. If the determination is affirmative, the process proceeds to the next step S42, and if the determination is negative, the process in step S40 is repeated.

[0073] In step S42, the CPU 40A transmits an authority acceptance signal to the vehicle control device 20 when the selected remote operator completes preparation for remote driving.

[0074] In step S44, the CPU 40A acquires vehicle information from the vehicle control device 20 (the surrounding image, the specification information of the vehicle 12, and the behavior information of the vehicle 12).

[0075] Next, in step S46, the CPU 40A creates an auxiliary image 60 (see the thick lines in FIG. 10) based on the specification information of the vehicle 12 and the behavior information of the vehicle 12. That is, a pair of vehicle width lines 62A, 62B are created (at positions corresponding to both ends) based on the overall width of the vehicle 12, and a vehicle center line 64 is created (at a position corresponding to the center of the vehicle width). In addition, a leading edge guide line 66 is created, which indicates the leading edge position of the vehicle 12 and extends in the vehicle width direction. Furthermore, a meter section 68 that displays the vehicle speed (numerical value) is created in the center of the leading edge guide line 66 in the vehicle width direction.

[0076] Based on the behavior information of the vehicle 12, the vehicle speed is displayed in the meter section 68 of the auxiliary image 60.

[0077] As shown in FIG. 13, the directions of the vehicle center line 64 and the vehicle width lines 62A, 62B are inclined to the left and right from the front in accordance with the steering angle of the behavior information.

[0078] The lengths of the vehicle width lines 62A, 62B and the vehicle center line 64 may be changed according to the vehicle speed. For example, as shown in Fig. 12, the vehicle width lines 62A, 62B and the vehicle center line 64 may be controlled to become longer as the vehicle speed increases.

[0079] Subsequently, in step S48, CPU 40A creates an operation image 80 by superimposing auxiliary image 60 on peripheral image 70, as shown in FIG.

[0080] Furthermore, in step S50, CPU 40A causes operation image 80 to be displayed on display .

[0081] Furthermore, in step S52, CPU 40A detects the line of sight of the remote operator from the image captured by camera 46, and acquires viewpoint (gazing point) information on operation image 80 displayed on display 42 from the line of sight.

[0082] Next, in step S54, the CPU 40A transmits the operation image and the viewpoint information to the vehicle control device 20.

[0083] In step S56, the CPU 40A judges whether the remote control operation has ended, that is, whether the authority transfer signal has been received. If the judgment is affirmative, the series of controls is ended, and if the judgment is negative, the processes from step S44 onwards are repeated.

[0084] (Summary of this embodiment)

[0085] In the remote operation system 10 of this embodiment, during remote driving, an operation image 80 is generated by superimposing an auxiliary image 60 on a surrounding image 70 of the vehicle 12 in the remote operation device 14 (remote control device 40) to facilitate remote driving by the remote operator, and a guide image 100 is generated by superimposing a viewpoint symbol 90 indicating the viewpoint of the remote operator on this operation image 80 in the vehicle 12 (vehicle control device 20), and is displayed on the display 32A of the vehicle 12.

[0086] That is, since the guide image 100 displays the viewpoint symbol 90 indicating the viewpoint of the remote operator on the operation image 80 (peripheral image 70), the occupant of the vehicle 12 can visually confirm where the remote operator is looking on the peripheral image 70, and can predict the movement (behavior) of the vehicle 12. For example, as shown in FIG. 11, when the viewpoint symbol 90 indicating the viewpoint of the remote operator in the guide image 100 is on a straight road, it is expected that the vehicle 12 will go straight through the intersection, and when the viewpoint symbol 90 indicating the viewpoint of the remote operator in the guide image 100 is on the left road, as shown in FIG. 14, it is expected that the vehicle 12 will turn left at the intersection. That is, when the occupant of the vehicle 12 visually confirms the viewpoint symbol 90 of the guide image 100 displayed on the display 32A, it is possible to confirm the operation (driving) intention of the remote operator and predict the behavior of the vehicle 12.

[0087] In addition, the auxiliary image 60 included in the guide image 100 displays the vehicle center line 64 and vehicle width lines 62A, 62B, the directions of which change according to the steering angle, so that the occupants of the vehicle 12 can predict in which direction the vehicle 12 will head simply by visually checking the directions of the vehicle center line 64 and the vehicle width lines 62A, 62B.

[0088] Furthermore, as shown in FIG. 12, the lengths of the vehicle center line 64 and vehicle width lines 62A, 62B change (extend) according to the vehicle speed, and the vehicle speed is displayed on the meter unit 68, so that the driver can actually feel the change in vehicle speed.

[0089] (Supplementary explanation of the embodiment)

[0090] In the above embodiment, the vehicle 12 is configured to be capable of performing manual driving and remote driving, but the present invention is not limited to this, and the vehicle 12 may be configured to be capable of performing manual driving, automatic driving, and remote driving. Alternatively, the vehicle 12 may be configured to be capable of performing only remote driving.

[0091] In addition, in the above embodiment, the auxiliary image 60 is superimposed on the peripheral image 70 to generate the operation image 80, and the viewpoint symbol 90 is superimposed on the operation image 80 to generate the guide image 100, but the guide image 100 may also be generated simply by superimposing the viewpoint symbol 90 on the peripheral image 70.

[0092] Furthermore, in the above embodiment, the lengths of the vehicle center line 64 and the vehicle width lines 62A, 62B in the auxiliary image 60 are configured to change according to the vehicle speed, but this is not essential. Similarly, the meter section 68 in the auxiliary image 60 is provided to display the vehicle speed, but this is also not essential.

[0093] In the above embodiment, the orientations of the vehicle center line 64 and the vehicle width lines 62A, 62B in the auxiliary image 60 are changed in accordance with the steering angle, but this is not essential.

[0094] Furthermore, when the vehicle 12 is turning (when there is a steering angle), the vehicle center line 64 and the vehicle width lines 62A, 62B in the guide image 100 (operation image 80) are displayed as straight lines inclined left and right from the front, but this is not limited to this. The vehicle center line 64 and the vehicle width lines 62A, 62B may be displayed as curved lines curving left and right from the front.

[0095] In the above embodiment, the remote operator's line of sight (viewpoint) is detected from an image (of the face of the remote operator) captured by the camera 46, but the present invention is not limited to this. For example, the remote operator may wear a glasses-type viewpoint detection device to detect the line of sight (viewpoint) of the remote operator.

[0096] Furthermore, although the guide image 100 is displayed on the display 32A provided in the passenger compartment of the vehicle 12, the present invention is not limited to this. For example, the guide image 100 may be displayed on the windshield of the vehicle 12 by a head-up display device.

[0097] In addition, in the above embodiment, the operation image 80 and viewpoint information are transmitted from the remote control device 40 to the vehicle control device 20, and the vehicle control device 20 generates the guide image 100 by superimposing the viewpoint symbol 90 on the operation image 80, but it is also possible to configure the remote control device 40 to generate the guide image 100 and transmit the guide image 100 to the vehicle control device 20.

[0098] Furthermore, in the above embodiment, various types of processors other than the CPU may execute each process executed by the CPU 20A, 40A by reading the software (program). In this case, examples of the processor include a PLD (Programmable Logic Device) such as an FPGA (Field-Programmable Gate Array) whose circuit configuration can be changed after manufacture, and a dedicated electric circuit such as an ASIC (Application Specific Integrated Circuit) which is a processor having a circuit configuration designed exclusively for executing a specific process. Each process may be executed by one of these various types of processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, etc.). The hardware structure of these various types of processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0099] In the above embodiment, the programs are described as being stored (installed) in advance in a non-transitory recording medium that can be read by a computer. For example, in the remote control system 10, the programs are stored in advance in the storages 20D and 40D. However, the present invention is not limited to this, and the programs may be provided in a form recorded in a non-transitory recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. Each program may also be downloaded from an external device via a network.

[0100] The process flow described in the above embodiment is also one example, and unnecessary steps may be deleted, new steps may be added, or the process order may be changed, without departing from the spirit of the invention.

[0101] In addition, the present invention can be implemented in various modifications without departing from the spirit and scope of the present invention. Furthermore, the scope of the present invention is not limited to the above-described embodiment. [Explanation of symbols]

[0102] 10 Remote Control System 12 Vehicles 14 Remote control device 24A Camera (imaging unit) 32A Display (second display) 42 Display (first display section) 70 Peripheral Images 90 Viewpoint symbols 420 Viewpoint detection unit

Claims

[Claim 1] A remote operation system in which a vehicle is driven by a remote operator operating a remote operation device provided outside the vehicle, An imaging unit provided in the vehicle and configured to capture an image of the surroundings of the vehicle; A detection unit provided in the vehicle for detecting a steering angle and a vehicle speed of the vehicle; a first display unit that is provided in the remote control device and displays the surrounding image transmitted from the vehicle; a viewpoint detection unit that detects a viewpoint of the remote operator with respect to the peripheral image displayed on the first display unit; a second display unit that is provided on the vehicle and displays a viewpoint symbol indicating the viewpoint on the peripheral image in a superimposed manner, and that increases the length of a line displayed at a position corresponding to the vehicle width center or both ends of the vehicle in the peripheral image as the vehicle speed increases, and changes the direction of the line in accordance with the steering angle or curves the line in accordance with the steering angle; A remote control system comprising:

Images