Remote operation device and method, remote driving system, and computer readable medium

Abstract

The present application relates to a remote operation device, a remote driving system, a remote operation method, and a computer-readable medium recording a remote operation program, the remote operation device driving a vehicle by remote operation from outside the vehicle, the remote operation device including: an image acquisition unit acquiring a surrounding image of the vehicle from the vehicle; a first display unit displaying the surrounding image of the vehicle; and a second display unit displaying an object moving from a central field of view area side toward a width direction end portion in a peripheral field of view area other than the central field of view area of the first display unit, disposed on an upper side, or a lower side, or the upper side and the lower side of the first display unit.

Description

Technical Field

[0001] This disclosure relates to remote operating devices, remote driving systems, remote operating methods, and computer-readable media containing remote operating procedures. Background Technology

[0002] Japanese Patent Application Publication No. 2010-173492 discloses a method in which a pair of liquid crystal displays extending in the longitudinal direction are arranged on the side of the vehicle's roof and display a Gaussian blob that moves at a speed corresponding to the vehicle speed. The driver can visually confirm the movement of the Gaussian blob with his peripheral vision, thus intuitively identifying the vehicle speed without hindering driving operations.

[0003] In recent years, there has been a growing pursuit of remote driving technology, which allows vehicles to be operated from a distance via an operation room or other external location. In the operation room, an image corresponding to the view from the vehicle (driver's seat) is displayed on a monitor.

[0004] When driving remotely, existing technologies can be considered to provide the operator in the control room with a sense of vehicle speed.

[0005] However, when using the aforementioned prior art, there is a possibility that: a separate liquid crystal display is needed to display the Gaussian spot; in addition, when the operator's line of sight shifts from front to left or right, such as when the vehicle is turning, the Gaussian spot enters the operator's central field of vision, making it impossible to fully perceive the vehicle speed. Summary of the Invention

[0006] This disclosure provides a remote control device, a remote driving system, a remote control method, and a computer-readable medium containing remote control procedures, which enable the operator to obtain a sense of vehicle speed even when the vehicle is turning.

[0007] The first aspect of this disclosure is a remote operating device for driving a vehicle remotely from outside the vehicle. The remote operating device includes: an image acquisition unit for acquiring a peripheral image of the vehicle; a first display unit for displaying the peripheral image of the vehicle; and a second display unit disposed above, below, or both above and below the first display unit in a peripheral field of view area other than the central field of view area, for displaying an object moving from the central field of view area toward an end in the width direction.

[0008] In the first embodiment, the remote control device for operating the vehicle from outside the vehicle includes: an image acquisition unit that acquires images of the vehicle's surroundings; a first display unit that displays images of the vehicle's surroundings; and a second display unit positioned above, below, or both above and below the first display unit within the peripheral viewing area of ​​the first display unit, excluding the central viewing area. The second display unit displays an object that moves from the central viewing area toward its width-direction end.

[0009] Here, "the second display unit disposed above, below, or both above and below the first display unit" includes the second display unit disposed in the upper part, lower part, or both above and below the first display unit.

[0010] In addition, the "central field of view area" refers to the area on the first display unit that corresponds to the central field of view of the operator of the remote operating device.

[0011] In this way, by moving the object on the second display unit from the center field of view toward the end in the width direction, the operator can feel the speed of the vehicle being operated.

[0012] In particular, even if the operator's gaze moves to the left or right, the object is only displayed on the second display unit located above, below, or both above and below the peripheral visual field area outside the central visual field area on the first display unit. That is, it can prevent the object displayed on the second display unit from entering the operator's central visual field, prevent or suppress the reduction of the operator's visual confirmation of the peripheral image, and enable the operator to obtain a sense of vehicle speed.

[0013] The second aspect of this disclosure is based on the first aspect described above, and includes: a steering angle information acquisition unit that acquires the steering angle information of the vehicle; and a center field of vision area determination unit that determines the center field of vision area based on the steering angle information.

[0014] According to the second method, the central field of view area of ​​the first display unit is determined (changed) based on the vehicle's steering angle information obtained by the steering angle information acquisition unit. For example, when the vehicle is turning left, the operator can grasp the situation ahead of the left turn by directing their gaze to the left of the vehicle's front. That is, when the vehicle is turning left, the operator's central field of view shifts to the left.

[0015] Therefore, in this method, the central field of view area on the first display unit is determined based on the vehicle's steering angle (information) (by shifting the central field of view area on the first display unit in the width direction), and a second display unit is provided above, below, or both above and below the peripheral field of view area of ​​the first display unit, excluding the central field of view area. This, for example, can prevent or suppress objects displayed on the second display unit from entering the central field of view of the operator who is looking to the left when the vehicle is turning left, thus reducing the operator's visual confirmation of the vehicle's peripheral image.

[0016] The third aspect of this disclosure is based on the first aspect described above, and includes: a gaze information acquisition unit that acquires gaze information of the operator of the remote operation device toward the first display unit; and a central field of view determination unit that determines the central field of view based on the gaze information.

[0017] According to the third method, the operator's central field of vision is determined (changed) based on the operator's line of sight information (line of sight) obtained by the line of sight information acquisition unit. For example, when the vehicle is turning left, the operator grasps the situation ahead of the left turn by directing their gaze to the left of the vehicle's front. That is, when the vehicle is turning left, the operator's central field of vision shifts to the left.

[0018] Therefore, in this method, the central field of view area on the first display unit is determined based on the operator's line of sight (information) (by shifting the central field of view area on the first display unit in the width direction), and a second display unit is provided above, below, or both above and below the peripheral field of view area of ​​the first display unit, excluding the central field of view area. This, for example, can prevent or suppress objects displayed on the second display unit from entering the central field of view of the operator who has shifted their gaze to the left when turning left, thus reducing the operator's visual confirmation of the vehicle's surrounding image.

[0019] The fourth aspect of this disclosure is based on the second or third aspect described above, wherein the central field of view determination unit determines the central field of view based on the distance from the viewpoint of the operator of the remote operating device to the first display unit.

[0020] According to the fourth method, the central field of view area in the first display unit is determined based on the distance from the operator's viewpoint to the first display unit.

[0021] The operator's central field of view is set within a predetermined angle range centered on the direction of the line of sight observed from the operator's viewpoint. Therefore, the central field of view area on the first display unit can be determined with high precision based on the predetermined angle centered on the direction of the line of sight observed from the operator's viewpoint and the distance from the operator's viewpoint to the first display unit.

[0022] The fifth aspect of this disclosure is based on any one of the first to fourth aspects described above, wherein the moving speed of the object varies according to the speed of the vehicle.

[0023] According to the fifth method, the moving speed of the object displayed on the second display unit varies with the vehicle speed. Therefore, the operator can actually perceive the change in vehicle speed simply by visually confirming the object displayed on the second display unit.

[0024] The sixth aspect of this disclosure is a remote driving system comprising a remote operating device according to any one of the aspects 1 to 5, and a vehicle driven by operating the remote operating device.

[0025] According to the sixth method, since the remote operating system has the aforementioned remote operating device and the vehicle operated by the aforementioned remote operating device, the operator can operate the vehicle well while obtaining a sense of vehicle speed.

[0026] The seventh aspect of this disclosure is a remote operation method in which a computer performs a process in a remote operation device for driving a vehicle remotely from outside the vehicle, including the following steps: causing an image acquisition unit to acquire a peripheral image of the vehicle from the vehicle, causing a first display unit to display the peripheral image of the vehicle, and causing a second display unit to display an object moving from the central field of view of the first display unit toward the end in the width direction, wherein the second display unit is disposed above, below, or both above and below the first display unit in the peripheral field of view of the first display unit other than the central field of view.

[0027] According to the seventh method, similar to the first method, the operator can perceive the speed of the vehicle being operated on. Furthermore, even if the operator's gaze moves left or right, the object is prevented from entering the operator's central field of vision, thus preventing or suppressing a decrease in the operator's visual confirmation of the surrounding image.

[0028] The eighth aspect of this disclosure is a (non-transitory) computer-readable medium containing a remote operation program that causes a computer in a remote operation device for driving a vehicle remotely from outside the vehicle to perform a process including the following steps: causing an image acquisition unit to acquire a peripheral image of the vehicle from the vehicle, causing a first display unit to display the peripheral image of the vehicle, and causing a second display unit to display an object moving from the central field of view of the first display unit toward the end in the width direction, wherein the second display unit is disposed above, below, or both above and below the first display unit in the peripheral field of view of the first display unit other than the central field of view.

[0029] According to the eighth method, similar to the first method, the operator can perceive the speed of the vehicle being operated on. Furthermore, even when the operator's gaze shifts to the left or right, the object is prevented from entering the operator's central field of vision, thus preventing or suppressing a decrease in the operator's visual confirmation of the surrounding image.

[0030] According to the above method, the remote operation device, remote driving system, remote operation method, and computer-readable medium containing the remote operation program disclosed herein can also enable the operator in remote driving operation to obtain a sense of vehicle speed when the vehicle is turning. Attached Figure Description

[0031] The specific embodiments are described in detail with reference to the following figures, wherein:

[0032] Figure 1 This is a schematic diagram illustrating a simplified structure of a remote operating system involved in an exemplary implementation.

[0033] Figure 2 It is a block diagram representing the hardware structure of a vehicle.

[0034] Figure 3 This is a block diagram illustrating an example of the functional structure of a vehicle control device.

[0035] Figure 4 It is a block diagram representing the hardware structure of a remote operating device.

[0036] Figure 5 This is a block diagram illustrating an example of the functional structure of a remote control device possessed by a remote operating device.

[0037] Figure 6 This is a schematic top view of the display device from above.

[0038] Figure 7 This is a diagram showing the unfolded image display surface of the display device.

[0039] Figure 8 This is a schematic top-down view of the display device during a left turn operation, viewed from above.

[0040] Figure 9 This is a diagram showing the unfolded image display surface of the display device during the left turn operation.

[0041] Figure 10 It is a flowchart representing the process executed by the vehicle control unit.

[0042] Figure 11 It is a flowchart representing the process executed by the vehicle control unit.

[0043] Figure 12It is a flowchart representing the process performed by a remote operating device.

[0044] Figure 13 It is a flowchart representing the process performed by a remote operating device.

[0045] Figure 14 It is a block diagram representing the hardware structure of the remote operating device involved in the variation.

[0046] Figure 15 This is a block diagram illustrating an example of the functional structure of the remote control device possessed by the remote operating device involved in the change.

[0047] Figure 16 This is a diagram showing the unfolded image display surface of the display device involved in other examples.

[0048] Figure 17 This is a schematic top view of the display devices involved in other examples, viewed from above.

[0049] Figure 18 This is a diagram showing the unfolded image display surface of the display device involved in other examples. Detailed Implementation

[0050] The following uses Figures 1 to 13 A remote operating system 10 according to one implementation method will be described.

[0051] like Figure 1 As shown, the remote operating system 10 of this exemplary embodiment is configured to include a vehicle 12 and a remote operating device 14.

[0052] The vehicle 12 is equipped with a vehicle control device 20. Additionally, the remote operation device 14 is configured to include a remote control device 40. Furthermore, in the aforementioned remote operating system 10, the vehicle control device 20 of the vehicle 12 and the remote control device 40 of the remote operation device 14 are connected via a network N1 and are able to communicate with each other. As an example, this network N1 is a wired or wireless communication network using public lines, such as the Internet.

[0053] (Vehicle structure)

[0054] Vehicle 12 is configured to perform both manual and remote driving. Manual driving is based on driving operations by a driver who is an occupant of vehicle 12, while remote driving is based on driving operations by an operator (remote driver) using a remote control device 14. Vehicle 12 is typically set to manual driving mode.

[0055] Figure 2This is a block diagram showing the hardware structure of the equipment mounted on vehicle 12. In addition to the vehicle control device 20 mentioned above, vehicle 12 also includes external sensor 24, internal sensor 26, input device 28, actuator 30, and user I / F (Interface) 32.

[0056] like Figure 2 As shown, the vehicle control device 20 is configured to include a CPU (Central Processing Unit) 20A, a ROM (Read Only Memory) 20B, a RAM (Random Access Memory) 20C, a storage device 20D, a communication I / F (Interface) 20E, and an input / output I / F 20F. The CPU 20A, ROM 20B, RAM 20C, storage device 20D, communication I / F 20E, and input / output I / F 20F are connected via a bus 20G and are able to communicate with each other.

[0057] CPU 20A is the central processing unit, which executes various programs and controls various components. Specifically, CPU 20A reads programs from ROM 20B and uses RAM 20C as its working area to execute the programs. In this exemplary embodiment, the programs are stored in ROM 20B. By executing the programs through CPU 20A, the vehicle control device 20... Figure 3 The unit shown functions as the peripheral information acquisition section 200, the behavior information acquiring section 210, the operation receiving section 220, the driving control section 230, and the permission granting section 240.

[0058] like Figure 2 As shown, ROM 20B stores various programs and data. RAM 20C serves as a working area to temporarily store programs or data. Storage 20D, consisting of an HDD (Hard Disk Drive) or SSD (Solid State Drive), stores various programs, including the operating system, and various data. Communication I / F 20E includes an interface for connecting to the network N1 to communicate with the remote control device 40. This interface can use communication standards such as LTE or Wi-Fi (registered trademark).

[0059] In this exemplary embodiment, the communication I / F20E transmits images captured by the camera 24A and information related to the vehicle's driving status detected by the internal sensor 26 to a remote operating device 14 outside the vehicle 12 via network N1, and receives operating information for operating the vehicle 12, i.e., remote operating information, from the remote operating device 14. The camera 24A is equipped with a microphone (not shown), which transmits and receives sound data signals along with the captured image data.

[0060] The input / output I / F 20F is an interface for communicating with various devices mounted on the vehicle 12. In this exemplary embodiment, the vehicle control device 20 is connected to external sensor 24, internal sensor 26, input device 28, actuator 30, and user I / F 32 via the input / output I / F 20F. Furthermore, the external sensor 24, internal sensor 26, input device 28, actuator 30, and user I / F 32 can also be directly connected to the bus 20G.

[0061] External sensor 24 is a sensor group that detects surrounding information of the vehicle 12. External sensor 24 includes a camera 24A that captures images within a defined range.

[0062] The internal sensor 26 is a sensor group for detecting the driving status of the vehicle 12. The internal sensor 26 includes a steering angle sensor 26A for detecting the amount of steering wheel operation and a vehicle speed sensor 26B for detecting the vehicle speed.

[0063] The input device 28 is a set of switches for operation by an occupant riding in the vehicle 12. The input device 28 includes a steering wheel 28A as a switch to steer the steering wheels of the vehicle 12, an accelerator pedal 28B as a switch to accelerate the vehicle 12, and a brake pedal 28C as a switch to decelerate the vehicle 12.

[0064] The actuator 30 includes a steering wheel actuator that drives the steering wheels of the vehicle 12, an acceleration actuator that controls the acceleration of the vehicle 12, and a braking actuator that controls the deceleration of the vehicle 12.

[0065] The user I / F32 includes a display and speakers (both omitted from the illustration). The display is, for example, located on the dashboard of vehicle 12, forming an electrostatic capacitive touch panel. The speakers are, for example, located in the left and right side doors of vehicle 12.

[0066] Figure 3 This is a block diagram illustrating an example of the functional structure of the vehicle control device 20. For example... Figure 3As shown, the vehicle control device 20 includes a peripheral information acquisition unit 200, a movement information acquisition unit 210, an operation receiving unit 220, a driving control unit 230, and an authorization unit 240. Each functional structure is implemented by the CPU 20A reading and executing the program stored in the ROM 20B.

[0067] The surrounding information acquisition unit 200 has the function of acquiring surrounding information of the vehicle 12. The surrounding information acquisition unit 200 acquires the surrounding information of the vehicle 12 from the external sensor 24 via the input / output I / F 20F. The surrounding information includes the front side surrounding image of the vehicle 12 captured by the camera 24A (hereinafter referred to as "surrounding image").

[0068] The motion information acquisition unit 210 has the function of acquiring motion information related to the vehicle's movement, such as steering angle information and vehicle speed information. The motion information acquisition unit 210 acquires the vehicle's motion information from the internal sensor 26 via the input / output I / F 20F.

[0069] The operation receiving unit 220 has the function of receiving signals output from each input device 28 during manual driving based on the operation of the occupants of the vehicle 12. The operation receiving unit 220 generates operation information, i.e., vehicle operation information, for controlling the actuator 30 based on the signals received from each input device 28.

[0070] The driving control unit 230 has the function of controlling remote driving based on remote operation information received (acquired) from the remote operation device 14, and manual driving based on vehicle operation information received from the operation receiving unit 220.

[0071] The authorization unit 240 has the function of granting operation permissions to the operator of the remote operation device 14. These permissions are for operating the vehicle 12 equipped with the vehicle control device 20. When operation permissions are granted to the operator, the vehicle control device 20 sends an authorization signal to the remote operation device 14 operated by that operator. By transferring operation permissions from the vehicle 12 to the remote operation device 14, the vehicle 12 is driven based on remote operation information received from the remote operation device 14. In other words, remote driving based on the operator is performed.

[0072] Furthermore, upon termination of remote driving involving the remote operating device 14, the authorization unit 240 sends a handover notification signal to the remote operating device 14, which notifies that operating authority is being transferred to the vehicle 12. Upon receiving the handover notification signal, the remote operating device 14 executes termination processing to end the remote driving.

[0073] (Remote operating device)

[0074] Figure 4 This is a block diagram showing the hardware structure of the device mounted on the remote operation device 14 in this exemplary embodiment. In addition to the remote control device 40 described above, the remote operation device 14 also includes a display device 42, a speaker 44, and an input device 48 as an operation unit.

[0075] The remote control device 40 is configured to include a CPU 40A, a ROM 40B, a RAM 40C, a memory 40D, a communication I / F 40E, and an input / output I / F 40F. The CPU 40A, ROM 40B, RAM 40C, memory 40D, communication I / F 40E, and input / output I / F 40F are connected via a bus 40G and are capable of communicating with each other. The functions of the CPU 40A, ROM 40B, RAM 40C, memory 40D, communication I / F 40E, and input / output I / F 40F are the same as those of the CPU 20A, ROM 20B, RAM 20C, memory 20D, communication I / F 20E, and input / output I / F 20F of the vehicle control device 20 described above.

[0076] CPU 40A reads the program from ROM 40B and executes the program using RAM 40C as the working area. In this exemplary embodiment, the program is stored in ROM 40B. By executing the program through CPU 40A, the remote control device 40 is used as... Figure 5 The vehicle information acquisition unit 400, distance acquisition unit 410, center field of view area determination unit 420, display area determination unit 430, speed determination unit 440, display control unit 450, operation information generation unit 460, and operation switching unit 470 shown in the diagram perform their functions.

[0077] In this exemplary embodiment, the remote control device 40 is connected to a display device 42, a speaker 44, and an input device 48 via an input / output I / F 40F. Alternatively, the display device 42, speaker 44, and input device 48 can also be directly connected to the bus 40G.

[0078] Display device 42 is a liquid crystal display used to display images of the surroundings of vehicle 12 captured by camera 24A of vehicle 12, as well as various information related to vehicle 12. For example... Figure 6 As shown, the display device 42 is positioned in the remote control room (not shown) in front of the operator P (only the head is shown) seated in the operating seat (not shown), and has a generally curved shape when viewed from above. That is, the image display surface 42A of the display device 42 extends from one side of the operator P through the front to the other side. Therefore, it is configured to display the peripheral image of the vehicle 12 across the entire field of view of the operator P facing forward.

[0079] In addition, such as Figure 7As shown, the image display surface 42A on the inner side of the display device 42 is rectangular when unfolded into a plane. It is configured to display peripheral images on its entire surface, and to overlap the object 52 and the background 58 in the object display areas 54L, 54R, 56L, and 56R (hereinafter referred to as "object display areas 54L to 56R") determined at its four corners. The image display surface 42A corresponds to the "first display unit". The object display areas 54L to 56R correspond to the "second display unit".

[0080] The speaker 44 outputs sound that is recorded along with the captured image through the microphone of the camera 24A attached to the vehicle 12.

[0081] Input device 48 is a controller for operation by an operator using remote control device 14. Input device 48 includes a steering wheel 48A as a switch to steer the steering wheels of vehicle 12, an accelerator pedal 48B as a switch to accelerate vehicle 12, and a brake pedal 48C as a switch to decelerate vehicle 12. However, the configuration of each input device 48 is not limited to these. For example, a handle switch may be provided instead of steering wheel 48A. Additionally, push-button switches or handle switches may be provided instead of pedal switches for accelerator pedal 48B and brake pedal 48C.

[0082] Figure 5 This is a block diagram illustrating an example of the functional structure of the remote control device 40. For example... Figure 5 As shown, the remote control device 40 includes a vehicle information acquisition unit 400, a distance acquisition unit 410, a center field of view determination unit 420, a display area determination unit 430, a speed determination unit 440, a display control unit 450, an operation information generation unit 460, and an operation switching unit 470.

[0083] The vehicle information acquisition unit 400 has the functions of acquiring images captured by the camera 24A sent from the vehicle control device 20, namely, images of the surroundings of the vehicle 12 (hereinafter referred to as "surrounding images"), recording sound, and vehicle information such as vehicle speed and steering angle. The acquired images are displayed on the image display surface 42A of the display device 42, and the recorded sound is output from the speaker 44. The vehicle information acquisition unit 400 is equivalent to an "image acquisition unit," a "steering angle information acquisition unit," and a "vehicle speed information acquisition unit."

[0084] The distance acquisition unit 410 acquires the distance L from the eyepoint of the operator P seated in the remote control room (not shown) to the image display surface 42A of the display device 42. In this exemplary embodiment, the distance L is measured in advance, and the value for each operator P is stored in the memory 40D. Therefore, by determining the operator P, the distance L up to the image display surface 42A of the display device 42 is acquired from the memory 40D.

[0085] In addition, the distance from the operator P's viewpoint to the image display surface 42A of the display device 42 can also be detected by a distance sensor or the like.

[0086] The central field of view determination unit 420 takes the direction of the vehicle 12 tilted to the left and right from the operator P's frontal direction as the line of sight, and determines the range of a predetermined angle from the operator P's viewpoint centered on this line of sight as the central field of view area 50. Specifically, as follows... Figure 8 As shown, the direction of the viewpoint LV is defined as the direction tilted left and right by a turning angle θ1 from the operator's frontal direction C. The central field of view area 50 on the image display surface 42A of the display device 42 is determined based on the distance L from the operator P's viewpoint to the image display surface 42A of the display device 42 and the field of view angle θ2 of the central field of view centered on the viewpoint LV (20 degrees downward from above in this exemplary embodiment). Figure 9 (The shaded area).

[0087] The display area determination unit 430 determines the display object 52 (see reference) on the image display surface 42A of the display device 42. Figure 7 The area of ​​). Specifically, such as Figure 7 As shown, on the image display surface 42A, the portions on the left and right sides (peripheral viewing areas 51) other than the central viewing area 50 are defined as the upper object display areas 54L and 54R by a specified distance from the top, and the portions on the lower side are defined as the lower object display areas 56L and 56R by a specified distance from the bottom.

[0088] The speed determination unit 440 determines the movement speed of the object 52 from the central viewing area 50 to the left and right ends in such a way that the operator perceives the object 52 as moving at a certain speed corresponding to the vehicle speed within the object display area 54L to 56R. Specifically, the movement speed of the object 52 corresponding to the vehicle speed is read from a table pre-stored in the memory 40D that sets the correspondence between vehicle speed and the movement speed of the object 52.

[0089] The display control unit 450 displays the acquired peripheral image on the entire surface of the image display surface 42A of the display device 42, and displays the object 52 moving at a predetermined speed corresponding to the vehicle speed and the peripheral image overlapping in the object display area 54L to 56R of the image display surface 42A.

[0090] In this exemplary embodiment, the display control unit 450 displays a monochrome background 58 overlapping with the surrounding image in the entire area of ​​the object display area 54L to 56R.

[0091] The operation information generation unit 460 has the function of receiving signals output from each input device 48 in the case of remote driving based on operator operation. In addition, the operation information generation unit 460 generates remote operation information to be sent to the vehicle control unit 20 based on the signals received from each input device 48.

[0092] The operation switching unit 470 has the function of controlling switching related to remote driving of the vehicle 12. Specifically, when an authorization signal is obtained from the vehicle 12, the operation switching unit 470 performs processing for remote driving of the vehicle 12. In addition, when a handover notification signal is obtained from the vehicle 12, the operation switching unit 470 performs termination processing to end the remote driving of the vehicle 12.

[0093] (Control process)

[0094] Next, use Figures 10-13 The control flow in the vehicle 12 constituting the remote operating system 10 and the remote operating device 14 will be described.

[0095] (Processing in vehicle control devices)

[0096] First, use Figure 10 as well as Figure 11 The flowchart illustrates the process executed in the vehicle control device 20 of vehicle 12. In this process description, remote operation information is sent from the remote control device 40 to the vehicle control device 20 via general remote operation, i.e., by operating the input device 48 of the remote operation device 14. Details regarding the control of the actuator 30 of vehicle 12 based on this are omitted. Specifically, the control for displaying object 52 on the display device 42 via the remote operation device 14 is explained.

[0097] exist Figure 10 In step S10, CPU 20A determines whether the occupants of vehicle 12 have selected remote driving. Specifically, it determines whether a remote switching signal has been received. If the determination is positive, it proceeds to the next step S12; if the determination is negative, the process in step S10 is repeated.

[0098] Upon proceeding to step S12, CPU 20A sends an authorization signal to remote control device 40.

[0099] In step S14, CPU 20A determines whether an authorization acceptance signal has been received from the remote control device 40. If the determination is positive, the process proceeds to the next step S16; if the determination is negative, the process in step S14 is repeated.

[0100] In step S16, CPU 20A generates vehicle information consisting of the surrounding image captured by camera 24A and the motion information including the steering angle information and vehicle speed information detected by steering angle sensor 26A and vehicle speed sensor 26B respectively at the time of shooting.

[0101] Next, in step S18, CPU20A sends the generated vehicle information to remote control device 40.

[0102] Furthermore, in Figure 11 In step S20, CPU 20A determines whether the occupants of vehicle 12 have selected manual driving. Specifically, it determines whether a manual switching signal has been received. If the determination is positive, the process proceeds to step S22; if the determination is negative, the process after step S16 is repeated.

[0103] Upon proceeding to step S22, CPU 20A sends a handover notification signal to remote control device 40 and terminates a series of processes.

[0104] (Processing in remote operating devices)

[0105] Next, use Figure 12 as well as Figure 13 The flowchart illustrates the process performed in the remote operation device 14.

[0106] exist Figure 12 In step S30, CPU 40A determines whether a permission granting signal has been received. If the determination is positive, it proceeds to the next step S32; if the determination is negative, the process in step S30 is repeated.

[0107] In step S32, if the selected operator has completed the preparation for remote driving, the CPU40A sends an authorization acceptance signal to the vehicle control device 20.

[0108] In step S34, the CPU 40A obtains from the storage 40D the distance L from the selected operator's viewpoint to the image display surface 42A of the display device 42 (refer to...). Figure 6 ).

[0109] In step S36, CPU40A obtains vehicle information, namely surrounding images and vehicle 12 movement information (vehicle speed information, steering angle information, etc.) from vehicle control device 20.

[0110] Next, in step S38, CPU 40A determines the central field of view area 50 based on the distance L and the steering angle information of vehicle 12. For example, when the steering wheel is operated while vehicle 12 is turning left, such as... Figure 8 As shown, the direction tilted to the left by an angle θ1 from the operator P's frontal direction C when viewed from above is taken as the line of sight LV. The central field of view (CA) is defined as the range of the central field of view angle θ2 (20 degrees from above in this example embodiment, 10 degrees to the left and right of the line of sight LV) centered on the line of sight LV. Furthermore, in the viewpoint of the operator observing the entire area of ​​the image display surface 42A from above, the range (angles) to the left and right excluding the central field of view CA is defined as the peripheral field of view PA (see reference). Figure 6 , Figure 8 ).

[0111] The central field of view CA on the image display surface 42A of the display device 42 is the central field of view region 50. Specifically, as Figure 8 As shown, the central field of view region 50 on the image display surface 42A is determined based on the distance L from the operator P's viewpoint to the image display surface 42A of the display device 42 and the field of view angle θ2 of the central field of view centered on the line of sight LV (20 degrees downward in this exemplary embodiment). Figure 9 (The shaded area). Specifically, when the steering angle is 0 degrees, such as... Figure 6 as well as Figure 7 As shown, the area centered on the line of sight LV, which is the frontal direction C, and with a field of view angle θ2 (20 degrees in this example embodiment, 10 degrees to the left and right) is the central field of view region 50 (refer to...). Figure 7 (The shaded area).

[0112] Next, in step S40, as Figure 7 , Figure 9 As shown, the CPU 40A determines the object display areas 54L to 56R on the image display surface 42A of the display device 42. Specifically, as... Figure 7 , Figure 9 As shown, in the left and right sides (peripheral field of view 51) of the image display surface 42A excluding the central field of view 50, there are upper object display areas 54L and 54R with a range from the top to the bottom, and lower object display areas 56L and 56R with a range from the bottom to the top, which are determined by a specified distance from the top.

[0113] Furthermore, in Figure 13 In step S42, the CPU 40A determines the moving speed of the object 52 displayed in the object display areas 54L to 56R based on the vehicle speed. At this time, the moving speed is determined in a way that the object 52 appears to move at a certain speed on the image display surface 42A of the display device 42. In this exemplary embodiment, the CPU 40A reads the moving speed of the object 52 corresponding to the vehicle speed from a table stored in the memory 40D that sets the correspondence between vehicle speed and the moving speed of the object 52, and determines this as the moving speed of the object 52.

[0114] Next, in step S44, CPU 40A displays a peripheral image over the entire area of ​​the image display surface 42A of display device 42, and displays object 52 and background 58 overlapping in the peripheral image across object display areas 54L to 56R. At this time, object 52 is displayed as moving from the central viewing area 50 towards the left and right ends within each object display area 54L to 56R at a determined moving speed (see reference). Figure 7 , Figure 9 (arrow).

[0115] In step S46, CPU40A determines whether remote driving operation has ended, i.e., whether a handover notification signal has been received. If the determination is positive, a series of controls are terminated; if the determination is negative, the processing after step S36 is repeated.

[0116] (This example summarizes the implementation methods)

[0117] In the remote operating system 10 of this exemplary embodiment, since the object 52 moving from the central field of view 50 toward the left and right ends at a speed corresponding to the vehicle speed is displayed in the object display area 54L to 56R of the image display surface 42A of the display device 42 of the remote operating device 14, the operator can further obtain a sense of vehicle speed by visually confirming the movement of the object 52.

[0118] Specifically, in the image display surface 42A of the display device 42, the object display areas 54L to 56R, excluding the central field of view area 50, are set to a predetermined range extending from the top or bottom of the image display surface 42A in the left-right direction. Therefore, it is possible to prevent the object 52 displayed in the object display areas 54L to 56R from entering the operator's central field of view, and to suppress any obstruction to the operator's remote driving operation.

[0119] Furthermore, the direction of the steering angle θ1 is considered as the operator's line of sight. The central field of vision area 50 is determined by the range of the central field of vision angle θ2 (20 degrees in this exemplary embodiment), centered on the line of sight LV tilted from the frontal direction C with the steering angle θ1 as the center, i.e., the central field of vision range CA. As a result, even when the operator's line of sight is oriented to the left or right, the central field of vision area 50 moves left or right accordingly. Therefore, since the object display areas 54L to 56R provided in the peripheral field of vision area 51 also shift left and right along with the movement of the central field of vision area 50, even when the vehicle is turning, the objects 52 displayed in the object display areas 54L to 56R can be prevented from entering the operator's central field of vision, thus preventing or suppressing any obstruction to the operator's remote driving operation.

[0120] Furthermore, in this exemplary embodiment, by displaying a single-color background 58 overlaid with the surrounding image in the object display areas 54L to 56R, the surrounding image is no longer visually confirmed by the operator in the object display areas 54L to 56R. That is, by displaying the object 52 on the background 58 in all object display areas 54L to 56R, differences in the way the object 52 is observed can be prevented. As a result, the object 52 is observed in the same way in each object display area 54L to 56R, making it easier for the operator to perceive the speed of the vehicle 12.

[0121] Furthermore, in this exemplary embodiment, an object display area 54L to 56R is set in a portion of the image display surface 42A of the display device 42 displaying peripheral images, and the object 52 is displayed in this object display area 54L to 56R. That is, in the existing remote operation device 14, the display of the object 52 can be added simply by changing the image display control program for the display device 42 displaying peripheral images. Therefore, the object 52 can be displayed in the existing remote operation device 14 at low cost without adding a new display device.

[0122] Furthermore, in this exemplary embodiment, since the moving speed of object 52 changes in accordance with the vehicle speed, the operator can easily and intuitively identify the change in vehicle speed.

[0123] (change)

[0124] The modifications to this exemplary embodiment will be explained. Since the only difference from the above exemplary embodiment is the detection of the operator's line of sight in the remote operating device 14, only the corresponding parts will be described. Furthermore, components that are the same as those in the above exemplary embodiment will be labeled with the same reference numerals, and detailed descriptions of these components will be omitted.

[0125] like Figure 14As shown, the remote operating device 114 has a camera 102 that captures the face of the operator seated in front of the image display surface 42A of the display device 42.

[0126] In addition, such as Figure 15 As shown, the CPU 40A of the remote control device 40 has the ability to detect the operator's gaze direction LV (refer to the image captured by the camera 102 of the operator's face) Figure 8 The line-of-sight detection unit 480.

[0127] Furthermore, the central field of view determination unit 420 determines the range of the field of view angle θ2 (20 degrees in this modification) on the image display surface 42A of the display device 42, with the line of sight LV detected by the line of sight detection unit 480 as the center, as the central field of view area 50.

[0128] (The purpose of this change)

[0129] In this modification, the remote operating device 114 determines the operator's gaze direction LV based on the image of the operator's face captured by the camera 102 instead of the steering angle, and determines the central field of view 50 and the object display areas 54L to 56R based on the gaze direction LV.

[0130] Therefore, in this modification, the central field of view area 50 on the image display surface 42A of the display device 42 is further aligned with the operator's central field of view.

[0131] In this way, the remote control device 114 can more reliably prevent the object 52 from entering the operator's central field of vision, the operator can visually confirm the surrounding field of vision (and can perform remote driving well), and the operator can obtain a sense of vehicle speed.

[0132] (Supplementary explanation of the illustrative implementation)

[0133] In the above exemplary embodiment, a portion of the image display surface 42A of the display device 42 of the remote operation device 14 is used as the object display area 54L to 56R, and the object 52, etc., is displayed overlapping with the surrounding image. However, it is also possible to consider setting up an object display display independently from the display device 42 that displays the surrounding image. For example, it is possible to consider... Figure 16 As shown, object display devices 110A and 110B, having a length equal to the entire width of the display device 42, are installed adjacent to the upper and lower parts of the display device 42 at a predetermined height. In this case, object display areas 54L to 56R are also provided on the left and right sides of the central viewing area 50 in each object display device 110A and 110B.

[0134] Furthermore, in the illustrated embodiment described above, the display device 42 is depicted as a bent display, but it is not limited to this. For example, it could also be as follows: Figure 17 As shown, the display devices 120, 122, and 124 are composed of three flat panel displays. Even in this case, it is as follows... Figure 18 As shown, continuous peripheral images are displayed on the image display surfaces 120A, 122A, and 124A of the three display devices 120, 122, and 124, and object display areas 54L to 56R are set at predetermined positions other than the central field of view area 50.

[0135] Furthermore, in the above exemplary embodiment, the object display areas 54L to 56R are disposed on the upper and lower sides of the image display surface 42A of the display device 42, but they may also be disposed on either side.

[0136] Furthermore, in the illustrated embodiment described above, a monochrome background 58 is overlaid across the entire object display area 54L to 56R to prevent the surrounding image from being visually confirmed by the operator, but this is not a limitation. Alternatively, only the object 52 may be displayed over the surrounding image.

[0137] Furthermore, in the above exemplary embodiments, the vehicle 12 is configured to perform both manual and remote driving, but it is not particularly limited as long as remote driving is possible. For example, the vehicle 12 may be configured to perform manual, automatic, and remote driving, or it may be configured to perform only remote driving.

[0138] Furthermore, in the above exemplary embodiment, the field of view angle θ2 of the central field of view is 20 degrees when viewed from above, but it is not limited to this.

[0139] Furthermore, in the illustrated embodiments described above, the processes executed by reading software (programs) from CPUs 20A, 40A, 60A, and 80A can also be executed by various processors other than CPUs. Examples of processors in this case include FPGAs (Field-Programmable Gate Arrays) and processors with dedicated circuit structures designed for executing specific processes, such as PLDs (Programmable Logic Devices) and ASICs (Application Specific Integrated Circuits). Additionally, each process can be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is a circuit that incorporates circuit elements such as semiconductor components.

[0140] Furthermore, in the illustrated embodiments described above, the program is pre-stored (installed) on a computer-readable non-transitory recording medium. For example, in the remote operating system 10, the program is pre-stored in storage 40D. However, it is not limited to this; the program may also be provided on a non-transitory recording medium such as a CD-ROM (Compact Disc Read Only Memory), DVD-ROM (Digital Versatile Disc Read Only Memory), or USB (Universal Serial Bus) memory. Additionally, each program may be downloaded from an external device via a network.

[0141] The processing flow described in the above exemplary embodiments is also an example, and unnecessary steps can be deleted, new steps can be added, and the processing order can be changed without departing from the main idea.

[0142] In addition, the present invention can be implemented with various modifications without departing from its spirit. Furthermore, the technical solutions of the present invention are not limited to the exemplary embodiments described above.

Claims

1. A remote control device for driving a vehicle by remote operation from outside the vehicle, wherein, The remote operation device includes: The image acquisition unit acquires images of the vehicle's surroundings from the vehicle. The first display unit displays an image of the vehicle's surroundings; The steering angle information acquisition unit acquires the steering angle information of the vehicle; The central field of view determination unit determines the central field of view of the first display unit based on the turning angle information; as well as The second display unit is disposed above, below, or both above and below the first display unit in a peripheral field of view area other than the central field of view area, and displays an object that moves from the central field of view area toward the end in the width direction.

2. The remote operation device according to claim 1, wherein, The central field of view determination unit determines the central field of view based on the distance from the operator's viewpoint of the remote operating device to the first display unit.

3. The remote operation device according to claim 1 or 2, wherein, The moving speed of the object varies depending on the speed of the vehicle.

4. A remote driving system, wherein, The vehicle is equipped with a remote operating device as described in any one of claims 1 to 3, and a vehicle driven by operating the remote operating device.

5. A remote operation method, wherein, The remote operation method enables a computer to perform processing in a remote operation device that drives a vehicle remotely from outside the vehicle, including the following steps: The image acquisition unit acquires images of the vehicle's surroundings from the vehicle. The first display unit displays an image of the vehicle's surroundings. The steering angle information acquisition unit acquires the steering angle information of the vehicle. The central field of view determination unit determines the central field of view of the first display unit based on the steering angle information. The second display unit displays an object that moves from the central viewing area side of the first display unit toward the end in the width direction. The second display unit is located above, below, or both of the first display unit in the peripheral viewing area other than the central viewing area.

6. A computer-readable medium having a remote operation program recorded thereon, the remote operation program being used to cause a computer in a remote operation device for driving a vehicle remotely from outside the vehicle to perform processing including the following steps: The image acquisition unit acquires images of the vehicle's surroundings from the vehicle. The first display unit displays an image of the vehicle's surroundings. The steering angle information acquisition unit acquires the steering angle information of the vehicle. The central field of view determination unit determines the central field of view of the first display unit based on the steering angle information. The second display unit displays an object that moves from the central viewing area side of the first display unit toward the end in the width direction. The second display unit is disposed above, below, or both above and below the first display unit in the peripheral viewing area other than the central viewing area.

Images