Remote Operation System

Abstract

To make a direction on video displayed on a monitor of a remote driving device agree with an operation direction of a remote operation member even when a vehicle travels backward.SOLUTION: A remote driving device comprises a display unit, a video switching unit, a remote operation member, and an operation amount setting unit. Video data on a periphery of a vehicle are acquired by front and rear cameras of the vehicle. The display unit displays video of the periphery of the vehicle based on the video data. The video switching unit takes, as the video on the display unit, video of the front of the vehicle acquired by the front camera when a direction of travel of the vehicle is a forward movement direction and video of the rear of the vehicle acquired by the rear camera when the direction of travel of the vehicle is a backward movement direction. The remote operation member remotely operates a steering gear of the vehicle. The operation amount setting unit sets an operation amount of the steering gear of the vehicle according to an operation amount of the remote operation member. When the vehicle travels backward, the operation amount setting unit sets the operation amount of the steering gear so that rotation directions of operation amount of the steering gear and the operation amount of the remote operation member are opposite to each other.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] This disclosure relates to a remote driving system for a vehicle. [Background technology]

[0002] Patent Document 1 describes a camera image display device used in a vehicle remote control system. In this remote control system, a remotely controlled vehicle captures images of the surroundings of the vehicle using television cameras mounted on the front, rear, left and right sides, and transmits the images to the remote control system. The camera image display device of the remote control system displays the received images of the surroundings of the vehicle on a monitor. At this time, the camera image display device switches the display state of the monitor image so that the image in the traveling direction or operating direction of the vehicle is enlarged. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 10-145777 Summary of the Invention [Problem to be solved by the invention]

[0004] Consider an operator watching an image displayed on a monitor and operating a remote control member of a remote driving system to remotely control a vehicle to drive it in reverse. If an image of the rear of the vehicle taken from the front by the vehicle's rear camera is displayed on the monitor as is, the steering direction of the wheels when driving in reverse (i.e., the steering direction of the vehicle's steering device) will be opposite to the left-right direction on the displayed screen. In this state, when driving the vehicle in reverse, the operator must operate the remote control member in the opposite direction to the direction shown in the image displayed on the monitor, which makes the remote control feel strange.

[0005] In order to solve the above-mentioned problems, the present disclosure aims to provide a technology that can match the left and right directions on the image displayed on the monitor with the operation direction of the remote control members of the remote driving device, even when the vehicle is reversing. [Means for solving the problem]

[0006] The present disclosure provides a remote driving device as an example. The remote driving device includes a receiving unit, a display unit, an image switching unit, a remote control member, and an operation amount setting unit. The receiving unit receives image data of the surroundings of the vehicle. The image data of the surroundings of the vehicle is acquired by a front camera mounted on the vehicle to acquire image data in front of the vehicle, and a rear camera mounted on the vehicle to acquire image data behind the vehicle. The display unit displays an image of the surroundings of the vehicle based on the image data. When the traveling direction of the vehicle is a forward direction, the image switching unit displays an image of the front of the vehicle based on the image data acquired by the front camera, and when the traveling direction of the vehicle is a backward direction, the image displayed on the display unit displays an image of the rear of the vehicle based on the image data acquired by the rear camera. The remote control member is a member for remotely operating a steering device of the vehicle. The operation amount setting unit sets an operation amount of the steering device of the vehicle according to an operation amount of the remote control member. Furthermore, when the vehicle is traveling backward, the operation amount setting unit sets the operation amount of the steering device so that the rotation direction of the operation amount of the steering device and the rotation direction of the operation amount of the remote control member are opposite to each other. Effect of the Invention

[0007] According to the remote driving device of the present disclosure, when the vehicle is traveling in reverse, the left-right direction of the image on the display unit and the left-right direction of the vehicle relative to the vehicle's traveling direction are reversed, but the amount of operation of the remote control member is converted so that the rotation direction is reversed and transmitted to the vehicle. Therefore, an operator who operates the remote control member can operate the steering device of the vehicle in a desired direction by operating the remote control member according to the left-right direction on the image displayed on the display unit. Therefore, the operator can remotely operate the vehicle without feeling any discomfort even when traveling in reverse. [Brief description of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an overview of a remote operation system according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a conceptual diagram for explaining an overview of remote control of a vehicle according to the first embodiment. [Diagram 3] FIG. 2 is a diagram for explaining a schematic configuration of a monitor of the remote operation device according to the first embodiment. [Figure 4] 5 is a diagram for explaining the relationship between the display on the monitor and the steering direction of the steering wheel of the remote control member when the vehicle is traveling backward in the first embodiment. FIG. [Diagram 5] 13 is a diagram for explaining the relationship between the display on the monitor and the steering direction of the steering wheel of the remote control member when the vehicle is traveling backward in another example of the first embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] 1. First embodiment 1-1. Configuration of remote operation system FIG. 1 is a conceptual diagram showing an overview of a remote driving system 1 according to the present embodiment. The remote driving system 1 includes a remote driving device 100 and a vehicle 200 that is the subject of remote operation (remote driving). The remote driving device 100 is a device for remotely operating the vehicle 200, and is used by an operator D. The remote driving device 100 and the vehicle 200 are connected to each other so as to be able to communicate with each other via a wireless communication network 10. The wireless communication network 10 is, for example, a 5G network.

[0010] The vehicle 200 is, for example, an automobile (e.g., a passenger car, a truck, and a bus) of various sizes and weights. The vehicle 200 is equipped with a steering device, a drive device, a braking device, an on-board electronic control unit (on-board ECU), a communication device, and the like. The steering device steers the wheels of the vehicle 200. The drive device generates a driving force for the vehicle 200, and the braking device generates a braking force for the vehicle 200.

[0011] Further, the vehicle 200 is equipped with a front camera 201 and a rear camera 202. The front camera 201 and the rear camera 202 are installed facing the front and rear sides of the vehicle 200, respectively, and capture the surroundings of the vehicle on the front and rear sides of the vehicle 200. The image data obtained by the front camera 201 and the rear camera 202 is transmitted to the remote driving device 100 via a communication device. In this embodiment, there is no limit to the number of cameras mounted on the vehicle 200 as long as the surrounding environment of the vehicle 200 can be captured. For example, the vehicle 200 may be equipped with a plurality of either the front camera 201 or the rear camera 202. The vehicle 200 may be equipped with a left camera and a right camera. In this case, the left camera captures the surroundings of the vehicle on the left side of the vehicle 200, and the right camera captures the surroundings of the vehicle on the right side of the vehicle 200, and transmits the obtained image data to the remote driving device 100.

[0012] The remote driving device 100 includes a monitor 110, a remote control member 120, a control device 130, and a receiving unit 140. The receiving unit 140 receives various information transmitted from the vehicle 200. The received information includes video data captured by a front camera 201 and a rear camera 202.

[0013] The remote control member 120 is a member operated by the operator D when remotely operating the vehicle 200. The remote control member 120 includes a steering wheel 121. Although not shown, the remote control member 120 includes, in addition to the steering wheel 121, an accelerator pedal and a brake pedal, for example. Note that the member for remotely operating the steering device is not limited to the steering wheel 121. For example, the remote driving device 100 may include a joystick or the like as the remote control member 120, which allows the driving of the vehicle 200 to be remotely controlled.

[0014] 2 is a conceptual diagram for explaining an overview of remote control of the vehicle 200. An operator D can remotely control the steering device of the vehicle 200 by operating the steering wheel 121. The operation amount of the steering wheel 121 by the operator D is a steering angle. Hereinafter, for convenience, the operation amount of the steering wheel 121 is referred to as a "first operation amount." The first operation amount of the steering wheel 121 represents the intention of the operator D.

[0015] The control device 130 has a function as an operation amount setting unit. When the control device 130 functions as an operation amount setting unit, it calculates a second operation amount by reflecting a first operation amount of the steering 121, and transmits the second operation amount to the vehicle 200 via the wireless communication network 10. The second operation amount calculated here may be the first operation amount itself, or may be a corrected operation amount obtained by correcting the operation amount.

[0016] The control device 220 of the vehicle 200 receives the second operation amount from the remote driving device 100. The control device 220 of the vehicle 200 controls the steering device 210 of the vehicle 200 in accordance with the second operation amount. The steering device 210 of the vehicle 200 may include, for example, an electric power steering such as a rack-parallel EPS.

[0017] The vehicle 200 also has an operation unit including a steering wheel, an accelerator pedal, and a brake pedal. The driver of the vehicle 200 can operate (drive) the vehicle 200 by operating the operation unit of the vehicle 200. In the case of manual driving, the control device of the vehicle 200 operates the vehicle 200 according to the amount of operation of the operation unit by the driver. On the other hand, in the case of remote operation (remote driving), the control device 220 of the vehicle 200 operates the vehicle 200 according to operation instruction information from the remote driving device 100. The vehicle 200 may be capable of switching between manual driving and automatic driving. The remote operation of the vehicle 200 may be performed to assist the automatic driving of the vehicle 200.

[0018] FIG. 3 is a diagram for explaining a schematic configuration of the monitor 110 of the remote driving device 100. In the example shown in FIG. 3, the monitor 110 is composed of a left monitor 111, a center monitor 112, and a right monitor 113. However, the number of monitors constituting the monitor 110 is not limited to three. In FIG. 3, the monitor 110 displays an image of the surroundings of the vehicle based on image data acquired by a front camera 201 and a rear camera 202 mounted on the vehicle 200. The center monitor 112 is a main display unit, and displays an image of the traveling direction of the vehicle 200. The left monitor 111 displays an image of the left or right side of the vehicle 200. The right monitor 113 displays an image of the left or right side of the vehicle 200. The operator D can operate the steering wheel 121 while watching the image displayed on the monitor 110.

[0019] The control device 130 of the remote driving device 100 also has a function as an image switching unit that switches the image displayed on the monitor 110. When the vehicle 200 travels forward, the control device 130 causes the monitor 110 to display an image of the surroundings of the vehicle 200 acquired by the forward camera 201. The example of FIG. 3 is an example of a display during forward travel, in which an image of the front of the vehicle 200 is displayed on the central monitor 112, an image of the surroundings on the left side of the vehicle 200 that follows the left end of the image on the central monitor 112 is displayed on the left monitor 111, and an image of the surroundings on the right side of the vehicle 200 that follows the right end of the image on the central monitor 112 is displayed on the right monitor 113.

[0020] 1-2. Control when reversing the vehicle The control device 130 of the remote driving device 100 can determine whether the vehicle 200 is moving forward or backward. Specifically, the control device 130 determines that the vehicle 200 is moving backward based on information from the remote cockpit, and changes the mode of the vehicle 200 to the reverse mode. Alternatively, the control device 130 determines that the vehicle 200 is moving backward based on a shift operation by the operator D, and changes the mode of the vehicle 200 to the reverse mode.

[0021] 4 is a diagram for explaining the relationship between the display on the monitor 110 and the steering direction of the steering wheel 121 when driving backward. When the vehicle 200 is switched to the reverse mode, the control device 130 switches the image on the central monitor 112 to the image from the rear camera 202. The displays on the left monitor 111 and the right monitor 113 are not switched, and the left monitor 111 and the right monitor 113 display the surrounding images on the left and right sides of the vehicle 200 captured by the front camera 201, which are the same as those when driving forward.

[0022] The rear camera 202 captures an image of the rear side of the vehicle 200 from the front side (i.e., the vehicle 200 side). Therefore, as shown in Fig. 4, the image from the rear camera 202 is displayed as is on the central monitor 112, and the left and right directions confirmed on the screen of the central monitor 112 are opposite to the steering direction of the wheels of the vehicle 200. Therefore, for example, as shown in Fig. 4, when the vehicle 200 is traveling backward and the driver tries to turn the vehicle 200 to the right of the screen while watching the monitor 110, the steering direction of the wheels of the actual vehicle needs to be the left rotation direction.

[0023] Therefore, in this embodiment, when the vehicle 200 is traveling backward, the control device 130 calculates an operation amount in which the rotation direction of the first operation amount, which is the operation amount of the steering wheel 121, is reversed from left to right, and transmits this to the vehicle 200 as the second operation amount. As a result, when the vehicle 200 is traveling backward, the turning direction of the vehicle 200 that can be confirmed on the central monitor 112 and the operation direction of the steering wheel 121 are made to match. In other words, the operator D can turn the vehicle 200 in a desired direction even when traveling backward by rotating the steering wheel 121 in the direction displayed on the screen while checking the central monitor 112, in the same way as when the vehicle 200 is traveling forward.

[0024] The first operation amount is reversed only when the steering wheel 121 is in the neutral position. In order to ensure that the first operation amount is reversed when traveling backward, the steering wheel 121 and the tire angle of the vehicle 200 are automatically transitioned to the neutral position when traveling backward.

[0025] When the first operation amount is reversed, a guard value is set for the change amount of the steering device of the vehicle 200 to suppress a sudden change in the tire angle. After the reversal instruction for the first operation amount is given, the vehicle is kept stopped until the change in the tire angle is completed.

[0026] As described above, according to this embodiment, even when driving backward, the vehicle 200 can be remotely controlled by operating the steering wheel 121 in the same direction as the screen while checking the screen displayed on the central monitor 112. This allows the vehicle 200 to be remotely controlled to travel forward or backward without distinction, particularly when the vehicle 200 to be remotely controlled is configured symmetrically on the front and rear and on the left and right and has no concept of forward or backward.

[0027] 2. Other embodiments When traveling backward, instead of controlling the first operation amount to be inverted left and right and set as an instruction value to the steering device of the vehicle 200, the screen of the central monitor 112 may be inverted left and right.

[0028] FIG. 5 is a diagram for explaining the relationship between the display on the monitor 110 and the steering direction of the steering wheel 121 during backward driving. The screen displayed on the central monitor 112 in FIG. 5 is in the same location as the screen displayed on the central monitor 112 in FIG. 4, but the screen in FIG. 5 is a left-right inverted version of the image in FIG. 4. In this way, by displaying the image from the rear camera 202 in a left-right inverted manner, the direction in which the wheels are steered during backward driving can be made to match the steering direction of the steering wheel 121. This allows the operator D to operate the steering wheel 121 while watching the central monitor 112 in the same way as when the vehicle 200 is traveling forward.

[0029] 5, when the screen of the central monitor 112 is switched to an inverted image of the rear of the vehicle 200 during reverse driving, the images of the left monitor 111 and the right monitor 113 may also be switched. In this case, for example, the screen displayed on the left monitor 111 and the screen displayed on the right monitor 113 may be switched. At this time, the screens displayed on the left monitor 111 and the right monitor 113 may also be displayed in a horizontally inverted manner.

[0030] In addition, if the remote driving device 100 is configured to be able to display a navigation screen that displays a car navigation map, and if the screen of the central monitor 112 is displayed in a left-right inverted manner when driving backward, the map displayed on the navigation screen may also be displayed in a left-right inverted manner. Similarly, if the remote driving device 100 is configured to have a voice guidance device that provides voice guidance for remote operation, and if the screen of the central monitor 112 is displayed in a left-right inverted manner when driving backward, the left-right direction of the voice instructions may be reversed to provide guidance.

[0031] Furthermore, the remote driving device 100 may be configured to be capable of executing both a first mode in which the first operation amount is transmitted to the vehicle 200 with left-right inversion during reverse driving, and a second mode in which the display on the monitor 110 is reversed left-right. In this case, the remote driving device 100 may be configured to be able to select the mode during reverse driving between the first mode and the second mode.

[0032] For example, in the first mode, the first operation amount is reversed, so that the delay is likely to be large compared to the second mode. Therefore, for example, in the case of parking the vehicle 200, the second mode is more convenient in the case of an operation in which switching between forward and reverse is frequent.

[0033] On the other hand, in the second mode, the screen of the central monitor 112 is inverted left and right when driving backward, so that the text, signs, etc. are also inverted left and right. This may reduce the readability of the text, signs, etc. displayed on the monitor 110. Therefore, the first mode is more convenient when driving a relatively long distance while driving backward, for example. Also, the first mode is more convenient for vehicles that do not have the concept of forward and reverse as described above, and do not switch between forward and reverse for a while after the traveling direction is determined when the IG is turned on. [Explanation of symbols]

[0034] 1 Remote Operation System 10 Wireless communication networks 100 Remote control device 110 Monitor 111 Left Monitor 112 Central Monitor 113 Right Monitor 120 Remote control components 121 Steering 130 Control device 200 vehicles 201 Front Camera 202 Rear Camera 210 Steering Gear 220 Control device

Claims

1. a receiving unit that receives image data of the surroundings of the vehicle acquired by a front camera mounted on the vehicle and acquiring image data of the front of the vehicle, and a rear camera mounted on the vehicle and acquiring image data of the rear of the vehicle; a display unit that displays an image of the surroundings of the vehicle based on the image data; A remote control member for remotely controlling a steering device that steers the wheels of the vehicle; A control device; Equipped with The control device includes: an image switching unit that switches the image to be displayed on the display unit to an image of the front of the vehicle based on image data acquired by the front camera when the traveling direction of the vehicle is a forward direction, and switches the image to an image of the rear of the vehicle based on image data acquired by the rear camera when the traveling direction of the vehicle is a backward direction; an operation amount setting unit that sets an operation amount of the steering device in accordance with an operation amount of the remote control member; Equipped with The operation amount setting unit is a first setting state in which an operation amount of the steering device is set so that a steering direction of the wheels by the steering device and a steering direction of the remote control member are opposite to each other, and a second setting state in which an operation amount of the steering device is set so that a steering direction of the wheels by the steering device and a steering direction of the remote control member are the same, The video switching unit is When displaying an image behind the vehicle on the display unit, if the operation amount setting unit sets the steering amount of the steering device in the first setting state, the image can be displayed on the display unit in a first state that is not inverted left and right, and if the operation amount setting unit sets the steering amount of the steering device in the second setting state, the image can be displayed on the display unit in a second state that is inverted left and right, The control device includes: When the vehicle is moving backwards, a first mode in which an image of the rear of the vehicle is displayed on the display unit in the first state, and an operation amount of the steering device is set in the first setting state; a second mode in which an image of the rear of the vehicle is displayed on the display unit in the second state and an operation amount of the steering device is set in the second setting state; A remote driving device characterized by:

2. Further comprising a voice guidance device for providing voice guidance on the remote operation, The voice guidance device includes: When the second mode is selected during reverse driving of the vehicle and an image of the rear of the vehicle is displayed on the display unit in the second state, a left-right direction of the voice instruction is reversed. The remote driving device according to claim 1.

Images