Remote control device and remote control system

Abstract

To provide a remote operation device and a remote operation system that enable an operator who remotely operates a mobile body to easily and accurately grasp the actual movement of the mobile body during steering.SOLUTION: A remote operation device 20 includes: a steering unit 36 for steering a mobile body 12; a reception unit 26 for receiving actual steering information on the mobile body 12; and a display unit 22 for displaying steering information for steering the steering unit 36 and the actual steering information on the mobile body 12 received by the reception unit 26. A remote operation system 10 includes the mobile body 12 and the remote operation device 20 for remotely operating the mobile body 12.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a remote control device and a remote control system. [Background technology]

[0002] 2. Description of the Related Art A remote control system including a moving body and a remote control device that controls the moving body from outside the moving body has been conventionally known (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-107374 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in a remote control system in which a moving body is remotely controlled from a cockpit installed outside the moving body, a communication delay between the cockpit and the moving body causes a discrepancy between the steering timing of the operator at the cockpit and the steering timing of the moving body that is actually traveling. Therefore, it is difficult for the operator to grasp the relationship between the steering at the cockpit and the actual movement of the moving body. As a result, the operator may make unnecessary steering corrections to the moving body, which may cause the behavior of the moving body to become unstable.

[0005] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide a remote control device and a remote control system that enable an operator who remotely controls a moving object to easily and accurately grasp the actual movement of the moving object when steering. [Means for solving the problem]

[0006] In order to achieve the above object, a remote control device according to a first aspect of the present invention includes a steering unit for steering a moving object, and actual steering information of the moving object. and information based on an image captured by an imaging device provided in the moving object.a reception unit for receiving the signal, and a steering unit for operating the steering unit. Made by and the actual steering information of the moving body received by the receiving unit. and combining it with information based on an image captured by the imaging device. and a display unit for displaying the same.

[0007] According to the invention described in claim 1, the steering unit is operated. Made by and the steering information of the actual moving body received by the reception unit. The information is synthesized with information based on an image captured by an imaging device installed in the moving object. The steering information is displayed on the display unit. Therefore, even if a delay in communication between the cockpit and the moving body causes a discrepancy between the steering timing of the operator at the cockpit and the steering timing of the moving body that is actually traveling, the operator who remotely operates the moving body can easily accurately grasp the actual movement of the moving body when steering by looking at the steering information displayed on the display unit.

[0008] The remote control device according to claim 2 is the remote control device according to claim 1, further comprising: Made by The steering information for steering the moving body to the target position is set as a first trajectory line, and the actual steering information of the moving body received by the receiving unit is set as a second trajectory line.

[0009] According to the invention described in claim 2, the steering unit is operated. Made by The steering information for steering the moving object is set as the first locus line, and the steering information of the actual moving object received by the receiving unit is set as the second locus line. Therefore, compared to a case where each steering information is not linear, it is easier for an operator who remotely operates the moving object to intuitively and accurately grasp the movement of the actual moving object when steering.

[0010] Further, a remote control device according to claim 3 is the remote control device according to claim 2, wherein when an overlap amount between the first trajectory line and the second trajectory line is equal to or greater than a threshold value, a display mode of the first trajectory line and the second trajectory line is changed.

[0011] According to the invention described in claim 3, when the overlap amount between the first locus line and the second locus line is equal to or greater than a threshold value, the display mode of the first locus line and the second locus line is changed. Therefore, compared to a case where the display mode of the first locus line and the second locus line is not changed, an operator who remotely operates the moving body can easily intuitively and accurately grasp the actual movement of the moving body during steering.

[0012] A remote control system according to a fourth aspect of the present invention includes a moving body, a steering unit for steering the moving body, and actual steering information of the moving body. and information based on an image captured by an imaging device provided in the moving object. a reception unit for receiving the signal, and a steering unit for operating the steering unit. Made by and the actual steering information of the moving body received by the receiving unit. and combining it with information based on an image captured by the imaging device. and a remote control device for remotely controlling the moving body.

[0013] According to the invention described in claim 4, a remote control device for remotely controlling a moving body controls a steering unit. Made by and the steering information of the actual moving body received by the reception unit. The information is synthesized based on an image captured by an imaging device installed in the moving object. Therefore, even if a delay in communication between the cockpit and the moving body causes a discrepancy between the steering timing of the operator at the cockpit and the steering timing of the moving body that is actually traveling, the operator who remotely operates the moving body can easily accurately grasp the actual movement of the moving body when steering by looking at the steering information displayed on the display unit. Effect of the Invention

[0014] As described above, according to the present invention, an operator who remotely operates a moving object can easily and accurately grasp the actual movement of the moving object when steering. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram showing a configuration of a remote control system according to an embodiment of the present invention. [Diagram 2]4 is an explanatory diagram showing a first trajectory line and a second trajectory line when traveling straight, which are displayed on a display unit of the remote control system according to the present embodiment. FIG. [Diagram 3] 4 is an explanatory diagram showing a first locus line and a second locus line displayed on a display unit of the remote control system according to the present embodiment when starting to make a left curve. FIG. [Figure 4] 4 is an explanatory diagram showing a first locus line and a second locus line at the end of left curve traveling, which are displayed on a display unit of the remote control system according to the present embodiment; FIG. [Diagram 5] 4 is a flowchart showing a processing process on the remote cockpit side of the remote control system according to the present embodiment. [Figure 6] 11 is a graph showing a deviation caused by a communication delay between the steering timing on the remote cockpit side and the steering timing on the vehicle side of the remote control system according to this embodiment, and the video feedback displayed on the display unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. As shown in Fig. 1, a remote control system 10 according to this embodiment includes a vehicle 12 as an example of a moving body, and a remote cockpit 20 as a remote control device having a cockpit in which a driver (operator) who operates the vehicle 12 remotely from outside the vehicle sits.

[0017] The vehicle 12 is equipped with a control unit 14 electrically connected to a drive actuator 15. That is, the control unit 14 receives operation instructions (control data) transmitted from a remote cockpit 20, and drives the drive actuator 15 based on the instructions to control actual driving, such as operation of the accelerator pedal (driving), operation of the steering wheel (steering), and operation of the brake pedal (stopping).

[0018] The vehicle 12 also includes a communication unit 16 having a receiving unit that receives operation instructions (control data) from a remote cockpit 20 via an antenna 18, and a transmitting unit that transmits steering information required for remote driving to the remote cockpit 20. The communication unit 16 is electrically connected to the control unit 14.

[0019] The vehicle 12 is also equipped with a camera 17 that captures images of the surroundings of the vehicle 12. The cameras 17 are provided at a plurality of appropriate positions on the vehicle 12. Note that the steering information required for remote driving, which is transmitted by the transmitting unit in the communication unit 16, also includes images (image data) captured by the camera 17. The camera 17 is also electrically connected to the control unit 14.

[0020] The remote cockpit 20 is equipped with a communication unit 26 having a receiving unit that receives steering information required for remote driving from the communication unit 16 via an antenna 28, and a transmitting unit that transmits operation instructions (control data) from the remote cockpit 20 to the communication unit 16. This communication unit 26 is an example of a receiving unit that receives steering information from the vehicle 12.

[0021] The remote cockpit 20 also includes a display unit 22 that displays actual steering information received by a communication unit 26, such as images captured by a plurality of cameras 17 provided on the vehicle 12. The display unit 22 is composed of a plurality of displays and LED devices, and is also configured to display various meters that show the state of the vehicle 12, such as a speedometer 40 (see Figs. 2 to 4) that shows the speed of the vehicle 12.

[0022] That is, the display unit 22 is capable of displaying the state of the vehicle 12, such as when it is moving or when it is stopped, in addition to the display of various meters such as the speedometer 40, and is capable of visually notifying the driver of the state of the vehicle 12 that is actually moving on the road. The display unit 22 is also capable of displaying steering information for steering a steering wheel 36, which will be described later.

[0023] The remote cockpit 20 also includes a control unit 24 electrically connected to the display unit 22 and the communication unit 26. The control unit 24 transmits operation instructions (control data) on the remote cockpit 20 side to the vehicle 12 side, receives steering information required for remote driving from the vehicle 12 side, and controls the remote cockpit 20 side by displaying the steering information on the display unit 22, for example.

[0024] The control unit 24 is also electrically connected to an operating device 30 for driving the vehicle 12. The operating device 30 has an accelerator pedal 32 for the driver to operate the accelerator, a brake pedal 34 for the driver to operate the brakes, a steering wheel 36 for the driver to operate the steering wheel, and switches (SW) 38 for the driver to perform other necessary operations.

[0025] That is, the control unit 24 transmits various actual driving operations performed by the driver on the remote cockpit 20 side to the vehicle 12 side via the communication unit 26. The steering wheel 36 is an example of a steering unit that steers the vehicle 12. The steering wheel 36 uses steer-by-wire (electronically controlled steering mechanism) technology, allowing the driver to obtain steering feedback (steering reaction force).

[0026] Here, the steering information displayed on the display unit 22 will be described.

[0027] 2 to 4, the display unit 22 displays a road D on which the vehicle 12 is traveling. A first trajectory line 42, which is a predicted trajectory indicating the direction in which the vehicle 12 should advance, and a second trajectory line 44, which is a predicted trajectory when the vehicle 12 actually travels, are superimposed and displayed at the center position in the road width direction of the road D. In other words, the first trajectory line 42 is steering information for steering the steering wheel 36, and the second trajectory line 44 is steering information of the actual vehicle 12 received by the communication unit 26.

[0028] The first locus line 42 is shown by a solid line of a predetermined length and thicker than the second locus line 44, and is displayed on the display unit 22 after being transformed into a straight line or a curve in accordance with the condition of the road D on which the vehicle 12 is traveling. The second locus line 44 is shown by a broken line of a predetermined length and thinner than the first locus line 42, and is displayed on the display unit 22 after being transformed into a straight line or a curve in conjunction with the actual steering amount of the vehicle 12 corresponding to the steering amount of the steering wheel 36 following the first locus line 42.

[0029] More specifically, the first locus line 42 and the second locus line 44 have the same length. As shown in Fig. 2, when the vehicle 12 is traveling on a straight road D, the first locus line 42 is a straight line, and the driver accordingly places the steering wheel 36 in the neutral position. As a result, the second locus line 44 is a straight line, and the second locus line 44 overlaps with the first locus line 42.

[0030] On the other hand, as shown in Fig. 3, when the vehicle 12 travels on road D that curves to the left, the first locus line 42 is set as a curve that curves to the left, and the driver steers the steering wheel 36 from the neutral position to the left in accordance with the first locus line 42. As a result, the second locus line 44 is set as a curve that curves to the left, but due to a communication delay, the first locus line 42 precedes and the second locus line 44 follows. Then, after a predetermined time (e.g., 0.2 seconds) has elapsed, the second locus line 44 catches up with and overlaps with the first locus line 42, as shown in Fig. 4.

[0031] The first locus line 42 and the second locus line 44 are shown by different lines so that it is easy to see that they overlap each other. Therefore, the first locus line 42 and the second locus line 44 may differ not only in the type and width of the line as described above, but also in the color and shade of the line. The type, width, color, etc. of the line of the first locus line 42 and the second locus line 44 may be set in advance by the driver, or may be automatically set by the control unit 24.

[0032] Furthermore, when the amount of overlap between the first trajectory line 42 and the second trajectory line 44 is equal to or greater than a threshold value, the display manner of the first trajectory line 42 and the second trajectory line 44 may be changed. For example, the first trajectory line 42 may be displayed in blue and the second trajectory line 44 in red, and when the first trajectory line 42 and the second trajectory line 44 overlap by 50% or more in terms of their respective widths, the overlapping portion may be displayed in purple or in a completely different color.

[0033] Furthermore, when the first locus line 42 and the second locus line 44 overlap by 80% or more in terms of their width, they may be displayed as a single line of a completely different line type, width, and color, rather than as two overlapping lines. Note that the threshold value for the amount of overlap between the first locus line 42 and the second locus line 44 is not limited to the above 50% and 80%, but is set appropriately by the driver or the control unit 24.

[0034] Next, the operation of the remote cockpit 20 and the remote control system 10 according to this embodiment configured as above will be described.

[0035] In Fig. 6, the steering amount of the steering wheel 36 on the remote cockpit 20 side is shown with a solid line versus time, and the steering amount of the steering wheel on the vehicle 12 side is shown with a dashed line versus time. Furthermore, in Fig. 6, the video feedback of the steering wheel on the vehicle 12 side is shown with a dashed line. The video feedback is displayed on the display unit 22.

[0036] 6(A), due to a communication delay between the remote cockpit 20 and the vehicle 12, the steering timing from the neutral position of the steering wheel on the vehicle 12 side is delayed by time T1 compared to the steering timing from the neutral position of the steering wheel 36 on the remote cockpit 20 side. In other words, time T1 is the time until the steering of the steering wheel 36 on the remote cockpit 20 side is reflected in the steering of the steering wheel on the vehicle 12 side.

[0037] And, because the image data has a larger capacity than the control data, the communication delay causes the image feedback of the steering wheel of the vehicle 12 (the timing of steering from the neutral position) displayed on the display unit 22 of the remote cockpit 20 to be delayed by time T2 compared to the timing of steering from the neutral position of the steering wheel of the vehicle 12. In other words, time T2 is the time until the steering of the steering wheel of the vehicle 12 is reflected in the image feedback.

[0038] Therefore, even if the steering of the steering wheel 36 on the remote cockpit 20 side is completed (even if the steering wheel 36 on the remote cockpit 20 side reaches the steering amount shown at time Te), the steering of the steering wheel on the vehicle 12 side in the video feedback displayed on the display unit 22 is in an incomplete state (the steering amount of the steering wheel on the vehicle 12 side in the video feedback shown at time Te has not reached the steering amount of the steering wheel 36 on the remote cockpit 20 side).

[0039] In other words, it takes time T1+time T2 for the steering of the steering wheel 36 on the remote cockpit 20 side to be reflected in the video feedback. Therefore, the driver on the remote cockpit 20 side may want to further steer the steering wheel 36 when he or she sees the video feedback displayed on the display unit 22, which may cause the behavior of the actual vehicle 12 to become unstable.

[0040] The same is true when returning the steering wheel 36 on the remote cockpit 20 side to the neutral position. That is, as shown in Fig. 6(B), due to a communication delay between the remote cockpit 20 and the vehicle 12, the timing of steering the steering wheel on the vehicle 12 side to the neutral position is delayed by a time T1 compared to the timing of steering the steering wheel 36 on the remote cockpit 20 side to the neutral position.

[0041] Furthermore, since image data has a larger capacity than control data, the communication delay causes the image feedback of the steering wheel on the vehicle 12 side (the timing of steering to the neutral position) displayed on the display unit 22 on the remote cockpit 20 side to be delayed by a time T2 compared to the actual timing of steering the steering wheel on the vehicle 12 side to the neutral position.

[0042] Therefore, even if the steering wheel 36 on the remote cockpit 20 side has been completely steered to the neutral position (even if the steering wheel 36 on the remote cockpit 20 side has reached the neutral position shown at time Tc), the steering of the steering wheel on the vehicle 12 side in the video feedback displayed on the display unit 22 is in an incomplete state (the steering amount of the steering wheel on the vehicle 12 side in the video feedback shown at time Tc has not yet reached the neutral position).

[0043] That is, in this case, it takes time T1+time T2 for the steering of the steering wheel 36 on the remote cockpit 20 side to be reflected in the video feedback. Therefore, the driver on the remote cockpit 20 side may want to further steer the steering wheel 36 when he or she sees the video feedback displayed on the display unit 22, which may cause the behavior of the actual vehicle 12 to become unstable.

[0044] Therefore, in the remote cockpit 20 and remote operation system 10 of this embodiment, as described above, the display unit 22 displays the first trajectory line 42, which is steering information for steering the steering wheel 36, and the second trajectory line 44, which is steering information of the actual vehicle 12 received by the communication unit 26, so that the driver can easily and accurately grasp the state (movement) of the actual vehicle 12 visually.

[0045] 5, in the remote cockpit 20, the control unit 24 first acquires the steering amount of the steering wheel 36 in the remote cockpit 20 (step S1). That is, it is determined whether the steering wheel 36 in the remote cockpit 20 has been steered from the neutral position (step S2).

[0046] If it is determined that the steering wheel 36 on the remote cockpit 20 side has not been steered from the neutral position (no change), the display unit 22 continues to display the current first trajectory line 42 and second trajectory line 44, which are in a straight-ahead state as shown in Figure 2 (step S3), and the process returns to step S1 again.

[0047] When it is determined that the steering wheel 36 on the remote cockpit 20 side has been steered from the neutral position (changed), the amount of steering from the neutral position of the steering wheel on the vehicle 12 side is acquired (step S4). Then, when the steering wheel 36 on the remote cockpit 20 side is steered, it is determined whether or not there is a difference between the amount of steering of the steering wheel 36 on the vehicle 12 side due to a communication delay (step S5).

[0048] When the steering wheel 36 on the remote cockpit 20 side is steered, if there is a difference between the amount of steering and the amount of steering of the steering wheel on the vehicle 12 side, the position of the second trajectory line 44 relative to the position of the first trajectory line 42 is displayed on the display unit 22 according to the amount of the difference, and the display method is set (step S6).

[0049] 2 to 4, in this embodiment, the first locus line 42 is set as a thick solid line, and the second locus line 44 is set as a thin dashed line by the driver or the control unit 24. Therefore, as shown in Fig. 3 and Fig. 4, the first locus line 42 set as a thick solid line and the second locus line 44 set as a thin dashed line are respectively displayed on the display unit 22 (step S7).

[0050] In addition, when the steering wheel 36 on the remote cockpit 20 side is steered, if there is no difference between the steering amount and the steering amount of the steering wheel on the vehicle 12 side, the first trajectory line 42 and the second trajectory line 44 always remain superimposed, so that only one trajectory line as shown in Figure 4 is displayed on the display unit 22, and the display method is set (step S8).

[0051] For example, as shown in Fig. 4, the first trajectory line 42 and the second trajectory line 44 are superimposed to form a single trajectory line, or, although not shown, the first trajectory line 42 and the second trajectory line 44 are formed into a single completely separate trajectory line. Then, the trajectory line is displayed on the display unit 22 (step S7). In this embodiment, the above-mentioned process is repeatedly executed, and the first trajectory line 42 and the second trajectory line 44 (or another trajectory line) are always displayed on the display unit 22.

[0052] Therefore, even if a communication delay between the remote cockpit 20 and the vehicle 12 causes a discrepancy between the steering timing (steering amount) of the driver at the remote cockpit 20 and the steering timing (steering amount) of the vehicle 12 that is actually traveling, the driver can easily and accurately grasp the actual movement of the vehicle 12 when steering the steering wheel 36 by looking at the first trajectory line 42 and the second trajectory line 44 displayed on the display unit 22 (by recognizing the difference in the positions of the two trajectory lines).

[0053] That is, even if a communication delay occurs between the driver who remotely operates the vehicle 12 and the actual vehicle 12, the driver can wait until the first trajectory line 42 overlaps with the second trajectory line 44, thereby understanding the movement of the vehicle 12. Therefore, the driver can be restrained or prevented from making unnecessary steering corrections to the vehicle 12, and the behavior of the vehicle 12 can be restrained or prevented from becoming unstable.

[0054] 2 to 4, the steering information for steering the steering wheel 36 is represented, for example, by a thick solid first trajectory line 42, and the steering information of the actual vehicle 12 received by the communication unit 26 is represented, for example, by a thin dashed second trajectory line 44. Therefore, compared to a case where each piece of steering information is not linear, the driver who remotely operates and drives the vehicle 12 can intuitively, easily, and accurately grasp the movement of the actual vehicle 12 when the steering wheel 36 is steered.

[0055] Furthermore, when the overlap amount between the first trajectory line 42 and the second trajectory line 44 is equal to or greater than a threshold value, the display mode of the first trajectory line 42 and the second trajectory line 44 is changed. Therefore, compared to a case where the display mode is not changed, the driver who remotely operates the vehicle 12 to drive the vehicle 12 can intuitively, easily, and accurately grasp the actual movement of the vehicle 12 when the steering wheel 36 is steered.

[0056] In this way, according to the remote cockpit 20 and remote operation system 10 of this embodiment, the driver on the remote cockpit 20 side can easily, accurately, and even intuitively grasp the state (movement) of the vehicle 12 visually from the display unit 22, regardless of communication delays (delay or stop of the image, reduced resolution, or reduced frame rate).

[0057] Although the remote cockpit 20 and the remote control system 10 as the remote control device according to the present embodiment have been described above with reference to the drawings, the remote cockpit 20 and the remote control system 10 according to the present embodiment are not limited to those shown in the drawings, and can be appropriately modified in design within the scope of the present invention. For example, the moving body is not limited to the vehicle 12.

[0058] In addition, the steering information for steering the steering wheel 36 is not limited to the first locus line 42, and the steering information of the actual vehicle 12 received by the communication unit 26 is not limited to the second locus line 44. The steering information for steering the steering wheel 36 and the steering information of the actual vehicle 12 received by the communication unit 26 may be, for example, numbers or the like.

[0059] That is, the display unit 22 may be configured to display 0 when the road D is straight (the steering wheel 36 is in the neutral position), and display numbers 1 to 5 according to the degree of curvature of the road D. In this case, it is preferable that the number 1 to 5 is followed by the letter R when turning right, and the number 1 to 5 is followed by the letter L when turning left. [Explanation of symbols]

[0060] 10 Remote Control System 12 Vehicles (moving objects) 20 Remote cockpit (remote control device) 22 Display section 26 Communications Department (Reception Department) 36 Steering wheel (steering section) 42 1st trajectory line 44 Second trajectory line

Claims

1. A steering unit that steers the moving body; a reception unit that receives actual steering information of the moving body and information based on an image captured by an imaging device provided in the moving body; a display unit that displays steering information for operating the steering unit and the actual steering information of the moving body received by the reception unit by combining the steering information with information based on an image captured by the imaging device; A remote control device comprising:

2. 2. The remote control device according to claim 1, wherein steering information for operating the steering unit is defined as a first trajectory line, and actual steering information of the moving body received by the receiving unit is defined as a second trajectory line.

3. The remote control device according to claim 2 , wherein a display mode of the first trajectory line and the second trajectory line is changed when an overlap amount between the first trajectory line and the second trajectory line is equal to or greater than a threshold value.

4. A moving body, a remote control device for remotely controlling the moving body, the remote control device comprising: a steering unit for steering the moving body; a reception unit for receiving actual steering information of the moving body and information based on an image captured by an imaging device provided on the moving body; and a display unit for displaying the steering information for operating the steering unit and the actual steering information of the moving body received by the reception unit, synthesized with information based on the image captured by the imaging device; Equipped with a remote control system.

Images