Remote operation device and remote operation system

Abstract

The purpose of the present disclosure is to provide a remote operation device and a remote operation system with which it is possible to perform the task of interlinking a towing vehicle and a towed vehicle without dispatching an assistant to a site even in the event of failure in the interlinking task as performed by automatic control. A remote operation device according to the present disclosure comprises: an operation detection unit that acquires, as an operation amount, an operation performed by an operator on a towing vehicle that has been switched to remote operation control when a predetermined condition has been met; an input detection unit that receives remote operation information including a setting of whether or not to perform remote operation; a transmitting unit that transmits the operation amount and the remote operation information; a reception unit that receives rear image information for the rear of the towing vehicle and sensor information for the towing vehicle detected by an internal sensor; a trajectory calculation unit that calculates, as trajectory information, a predicted trajectory of a predetermined portion of the towing vehicle on the basis of the operation amount, the sensor information, and a model of the motion of the towing vehicle; and an image generation unit that generates a presentation image to be presented to the operator by superimposing the trajectory information on the rear image information.

Description

Remote control device and remote control system

[0001] The present disclosure relates to a remote control device and a remote control system for remotely controlling a towing vehicle and connecting it to a towed vehicle.

[0002] BACKGROUND ART Conventionally, techniques have been disclosed for assisting a coupling operation for coupling a towing vehicle (also called a tractor) and a towed vehicle (also called a trailer).

[0003] For example, a technology has been disclosed that automates the task of coupling a tractor to a trailer by controlling the tractor based on image information from a camera attached to the rear (see, for example, Patent Document 1).Also disclosed is a technology that sets a virtual target steering angle for the trailer and controls the tractor to follow this target steering angle so that the operator can steer the trailer via an input device as if it were a standalone vehicle (see, for example, Patent Document 2).

[0004] Patent No. 6938793 International Publication No. 2022 / 270322

[0005] The technology of Patent Document 1 reduces the labor-saving effect because, when the tractor is unmanned to perform the coupling work completely, if the coupling work under automatic control fails, an assistant must be dispatched to the site. Also, the technology of Patent Document 2 is premised on operation when the tractor and trailer are coupled, and cannot be used for coupling work of coupling a tractor and a trailer.

[0006] The present disclosure has been made to solve such problems, and aims to provide a remote control device and remote control system that can perform the coupling operation between a towing vehicle and a towed vehicle without dispatching an assistant to the site, even if the coupling operation under automatic control fails.

[0007] In order to solve the above problems, the remote control device disclosed herein is a remote control device that remotely controls a towing vehicle to connect it to a towed vehicle, and includes: a steering detection unit that acquires, as a steering amount, the operator's steering of the towing vehicle that has switched to remote control control when predetermined conditions are met; an input detection unit that receives remote control information including a setting for whether or not to perform remote control; a transmitter that transmits the steering amount and remote control information; a receiver that receives rear image information behind the towing vehicle and sensor information of the towing vehicle detected by an internal sensor; a trajectory calculation unit that calculates, as trajectory information, a predicted trajectory of a predetermined portion of the towing vehicle based on the steering amount, sensor information, and a motion model of the towing vehicle; and an image generation unit that generates a presentation image to be presented to the operator by superimposing the trajectory information on the rear image information.

[0008] According to the present disclosure, even if the coupling operation under automatic control fails, it is possible to perform the coupling operation between the towing vehicle and the towed vehicle without dispatching an assistant to the site.

[0009] The objects, features, aspects, and advantages of the present disclosure will become more apparent from the following detailed description and the accompanying drawings.

[0010] FIG. 1 is a diagram showing an example of the configuration of a remote control system according to Embodiment 1. FIG. 2 is a diagram showing an example of the configuration of a remote control system according to Embodiment 1. FIG. 3 is a block diagram showing an example of the configuration of a remote control device according to Embodiment 1. FIG. 4 is a diagram for explaining a motion model of a combination vehicle according to Embodiment 1. FIG. 5 is a diagram for explaining a motion model of a towing vehicle according to Embodiment 1. FIG. 6 is a block diagram showing an example of the configuration of a towing vehicle control device according to Embodiment 1. FIG. 7 is a flowchart showing an example of the operation of a remote control device according to Embodiment 1. FIG. 8 is a diagram showing an example of a presented image displayed on an image display device according to Embodiment 1. FIG. 9 is a flowchart showing an example of the operation of a towing vehicle control device according to Embodiment 1. FIG. 10 is a block diagram showing an example of the configuration of a remote control device according to Embodiment 2. FIG. 11 is a flowchart showing an example of the operation of a remote control device according to Embodiment 2. FIG. 12 is a block diagram showing an example of the configuration of a remote control device according to Embodiment 3. FIG. 13 is a diagram for explaining a deviation according to Embodiment 3. FIG. 14 is a flowchart showing an example of the operation of a remote control device according to Embodiment 3. FIG. 15 is a diagram showing an example of the presented image displayed on an image display device according to Embodiment 3. FIG. 16 is a block diagram showing an example of the configuration of a remote control device according to Embodiment 4. FIG. 17 is a flowchart showing an example of the operation of a remote control device according to Embodiment 4. Fig. 10 is a diagram showing an example of a presentation image displayed on an image display device according to embodiment 4. Fig. 11 is a block diagram showing an example of a hardware configuration of a remote control device according to embodiments 1 to 4. Fig. 12 is a block diagram showing an example of a hardware configuration of a remote control device according to embodiments 1 to 4.

[0011] <First Embodiment> <Configuration of Remote Control System> Figure 1 is a diagram showing an example of the configuration of a remote control system according to the first embodiment. The remote control system is made up of a remote control device 1, a towing vehicle control device 7, and a towing vehicle 8. The remote control device 1 remotely controls the towing vehicle 8 and couples it to a towed vehicle 12. The remote control device 1 and the towing vehicle control device 7 are communicatively connected via a network 6. The network 6 interconnects multiple components via cables, radio waves, or the like, enabling data to be sent and received. The network 6 can take various forms, including a local area network (LAN), a wide area network (WAN), the Internet, telephone lines, and wireless communication.

[0012] For example, the towing vehicle control device 7 transmits image information (rear image) captured by a rear camera 10 installed on the towing vehicle 8 to the remote control device 1. The remote control device 1 controls the image display device 2 to display a presentation image obtained by processing the image information received from the towing vehicle control device 7. The operator 5 operates the control device 3 while viewing the presentation image displayed on the image display device 2. The remote control device 1 transmits the amount of control performed by the operator 5 on the control device 3 to the towing vehicle control device 7. The towing vehicle control device 7 controls the actuator of the towing vehicle 8 based on the control amount received from the remote control device 1. As a result, the coupler 11 of the towing vehicle 8 can be connected to the kingpin 13 of the towed vehicle 12 by remote control by the operator 5.

[0013] The remote control device 1 is communicatively connected to the image display device 2, the control device 3, and the input device 4. The remote control device 1 controls the image display device 2 to display a presentation image obtained by processing image information received from the towing vehicle control device 7. The operator 5 views the presentation image displayed on the image display device 2 and controls (remotely controls) the control device 3 or operates the input device 4. The remote control device 1 transmits to the towing vehicle control device 7 the amount of control performed by the operator 5 on the control device 3 and remote control information set by the operator 5 using the input device 4.

[0014] The control device 3 is a simulation of the control mechanisms of an actual vehicle, such as the steering, accelerator, brake, and shift, etc. The control device 3 may also be a tablet or a joystick.

[0015] The input device 4 is configured by a tablet or a switch, etc., and detects remote control information input by an operator 5. The remote control information includes shift operations including parking, forward, and reverse, and information on whether the remote control is on or off.

[0016] The towing vehicle 8 is equipped with a front camera 9 that takes pictures of the area in front of the towing vehicle 8 and a rear camera 10 that takes pictures of the area behind the towing vehicle 8. The front camera 9 and rear camera 10 may be installed on the roadside instead of on the towing vehicle 8. If the front camera 9 and rear camera 10 are installed on the roadside, a technique such as viewpoint conversion is used to convert the image into one in front of or behind the towing vehicle 8. The towing vehicle 8 is not limited to being equipped with the front camera 9 and rear camera 10, but may also be equipped with an external sensor such as LiDAR (Light Detection and Ranging) or radar that allows the operator 5 to recognize the area around the coupling.

[0017] The towing vehicle control device 7 transmits image information from the front camera 9 or rear camera 10 installed on the towing vehicle 8 to the remote control device 1. The towing vehicle control device 7 also controls the towing vehicle 8 based on the amount of steering received from the remote control device 1. Note that the towing vehicle control device 7 may also automatically control the towing vehicle 8 based on the image information from the front camera 9 or rear camera 10.

[0018] The towing vehicle 8 is equipped with a front camera 9, a rear camera 10, a steering actuator consisting of a motor, etc., and an acceleration / deceleration actuator consisting of an engine, a motor, and a brake. The steering actuator and acceleration / deceleration actuator operate based on the steering amount. The towing vehicle 8 is also equipped with a coupler 11 (coupling part) used to couple with the towed vehicle 12.

[0019] The towed vehicle 12 is provided with a kingpin 13 (coupling portion) that is used to couple the towing vehicle 8. As shown in FIG. 1, the kingpin 13 may not be visible unless viewed from underneath the towed vehicle 12.

[0020] FIG. 2 is a diagram showing an example of the configuration of the remote control system according to the first embodiment, showing a case where there are two towed vehicles.

[0021] The example in Figure 2 shows a state in which a towed vehicle 14 is coupled to the combination vehicle 8. The operator 5 remotely controls the towed vehicle 14 to couple the hitch receiver 16 to the trailer hitch 18 of the towed vehicle 17. In this case, the presented image displayed on the image display device 2 includes a rear view image captured by the rear camera 15 installed on the towed vehicle 14. Similarly, when two towed vehicles 14, 17 are coupled to the towing vehicle 8 and another towed vehicle is to be coupled to the towed vehicle 17, the operator 5 can remotely control the towed vehicle 17 while viewing the presented image including a rear view image captured by the rear camera (not shown) installed on the towed vehicle 17.

[0022] Examples of the towing vehicle 8 include an automobile, crossover, truck, van, sports utility vehicle (SUV), recreational vehicle (RV), or any vehicle configured to tow a towed vehicle.

[0023] The towed vehicles 12, 14, and 17 are typically non-powered vehicles towed by a towing vehicle, and examples of towed vehicles include utility trailers, pop-up campers, travel trailers, livestock trailers, flatbed trailers, enclosed car carriers, and boat trailers.

[0024] In Fig. 1, the towing vehicle 8 and the towed vehicle 12 form a combination of vehicles. In Fig. 2, the towing vehicle 8 and the towed vehicles 14, 17 form a combination of vehicles. The combination of vehicles includes the towing vehicle and one or more towed vehicles. The combination of vehicles may be equipped with a dolly (see Fig. 5).

[0025] The coupler 11 and kingpin 13 shown in Figure 1 are coupling devices that connect the towing vehicle 8 and the towed vehicle 12. The hitch receiver 16 and trailer hitch 18 shown in Figure 2 are coupling devices that connect the towed vehicle 14 and the towed vehicle 17. The trailer hitch 18 may be a ball and socket, a fifth wheel and gooseneck, or a trailer jack. Note that the towing vehicle 8 and the towed vehicles 12, 14, and 17 may be equipped with coupling mechanisms other than the coupling devices shown in Figures 1 and 2.

[0026] The towing vehicle 8 and the towed vehicle 12 may be electrically connected in addition to being mechanically connected (coupled). By electrically connecting the towing vehicle 8 and the towed vehicle 12, the towed vehicle 12 can receive power from the rear light circuit of the towing vehicle 8. This allows the towed vehicle 12 to install tail lights, turn signals, and brake lights that are synchronized with the lights of the towing vehicle 8.

[0027] 3 is a block diagram showing an example of the configuration of the remote control device 1. The remote control device 1 includes a control detection unit 19, an input detection unit 20, a transmission unit 21, a reception unit 22, a trajectory calculation unit 23, and an image generation unit 24. The remote control device 1 is also connected to an image display device 2, a control device 3, and an input device 4 so as to be able to communicate with each other.

[0028] The steering detection unit 19 acquires steering data related to the steering of the operator 5 from the control device 3, and detects a steering amount indicating what kind of steering the operator 5 performed and to what extent, based on the steering data. That is, the steering detection unit 19 acquires the steering of the operator 5 as a steering amount. The steering detection unit 19 outputs the steering amount to the transmission unit 21 and the trajectory calculation unit 23.

[0029] The input detection unit 20 acquires input data related to remote control input by the operator 5 from the input device 4, and detects remote control information indicating what input the operator 5 has made regarding remote control based on the input data. That is, the input detection unit 20 accepts the remote control information. The input detection unit 20 outputs the remote control information to the transmission unit 21.

[0030] The transmitter 21 transmits the steering amount and remote steering information to the towing vehicle control device 7 via the network 6 .

[0031] The receiver 22 receives image information, sensor information, and control success / failure information from the towing vehicle control device 7 via the network 6. The image information is image information from the front camera 9 or rear camera 10 installed on the towing vehicle 8. The sensor information is sensor information detected by sensors (internal sensors) installed on the towing vehicle. The control success / failure information is information indicating whether the towing vehicle control device 7 is able to automatically control the towing vehicle 8 (whether the automatic control was successful or unsuccessful). The receiver 22 outputs the sensor information to the trajectory calculation unit 23, and outputs the image information and control success / failure information to the image generation unit 24.

[0032] The trajectory calculation unit 23 calculates a predicted trajectory of the coupler 11 (a predetermined portion) of the towing vehicle 8 based on the steering amount, sensor information, and a motion model of the towing vehicle. The trajectory calculation unit 23 outputs the calculated predicted trajectory to the image generation unit 24 as trajectory information. When a towed vehicle 17 is coupled to a combination vehicle made up of the towing vehicle 8 and towed vehicle 14 as shown in FIG. 2, the trajectory calculation unit 23 calculates a predicted trajectory of a predetermined portion of the combination vehicle (for example, the hitch receiver 16 of the towed vehicle 14) using the motion model of the combination vehicle.

[0033] The image generation unit 24 generates a presentation image to be presented to the operator 5 by superimposing the trajectory information on the image information. The image generation unit 24 outputs the generated presentation image to the image display device 2. The presentation image is displayed on the image display device 2. The image generation unit 24 may also include control success / failure information in the presentation image. In this case, the operator 5 can look at the control success / failure information displayed on the image display device 2 and determine whether or not he or she needs to remotely control the towing vehicle 8.

[0034] <Motion Model> Figure 4 is a diagram illustrating the motion model of an articulated vehicle. Here, we assume a combination vehicle in which n towed vehicles, including towed vehicles 26-29, are connected to towing vehicle 25. In the following explanation, the global coordinate system refers to a reference coordinate system, and may be a geographic coordinate system, a planar rectangular coordinate system developed on a plane based on a geographic coordinate system, or a coordinate system determined by a designer. The vehicle coordinate system refers to a coordinate system fixed to each of the towing vehicle and towed vehicle. When the combination vehicle moves, the position and attitude (angle) of the vehicle coordinate system changes relative to the global coordinate system.

[0035] The axle center position of the k-th towed vehicle in the vehicle coordinate system relative to the global coordinate system (X, Y) is (x ｋ , y ｋ ), the following formula (1) is obtained from a geometric relationship. However, it is assumed that the following formula (2) holds in formula (1).

[0036] where t is time and k=0 represents the towing vehicle. That is, (x ０ , y ０ ) is the rear axle center position of the towing vehicle. ｋ is the attitude (angle) relative to the global coordinate system (not shown), and h ｋ ｆ is the distance between the axle center of the towed vehicle and the front hitch point 30 (front hitch offset), h ｋ ｒ is the distance between the axle center of the towed vehicle and the rear hitch point 30 (rear hitch offset).

[0037] Also, under the assumption that no sideslip angle occurs, the motion model expressed by the following (3) is obtained, where it is assumed that the following equation (4) holds in equation (3).

[0038] where ν ０ is the translational velocity of the towing vehicle, ζ ０ is the angular velocity of the towing vehicle.

[0039] The translational velocity of the towing vehicle, v ０ and the angular velocity of the towing vehicle ζ ０ From this, the angular velocity of the towed vehicle γ ｋ Then, the angular velocity γ ｋ By integrating γ ｋ can be obtained.

[0040] From equations (1) and (2), the rear axle center position (x ０ , y ０ ) and γ ｋ is given, the axle center position (x ｋ , y ｋ ) can be obtained.

[0041] Figure 5 shows a case where two loading platforms 34, 37 are connected to a towing vehicle 31. The loading platform 34 is made up of a dolly 32 and a towed vehicle 33. The loading platform 37 is made up of a dolly 35 and a towed vehicle 36. The dollies 32, 35 have axles that can rotate freely in a plane. In the example of Figure 5, by treating each of the dollies 32, 35 as a single towed vehicle, it is possible to construct a motion model using equations (1) to (4).

[0042] Next, the motion model of the towing vehicle will be explained.

[0043] As a motion model of the towing vehicle, for example, the translational velocity of the towing vehicle v ０ and the angular velocity of the towing vehicle ζ ０ A model expressed by the following equation (5) with the following input is considered.

[0044] ζ ０ =ν ０ / L ０ tan δ ０ Since the relationship of ０ is the steering angle of the towing vehicle, L ０ is the wheelbase of the towing vehicle.

[0045] In addition, the left and right wheels at the front and rear of the towing vehicle are often treated as one wheel each, as shown in Figure 6. This type of motion model is called a two-wheel model, because it treats a four-wheel vehicle as a two-wheel vehicle.

[0046] In the case of a two-wheel model, the steering angle δ of the towing vehicle ０ , and the acceleration of the towing vehicle α ０ is used as input, and the center of gravity of the towing vehicle (x ｇ０ , y ｇ０ ) and geometric relationships, a motion model expressed by the following equation (6) may be used.

[0047] <Configuration of Towing Vehicle Control Device> Figure 7 is a block diagram showing an example of the configuration of the towing vehicle control device 7. The towing vehicle control device 7 includes a receiving unit 38, a command value conversion unit 39, an image acquisition unit 40, a sensor acquisition unit 41, an automatic control unit 42, a transmission unit 43, and a setting unit 44. The towing vehicle control device 7 is also connected to be able to communicate with a front camera 9, a rear camera 10, a sensor 45, and an actuator 46 installed on the towing vehicle 8.

[0048] The receiving unit 38 receives the control amount and remote control information from the remote control device 1 via the network 6. The receiving unit 38 outputs the control amount to the command value conversion unit 39 and outputs the remote control information to the setting unit 44.

[0049] The command value converter 39 converts the operation amount into a first actuator command value. The first actuator command value is a command value, such as a current or a voltage, for controlling the actuator 46 by remote control by the operator 5. The command value converter 39 outputs the first actuator command value to the setting unit 44.

[0050] The image acquisition unit 40 acquires image data of the area in front of the towing vehicle 8 (forward image) from the forward camera 9, and acquires image data of the area behind the towing vehicle 8 (rear image) from the rear camera 10. The image acquisition unit 40 outputs the image data as image information to both the automatic control unit 42 and the transmission unit 43.

[0051] The sensor acquisition unit 41 acquires sensor data from the sensor 45. The sensor 45 is an internal sensor that detects the state of the towing vehicle 8, and is composed of, for example, a steering angle sensor, a vehicle speed sensor, an IMU (Inertial Measurement Unit), and the like.

[0052] The automatic control unit 42 calculates a second actuator command value for automatically controlling the towing vehicle 8 based on the image information and the sensor information. The automatic control unit 42 also determines the success or failure of the automatic control of the towing vehicle 8 based on at least one of the image information and the sensor information. For example, the automatic control unit 42 determines that the automatic control has failed if the coupling operation of automatically coupling the towed vehicle to the towing vehicle 8 fails. The automatic control unit 42 also determines that the automatic control has failed if an abnormality is detected in the sensor 45. Alternatively, the automatic control unit 42 may determine that the automatic control has failed if control is impossible due to the presence of many people around the coupled vehicles, or if an abnormality other than that of the sensor 45 is detected. In this way, the automatic control unit 42 determines whether or not a predetermined condition has been met (whether or not the automatic control has failed). The automatic control unit 42 outputs the determination result to the transmission unit 43 as control success / failure information.

[0053] If the automatic control unit 42 determines that the automatic control has failed, the automatic control unit 42 may automatically turn on the remote control. In this case, the control success / failure information includes information indicating that the remote control has been turned on.

[0054] The transmitter 43 transmits image information, sensor information, and control success / failure information to the remote control device 1 via the network 6 .

[0055] The setting unit 44 selects a first actuator command value or a second actuator command value based on the remote control information and outputs it to the actuator 46 as an actual actuator command value. Specifically, if the remote control information includes information to turn remote control on, the setting unit 44 selects the first actuator command value. In this case, the towing vehicle 8 operates in accordance with remote control by the operator 5. Furthermore, if the remote control information includes information to turn remote control off, the setting unit 44 selects the second actuator command value. In this case, the towing vehicle 8 operates in accordance with automatic control by the automatic control unit 42.

[0056] <Operation of Remote Control Device> Figure 8 is a flowchart showing an example of the operation of the remote control device 1. Note that Figure 8 describes the case where the towed vehicle 12 is coupled to the towing vehicle 8 (see Figure 1), but the same applies to the case where the towed vehicle 17 is coupled to a combination vehicle made up of the towing vehicle 8 and the towed vehicle 14 (see Figure 2).

[0057] In step S11 , the receiving unit 22 receives image information, sensor information, and control success / failure information from the towing vehicle control device 7 via the network 6 .

[0058] In step S12, the image generating unit 24 outputs the image information as a presentation image to the image display device 2. The presentation image is displayed on the image display device 2. At this time, the presentation image may include control success / failure information.

[0059] In step S13, the input detection unit 20 receives remote control information from the input device 4. For example, if the presented image includes control success / failure information, the operator 5 recognizes that automatic control has failed (if a predetermined condition is met) and inputs to the input device 4 that remote control will be turned on. Furthermore, if the automatic control unit 42 of the towing vehicle control device 7 automatically turns on remote control, the operator 5 recognizes that remote control has been turned on, and inputs to the input device 4 that remote control will be turned on.

[0060] In step S14, the operation detection unit 19 acquires the operation of the operator 5 as the operation amount.

[0061] In step S15, the trajectory calculation unit 23 calculates a predicted trajectory of the towing vehicle 8 as trajectory information based on the steering amount, sensor information, and the motion model of the towing vehicle. For example, the trajectory calculation unit 23 calculates a predicted trajectory of the coupler 11 of the towing vehicle 8 as trajectory information.

[0062] In step S16 , the image generating unit 24 generates a presentation image by superimposing the trajectory information on the image information, and outputs the generated presentation image to the image display device 2 .

[0063] Fig. 9 is a diagram showing an example of a presented image displayed on the image display device 2. Fig. 9 shows an example in which a predicted trajectory 47 of the coupler 11 calculated by the trajectory calculation unit 23 is superimposed on a rearward image captured by the rearward camera 10 of the towing vehicle 8. Furthermore, as shown in Fig. 10, a forward image 48 captured by the forward camera 9 of the towing vehicle 8 may also be included in the presented image.

[0064] In step S17 , the transmitter 21 transmits the steering amount and remote steering information to the towing vehicle control device 7 .

[0065] <Operation of Towing Vehicle Control Device> Figure 11 is a flowchart showing an example of the operation of the towing vehicle control device 7. Note that while Figure 1 describes the case where a towed vehicle 12 is coupled to a towing vehicle 8 (see Figure 1), the same applies to the case where a towed vehicle 17 is coupled to a combination vehicle made up of the towing vehicle 8 and the towed vehicle 14 (see Figure 2).

[0066] In step S21, the receiving unit 38 receives the operation amount and remote operation information from the remote control device 1 via the network 6.

[0067] In step S22, the command value converter 39 converts the operation amount into a first actuator command value.

[0068] In step S23, the image acquisition unit 40 acquires image data (rear image) as image information from the rear camera 10 of the towing vehicle 8. The image acquisition unit 40 may also acquire image data (front image) as image information from the front camera 9 of the towing vehicle 8. The sensor acquisition unit 41 also acquires sensor data from the sensor 45 of the towing vehicle 8 as sensor information.

[0069] In step S24, the automatic control unit 42 calculates a second actuator command value based on the image information and the sensor information, and determines whether the automatic control has been successful or unsuccessful.

[0070] In step S25, the setting unit 44 selects the first actuator command value or the second actuator command value based on the remote control information and outputs it to the actuator 46 of the towing vehicle 8.

[0071] In step S26, the transmitter 43 transmits the image information, the sensor information, and the control success / failure information to the remote control device 1 via the network 6.

[0072] <Effects> According to the first embodiment, if coupling work using automatic control fails, it is possible to switch to remote control by the operator. This allows the operator to perform the coupling work using remote control. Furthermore, since a predicted trajectory is displayed on the image display device during remote control, the operator can perform the coupling work smoothly. In this way, even if coupling work using automatic control fails, it is possible to perform the coupling work between the towing vehicle and the towed vehicle unmanned at the site.

[0073] 7 illustrates a configuration in which the towing vehicle control device 7 includes the automatic control unit 42, but the towing vehicle control device 7 may also be configured without the automatic control unit 42. In this case, automatic control of the towing vehicle 8 is not performed, and the setting unit 44 that selects the first actuator command value or the second actuator command value is also unnecessary. Furthermore, the input detection unit 20 of the remote control device 1 does not need to receive information from the operator 5 indicating whether remote control is on or off.

[0074] In the first embodiment, the trajectory calculation unit 23 calculates the predicted trajectory of the coupler 11 of the towing vehicle 8, but the present invention is not limited to this. The trajectory calculation unit 23 may calculate the predicted trajectory of any part of the towing vehicle 8, such as the trajectory of the wheels. In the case of an articulated vehicle such as that shown in FIG. 2 , the calculated predicted trajectory may be of any part of the towed vehicle 14, not just the hitch receiver 16 of the towed vehicle 14.

[0075] When the operator 5 remotely controls the articulated vehicle, the trajectory calculation unit 23 calculates steering that will result in stable operation of the articulated vehicle, and the image generation unit 24 may include information about this steering in the presented image. When an articulated vehicle is driven in reverse by a human, a phenomenon occurs in which the articulated vehicle bends midway and becomes immobile (a jackknife phenomenon). By calculating in advance steering that will result in stable operation of the articulated vehicle and displaying this on the image display device 2, the operator 5 can remotely control the articulated vehicle while viewing information about this steering, thereby avoiding the occurrence of this phenomenon.

[0076] The steering direction when the operator 5 uses the control device 3 to steer may be set to match the movement of the rear wheels of the towing vehicle 8 displayed on the rear screen. This allows the operator 5 to steer the control device 3 as if he were driving the towing vehicle 8 while looking back from the driver's seat of the towing vehicle 8.

[0077] <Embodiment 2> <Configuration> Figure 12 is a block diagram showing an example of the configuration of a remote control device 49 according to embodiment 2. The remote control device 49 is characterized by including a recognition unit 50. The other configuration is the same as that of the remote control device 1 according to embodiment 1 (see Figure 3), so detailed description will be omitted here. The towing vehicle control device according to embodiment 2 is also the same as the towing vehicle control device 7 according to embodiment 1 (see Figure 7).

[0078] The recognition unit 50 recognizes the position of the coupling part (e.g., kingpin 13) of the towed vehicle 12 as towed vehicle information based on the image information and outputs the towed vehicle information to the image generation unit 24. Specifically, the recognition unit 50 performs image processing or machine learning on the image information to detect the towed vehicle 12. The recognition unit 50 then obtains the position of the kingpin from the model number of the detected towed vehicle 12. Note that information indicating the correspondence between the model number of the towed vehicle and the position of the kingpin of the towed vehicle is stored in a memory unit (not shown) of the remote control device 49.

[0079] The image generating unit 24 generates a presentation image by superimposing the track information and towed vehicle information on the image information, and outputs the generated presentation image to the image display device 2 .

[0080] <Operation> Figure 13 is a flowchart showing an example of the operation of the remote control device 49. Note that steps S31 to S35 and step S38 in Figure 13 are similar to steps S11 to S15 and step S17 in Figure 8, and therefore will not be described here. Steps S36 and S37 will be described below.

[0081] In step S36, the recognition unit 50 recognizes the position of the coupling portion of the towed vehicle 12 as towed vehicle information based on the image information.

[0082] In step S37 , the image generating unit 24 generates a presentation image by superimposing the track information and towed vehicle information on the image information, and outputs the presentation image to the image display device 2 .

[0083] Fig. 14 is a diagram showing an example of a presented image displayed on the image display device 2. Fig. 14 shows an example in which the predicted trajectory 47 of the coupler 11 calculated by the trajectory calculation unit 23 and the position of the kingpin 13 of the towed vehicle 12 recognized by the recognition unit 50 are superimposed on the rear image captured by the rear camera 10 of the towing vehicle 8.

[0084] <Effects> Because the coupling section of the towed vehicle is located underneath the vehicle, it is often not possible to photograph it with the rear camera 10 of the towing vehicle 8. According to the second embodiment, the position of the coupling section of the towed vehicle 12 is displayed, making it easier for the operator 5 to recognize the position of the coupling section of the towed vehicle 12, improving the efficiency of coupling work by remote control.

[0085] <Embodiment 3> <Configuration> Figure 15 is a block diagram showing an example of the configuration of a remote control device 51 according to embodiment 3. The remote control device 51 is characterized by including a deviation calculation unit 52. The other configuration is the same as that of the remote control device 49 according to embodiment 2 (see Figure 12), so a detailed description will be omitted here. The towing vehicle control device according to embodiment 3 is also the same as the towing vehicle control device 7 according to embodiment 1 (see Figure 7).

[0086] The deviation calculation unit 52 calculates the deviation between a first coupling portion (e.g., coupler 11), which is a predetermined portion of the towing vehicle 8, and a second coupling portion (e.g., kingpin 13), which is a coupling portion of the towed vehicle 12, as deviation information based on the track information and towed vehicle information.

[0087] 16 is a diagram for explaining the deviation. The deviation calculation unit 52 calculates the lateral deviation and angular deviation between the predicted trajectory 47 of the coupler 11 of the towing vehicle 8 calculated by the trajectory calculation unit 23 and the kingpin 13 of the towed vehicle 12.

[0088] The image generating unit 24 generates a presentation image by superimposing the track information, towed vehicle information, and deviation information on the image information, and outputs the generated presentation image to the image display device 2 .

[0089] <Operation> Figure 17 is a flowchart showing an example of the operation of the remote control device 51. Note that steps S41 to S46 and step S49 in Figure 17 are similar to steps S31 to S36 and step S38 in Figure 13, and therefore will not be described here. Steps S47 and S48 will be described below.

[0090] In step S47, the deviation calculation unit 52 calculates the deviation between the coupling portion of the towing vehicle 8 and the coupling portion of the towed vehicle 12 based on the track information and the towed vehicle information.

[0091] In step S48 , the image generating unit 24 generates a presentation image by superimposing the track information, towed vehicle information, and deviation information on the image information, and outputs the generated presentation image to the image display device 2 .

[0092] Fig. 18 is a diagram showing an example of a presented image displayed on the image display device 2. Fig. 18 shows an example in which the predicted trajectory 47 of the coupler 11 calculated by the trajectory calculation unit 23, the position of the kingpin 13 of the towed vehicle 12 recognized by the recognition unit 50, and deviation information 53 calculated by the deviation calculation unit 52 are superimposed on a rear image captured by the rear camera 10 of the towing vehicle 8. In Fig. 18, if the operator 5 is performing good maneuvering (if the deviation is small), the border of the presented image may be emphasized (for example, the border may be colored), and if the deviation is large, the background of the presented screen may be colored red. Information regarding the magnitude of the deviation may also be notified by voice.

[0093] <Effects> According to the third embodiment, the operator 5 can easily recognize the deviation between the coupling portion of the towing vehicle 8 and the coupling portion of the towed vehicle 12, which improves the efficiency of coupling work by remote control.

[0094] <Fourth Embodiment> <Configuration> Figure 19 is a block diagram showing an example of the configuration of a remote control device 54 according to the fourth embodiment. The remote control device 54 is characterized by including a communication delay measurement unit 55 and a position prediction unit 56. The other configuration is the same as that of the remote control device 51 according to the third embodiment (see Figure 15), and therefore detailed description will be omitted here. The towing vehicle control device according to the fourth embodiment is also the same as the towing vehicle control device 7 according to the first embodiment (see Figure 7).

[0095] The communication delay measurement unit 55 measures the communication delay of the network 6 between the remote control device 54 and the towing vehicle control device 7. For example, the communication delay measurement unit 55 measures the communication delay of the network 6 using RTT (Round Trip Time) or the like.

[0096] The position prediction unit 56 predicts the position of the coupling part (for example, the coupler 11) of the towing vehicle 8 as a predicted position based on the trajectory information and communication delay.

[0097] The image generating unit 24 generates a presentation image by superimposing the track information, towed vehicle information, deviation information, and predicted position on the image information, and outputs the generated presentation image to the image display device 2 .

[0098] <Operation> Figure 20 is a flowchart showing an example of the operation of the remote control device 54. Note that steps S51 to S57 and step S61 in Figure 20 are similar to steps S41 to S47 and step S49 in Figure 17, and therefore will not be described here. Steps S58 to S60 will be described below.

[0099] In step S58 , the communication delay measurement unit 55 measures the communication delay of the network 6 between the remote control device 54 and the towing vehicle control device 7 .

[0100] In step S59, the position prediction unit 56 predicts the position of the coupling part of the towing vehicle 8 as a predicted position based on the trajectory information and the communication delay.

[0101] In step S60, the image generating unit 24 generates a presentation image by superimposing the track information, towed vehicle information, deviation information, and predicted position on the image information, and outputs the generated presentation image to the image display device 2.

[0102] Fig. 21 is a diagram showing an example of a presented image displayed on the image display device 2. Fig. 21 shows an example in which the predicted trajectory 47 of the coupler 11 calculated by the trajectory calculation unit 23, the position of the kingpin 13 of the towed vehicle 12 recognized by the recognition unit 50, the deviation information 53 calculated by the deviation calculation unit 52, and the predicted position 57 of the coupler 11 predicted by the position prediction unit 56 are superimposed on the rear image captured by the rear camera 10 of the towing vehicle 8.

[0103] <Effects> According to the fourth embodiment, the operator 5 can easily recognize the position of the coupler 11 of the towing vehicle 8, which improves the efficiency of the coupling work by remote control.

[0104] <Embodiment 5> In the first to fourth embodiments, the case where a rearward image is displayed on the image display device 2 has been described, but a forward image may also be displayed on the image display device 2. Specifically, in embodiment 5, when the towing vehicle 8 moves forward, a forward image is displayed on the image display device 2, and when the towing vehicle 8 moves backward, a rearward image is displayed on the image display device 2.

[0105] The remote control device according to the fifth embodiment may have the same configuration as the remote control device 1 according to the first embodiment (see FIG. 3), the remote control device 49 according to the second embodiment (see FIG. 12), the remote control device 51 according to the third embodiment (see FIG. 15), or the remote control device 54 according to the fourth embodiment (see FIG. 19). The towing vehicle control device according to the fifth embodiment is similar to the towing vehicle control device 7 according to the first embodiment (see FIG. 7).

[0106] The receiving unit 22 receives image information including a forward image taken by a forward camera 9 installed on the towing vehicle 8 and a rearward image taken by a rearward camera 10 installed on the towing vehicle 8.

[0107] The trajectory calculation unit 23 calculates the predicted trajectory of the wheels (predetermined portion) of the towing vehicle 8 as trajectory information when moving forward, and calculates the predicted trajectory of the coupling portion of the towing vehicle 8 as trajectory information when moving backward. The method of calculating the predicted trajectory is the same as in Embodiment 1. The trajectory calculation unit 23 may also obtain information related to the shift operation from the steering detection unit 19 or the input detection unit 20.

[0108] When moving forward, the image generation unit 24 generates a presentation image in which the predicted wheel trajectory of the towing vehicle 8 is superimposed on the forward image, and outputs this to the image display device 2. For example, as shown in Fig. 22, the image display device 2 displays a predicted trajectory 58 of the front wheels of the towing vehicle 8 and a predicted trajectory 59 of the rear wheels of the towed vehicle (for example, towed vehicle 14 in Fig. 2) superimposed on the forward image. Note that the example in Fig. 22 shows a presentation image that is displayed on the image display device 2 when the combination vehicle is moving forward.

[0109] <Effects> According to the fifth embodiment, the predicted trajectory of the coupling portion is displayed when moving backward, and the predicted trajectory of the wheels is displayed when moving forward, thereby improving the efficiency of coupling work by remote control.

[0110] <Hardware Configuration> The functions of the operation detection unit 19, input detection unit 20, transmitter 21, receiver 22, trajectory calculation unit 23, and image generation unit 24 in the remote control device 1 described in embodiment 1 are realized by processing circuits. That is, the remote control device 1 includes a processing circuit for acquiring operation amounts, acquiring remote control information, transmitting the operation amounts and the remote control information, receiving image information, sensor information, and control success / failure information, calculating trajectory information, and generating a presentation image including the trajectory information. The processing circuit may be dedicated hardware, or may be a processor (also referred to as a CPU, central processing unit, processing device, arithmetic unit, microprocessor, microcomputer, or DSP (Digital Signal Processor)) that executes a program stored in a memory.

[0111] When the processing circuit is dedicated hardware, the processing circuit 60 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof, as shown in Fig. 23. The functions of the maneuver detection unit 19, the input detection unit 20, the transmission unit 21, the reception unit 22, the trajectory calculation unit 23, and the image generation unit 24 may be realized individually by the processing circuit 60, or these functions may be realized together by a single processing circuit 60.

[0112] When the processing circuit 60 is the processor 70 shown in FIG. 24 , the functions of the steering detection unit 19, the input detection unit 20, the transmission unit 21, the reception unit 22, the trajectory calculation unit 23, and the image generation unit 24 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 71. The processor 70 realizes each function by reading and executing the program recorded in the memory 71. That is, the remote control device 1 includes the memory 71 for storing a program that ultimately executes the steps of acquiring a steering amount, acquiring remote control information, transmitting the steering amount and the remote control information, receiving image information, sensor information, and control success / failure information, calculating trajectory information, and generating a presentation image including the trajectory information. Furthermore, these programs can also be said to cause a computer to execute the procedures or methods of the steering detection unit 19, the input detection unit 20, the transmission unit 21, the reception unit 22, the trajectory calculation unit 23, and the image generation unit 24. Here, the memory may be, for example, a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), a magnetic disk, a flexible disk, an optical disk, a compact disk, a DVD (Digital Versatile Disc), or any storage medium that will be used in the future.

[0113] In addition, with regard to each function of the steering detection unit 19, input detection unit 20, transmission unit 21, reception unit 22, trajectory calculation unit 23, and image generation unit 24, some functions may be realized by dedicated hardware, and other functions may be realized by software or firmware.

[0114] Thus, the processing circuitry can implement each of the above-described functions through hardware, software, firmware, or a combination thereof.

[0115] The hardware configuration of the remote control device 1 (FIG. 3) according to the first embodiment has been described above, but the same applies to the hardware configurations of the remote control device 49 (see FIG. 12) according to the second embodiment, the remote control device 51 (see FIG. 15) according to the third embodiment, and the remote control device 54 (see FIG. 19) according to the fourth embodiment. The same also applies to the hardware configuration of the towing vehicle control device 7 according to the first embodiment.

[0116] Within the scope of the present disclosure, the embodiments can be freely combined, modified, or omitted as appropriate.

[0117] Although the present disclosure has been described in detail, the above description is illustrative in all respects and is not restrictive. It is understood that countless variations not illustrated can be envisioned.

[0118] 1 Remote control device, 2 Image display device, 3 Control device, 4 Input device, 5 Operator, 6 Network, 7 Towing vehicle control device, 8 Towing vehicle, 9 Front camera, 10 Rear camera, 11 Coupler, 12 Towed vehicle, 13 Kingpin, 14 Towed vehicle, 15 Rear camera, 16 Hitch receiver, 17 Towed vehicle, 18 Trailer hitch, 19 Steering detection unit, 20 Input detection unit, 21 Transmitter, 22 Receiver, 23 Trajectory calculation unit, 24 Image generation unit, 25 Towing vehicle, 26 Towed vehicle, 27 Towed vehicle, 28 Towed vehicle, 29 Towed vehicle, 30 Hitch point, 31 Towing vehicle, 32 Dolly, 33 Towed vehicle, 34 Loading platform, 35 Dolly, 36 Towed vehicle, 37 Loading platform, 38 Receiver, 39 Command value conversion unit, 40 Image acquisition unit, 41 sensor acquisition unit, 42 automatic control unit, 43 transmission unit, 44 setting unit, 45 sensor, 46 actuator, 47 predicted trajectory, 48 forward image, 49 remote control device, 50 recognition unit, 51 remote control device, 52 deviation calculation unit, 53 deviation information, 54 remote control device, 55 communication delay measurement unit, 56 position prediction unit, 57 predicted position, 58 predicted trajectory, 59 predicted trajectory, 60 processing circuit, 70 processor, 71 memory.

Claims

1. A remote control device that remotely controls a towing vehicle to couple it to a towed vehicle, comprising: a control detection unit that acquires, as a control amount, the operator's control of the towing vehicle, which has been switched to remote control control when predetermined conditions are met; an input detection unit that receives remote control information including a setting for whether or not to perform the remote control; a transmission unit that transmits the control amount and the remote control information; a reception unit that receives rear image information of the area behind the towing vehicle and sensor information of the towing vehicle detected by an internal sensor; a trajectory calculation unit that calculates, as trajectory information, a predicted trajectory of a predetermined portion of the towing vehicle based on the control amount, the sensor information, and a motion model of the towing vehicle; and an image generation unit that generates a presentation image to be presented to the operator by superimposing the trajectory information on the rear image information.

2. A remote control device that remotely controls a towing vehicle and couples it to a towed vehicle, comprising: a steering detection unit that acquires the operator's steering, including steering and acceleration / deceleration, as steering quantities; a transmitter that transmits the steering quantities; a receiver that receives rear image information of the area behind the towing vehicle and sensor information of the towing vehicle detected by an internal sensor; a trajectory calculation unit that calculates a predicted trajectory of a predetermined part of the towing vehicle as trajectory information based on the steering quantities, the sensor information, and a motion model of the towing vehicle; and an image generation unit that generates a presentation image to be presented to the operator by superimposing the trajectory information on the rear image information.

3. A remote control device as described in claim 1 or 2, further comprising a recognition unit that recognizes the position of the coupling part of the towed vehicle as towed vehicle information based on the rear image information, and the image generation unit generates the presentation image in which the towed vehicle information is further superimposed on the rear image information.

4. A remote control device as described in claim 3, further comprising a deviation calculation unit that calculates, based on the trajectory information and the towed vehicle information, the deviation between a first coupling section, which is a predetermined part of the towing vehicle, and a second coupling section, which is a coupling section of the towed vehicle, as deviation information, and wherein the image generation unit generates the presentation image by further superimposing the deviation information on the rear image information.

5. A remote control device as described in claim 4, further comprising: a communication delay measurement unit that measures the communication delay of the network used when receiving the rear image information and the sensor information; and a position prediction unit that predicts the position of the first coupling part as a predicted position based on the trajectory information and the communication delay, wherein the image generation unit generates the presentation image in which the predicted position is further superimposed on the rear image information.

6. A remote control device as described in claim 1 or 2, wherein the receiving unit further receives forward image information of the area ahead of the towing vehicle, the trajectory calculation unit calculates, as the trajectory information, a predicted trajectory of the wheels, which are predetermined parts of the towing vehicle, when moving forward, and calculates, as the trajectory information, a predicted trajectory of the coupling, which is a predetermined part of the towing vehicle, when moving backward, and the image generation unit generates the presentation image by superimposing the predicted trajectory of the wheels, which is the trajectory information, on the forward image when moving forward, and generates the presentation image by superimposing the predicted trajectory of the coupling, which is the trajectory information, on the rearward image when moving backward.

7. A remote control system for remotely controlling a towing vehicle to connect it to a towed vehicle, comprising: a steering detection unit that obtains operator steering as a steering amount; an input detection unit that receives remote control information including a setting for whether or not to perform the remote control; a command value conversion unit that converts the steering amount into a first actuator command value for controlling the towing vehicle; an automatic control unit that calculates a second actuator command value for automatically controlling the towing vehicle and determines whether a predetermined condition is met; a setting unit that selects the first actuator command value or the second actuator command value based on the remote control information and outputs it to the actuator of the towing vehicle; a trajectory calculation unit that calculates a predicted trajectory of a predetermined portion of the towing vehicle as trajectory information based on the steering amount, sensor information of the towing vehicle detected by an internal sensor, and a motion model of the towing vehicle; and an image generation unit that generates a presentation image to be presented to the operator by superimposing the trajectory information on image information behind the towing vehicle.

8. A remote control system for remotely controlling a towing vehicle to connect it to a towed vehicle, comprising: a steering detection unit that obtains the operator's steering, including steering and acceleration / deceleration, as steering quantities; a command value conversion unit that converts the steering quantities into actuator command values ​​for controlling the towing vehicle; a trajectory calculation unit that calculates a predicted trajectory of a predetermined portion of the towing vehicle as trajectory information based on the steering quantities, sensor information of the towing vehicle detected by an internal sensor, and a motion model of the towing vehicle; and an image generation unit that superimposes the trajectory information on image information behind the towing vehicle to generate a presentation image to be presented to the operator.

Images