Remote control device, remote control method, and remote control program

Abstract

To allow an appropriate remote control.SOLUTION: A remote control device includes: a receiving unit that receives a captured image captured by a camera mounted on a vehicle; a transmission unit that transmits a control signal to the vehicle by remote control of an operator; a calculation unit that calculates a predicted position of the vehicle based on a total delay time, which is the sum of a first delay time from the transmission of the control signal to the vehicle until the vehicle receives the control signal and a second delay time from the transmission of the captured image by the vehicle until the captured image is received; generation unit that generates a predicted captured image at a predicted position of the vehicle based on the captured image; and a control unit that controls the display unit to display the generated predicted captured image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a remote control device, a remote control method, and a remote control program. [Background technology]

[0002] Technologies for remotely controlling a vehicle have been proposed. For example, Patent Document 1 discloses a remote control system including a moving body equipped with an imaging unit for acquiring an image of a moving area, a display unit for displaying the image acquired by the imaging unit, and a remote control device equipped with a control unit for remotely controlling the moving body based on the image displayed on the display unit, the remote control system being characterized in that it is provided with a delay time estimation means for estimating a communication delay time between the moving body and the remote control device, a moving body position estimation means for estimating a moving body position at a time when the moving body is controlled by the remote control device based on a planned moving path, moving speed, and estimated delay time of the moving body from the time when the image of the moving area is acquired until a required time has elapsed, and a superimposition display means for superimposing a turning operation reference point corresponding to the estimated moving body position on an image displayed on the display unit.

[0003] Furthermore, Non-Patent Document 1 discloses a technology for determining the viewpoint movement destination based on the current coordinates of the actual vehicle obtained from RTK-GNSS and the coordinates of the actual vehicle before the delay time in the camera image, and generating a viewpoint movement image by perspective projection transformation, thereby generating a camera image with a viewpoint close to that of the actual vehicle. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2010-61346 A [Non-patent literature]

[0005] [Non-Patent Document 1] "2020 Mori Taikichirō Memorial Research Promotion Fund Results Report: Construction of a remote control support system using video viewpoint movement", Kodai Matsubara, Graduate School of Media and Governance, Keio University,<URL:http: / / www.kri.sfc.keio.ac.jp / report / mori / 2020 / c-22.pdf> Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Patent Document 1 has a problem in that the operability of remote control is reduced, making it difficult to perform appropriate remote control.

[0007] In addition, the technology described in Non-Patent Document 1 does not take into account the delay time it takes for a signal instructing remote control to reach the vehicle, making it difficult to perform appropriate remote control.

[0008] The present disclosure aims to provide a remote control device, a remote control method, and a remote control program that enable appropriate remote control. [Means for solving the problem]

[0009] A remote control device according to a first aspect of the present disclosure includes a receiving unit that receives an image captured by a camera mounted on a vehicle, a transmitting unit that transmits a steering signal for the vehicle by remote operation by an operator to the vehicle, a calculating unit that calculates a predicted position of the vehicle based on a total delay time obtained by adding up a first delay time from when the steering signal is transmitted to the vehicle to when the vehicle receives the steering signal and a second delay time from when the vehicle transmits the captured image to when the captured image is received, a generating unit that generates a predicted captured image at the predicted position of the vehicle based on the captured image, and a control unit that controls the generated predicted captured image to be displayed on a display unit.

[0010] A remote control method according to a second aspect of the present disclosure includes a computer executing a process including receiving an image captured by a camera mounted on a vehicle, transmitting a control signal for the vehicle by remote control by an operator to the vehicle, calculating a predicted position of the vehicle based on a total delay time obtained by adding up a first delay time from when the control signal is transmitted to the vehicle to when the vehicle receives the control signal and a second delay time from when the vehicle transmits the captured image to when the vehicle receives the captured image, generating a predicted captured image at the predicted position of the vehicle based on the captured image, and controlling so that the generated predicted captured image is displayed on a display unit.

[0011] A remote control program according to a third aspect of the present disclosure causes a computer to execute processing including receiving an image captured by a camera mounted on a vehicle, transmitting a steering signal for the vehicle by remote control by an operator to the vehicle, calculating a predicted position of the vehicle based on a total delay time obtained by adding up a first delay time from when the steering signal is transmitted to the vehicle to when the vehicle receives the steering signal and a second delay time from when the vehicle transmits the captured image to when the captured image is received, generating a predicted captured image at the predicted position of the vehicle based on the captured image, and controlling so that the generated predicted captured image is displayed on a display unit. Effect of the Invention

[0012] The present disclosure has the effect of enabling appropriate remote control. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a configuration diagram of a remote support system. [Diagram 2] FIG. 2 is a diagram illustrating a hardware configuration of a remote control device. [Diagram 3] FIG. 2 is a functional block diagram of the remote control device. [Figure 4] FIG. 2 is a diagram showing an example of a captured image. [Diagram 5] FIG. 2 is a diagram showing an example of a captured image. [Figure 6] FIG. 1 is a configuration diagram of a feedback control system using a Smith predictor. [Figure 7] 13 is a flowchart of a remote control process. [Figure 8] 1 is an example of a captured image in which a preceding vehicle is present. [Figure 9] 1 is an example of a captured image in which a preceding vehicle is present. [Figure 10] 1 is an example of a captured image in which a preceding vehicle is present. [Figure 11] 1 is an example of a captured image in which a preceding vehicle is present. [Figure 12] FIG. 13 is a diagram illustrating adjustment of the size of a bounding box. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0015] As shown in FIG. 1, a remote control system 10 of this embodiment has a configuration in which a remote control device 20 and a vehicle 30 are connected via a wireless network 40.

[0016] The vehicle 30 is capable of traveling in response to instructions from the remote control device 20.

[0017] The remote control device 20 is connected to a vehicle operation device 50. The vehicle operation device 50 includes a steering wheel 51 for instructing the steering angle of the vehicle 30, an accelerator 52 for instructing the acceleration of the vehicle 30, and a brake 53 for instructing the deceleration of the vehicle 30, i.e., negative acceleration.

[0018] When an operator operates at least one of the steering wheel 51, the accelerator 52, and the brake 53, a control signal including at least one of a steering angle signal, an acceleration signal, and a deceleration signal is output to the remote control device 20. The remote control device 20 transmits the control signal output from the vehicle operation device 50 by the operator's operation to the vehicle 30 via the network 40.

[0019] The vehicle 30 includes a steering control unit 31, a camera 32, and a current position acquisition unit 33. When the steering control unit 31 receives a steering signal transmitted from the remote control device 20, it controls the steering of the vehicle 30 in accordance with the received steering signal. That is, when the steering signal includes a steering angle signal, it controls the vehicle 30 so that the vehicle 30 is steered at the steering angle indicated by the steering angle signal. When the steering signal includes an acceleration signal, it controls the vehicle 30 so that the vehicle 30 is accelerated at the acceleration indicated by the acceleration signal. When the steering signal includes a deceleration signal, it controls the vehicle 30 so that the vehicle 30 is decelerated at the deceleration indicated by the deceleration signal.

[0020] The camera 32 is attached in a position where it can capture an image of the area in front of the vehicle 30. Images captured by the camera 32 are sequentially transmitted to the remote control device 20 via the network 40.

[0021] The current position acquisition unit 33 acquires the current position of the vehicle 30, for example, current position information represented by latitude and longitude, and sequentially transmits the information to the remote control device 20 via the network 40. The current position acquisition unit 33 includes, for example, a GPS (Global Positioning System) receiver or a GNSS (Global Navigation Satellite System) receiver.

[0022] Fig. 2 is a diagram showing the hardware configuration of the remote control device 20. As shown in Fig. 2, the remote control device 20 includes a controller 21. The controller 21 is configured as a device including a general computer.

[0023] 2, the controller 21 includes a central processing unit (CPU) 21A, a read only memory (ROM) 21B, a random access memory (RAM) 21C, and an input / output interface (I / O) 21D. The CPU 21A, the ROM 21B, the RAM 21C, and the I / O 21D are connected to each other via a bus 21E. The bus 21E includes a control bus, an address bus, and a data bus.

[0024] In addition, an operation unit 22, a display unit 23, a communication unit 24, and a storage unit 25 are connected to the I / O 21D.

[0025] The operation unit 22 includes, for example, a mouse and a keyboard.

[0026] The display unit 23 is configured with, for example, a liquid crystal display. An image captured by the camera 32 of the vehicle 30 is displayed on the display unit 23. The operator remotely controls the vehicle 30 by operating the vehicle operation device 50 based on the captured image displayed on the display unit 23. Note that, in this embodiment, the vehicle operation device 50 for operating the vehicle 30 is separately provided, but the remote control device 20 may be configured so that an operation signal is transmitted to the vehicle 30 by the operator operating the operation unit 22 of the remote control device 20.

[0027] The communication unit 24 is an interface for performing data communication with an external device such as a vehicle 30 .

[0028] The storage unit 25 is configured with a non-volatile external storage device such as a hard disk, etc. As shown in Fig. 2, the storage unit 25 stores a remote control program 25A and the like.

[0029] The CPU 21A is an example of a computer. The term "computer" used here refers to a processor in a broad sense, and includes a general-purpose processor (e.g., a CPU) or a dedicated processor (e.g., a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, etc.).

[0030] The remote control program 25A may be realized by storing it in a non-volatile, non-transitory recording medium, or by distributing it via a network and installing it in the remote control device 20 as appropriate.

[0031] Examples of non-volatile non-transient recording media include CD-ROMs (Compact Disc Read Only Memory), magneto-optical disks, HDDs (Hard Disk Drives), DVD-ROMs (Digital Versatile Disc Read Only Memory), flash memories, memory cards, etc.

[0032] Fig. 3 is a block diagram showing the functional configuration of the CPU 21A of the remote control device 20. As shown in Fig. 3, the CPU 21A functionally includes a receiving unit 60, a transmitting unit 61, a calculating unit 62, a generating unit 63, and a control unit 64. The CPU 21A functions as each of the functional units by reading and executing a remote control program 25A stored in the storage unit 25.

[0033] The receiving unit 60 receives images captured by the camera 32 mounted on the vehicle 30 .

[0034] The transmitter 61 transmits to the vehicle 30 a control signal for the vehicle 30 remotely controlled by an operator.

[0035] The calculation unit 62 calculates the predicted position of the vehicle 30 based on a total delay time t3 obtained by adding up a first delay time t1 from when a steering signal is transmitted to the vehicle 30 until the vehicle 30 receives the steering signal, and a second delay time t2 from when the vehicle 30 transmits a captured image until the vehicle 30 receives the captured image.

[0036] The first delay time t1 is the time from when the remote control device 20 transmits a steering signal to when the vehicle 30 receives the steering signal. The second delay time t2 is the time from when the vehicle 30 transmits a captured image to when the remote control device 20 receives the captured image. In other words, the vehicle 30 operates based on the steering signal after the first delay time t1 has elapsed since the remote control device 20 transmitted the steering signal, and the captured image at that time is received by the remote control device 20 after the second delay time t2 has elapsed since the image was transmitted from the vehicle 30.

[0037] Therefore, the time from when the remote control device 20 transmits a control signal until the captured image captured by the vehicle 30 operated by the control signal is received by the remote control device 20 is the total delay time t3. For example, as shown in Fig. 4, when a captured image G1 is displayed on the display unit 23 of the remote control device 20, the captured image G1 is an image delayed by the total delay time t3. For this reason, if an operator tries to remotely control the vehicle 30 based on the captured image G1, it becomes difficult to perform appropriate remote control.

[0038] Therefore, the generating unit 63 generates a predicted captured image at the predicted position of the vehicle 30 based on the captured image G1. For example, as shown in FIG. 4, it is assumed that the predicted position of the vehicle 30 after the total delay time t3 has elapsed is the predicted position P1. Also, if the direction indicated by the predicted angle (rotation angle) of the vehicle 30 after the total delay time t3 has elapsed is the direction of the arrow D1, the visual field range visible from the vehicle 30 after the total delay time t3 has elapsed is the visual field range S1 indicated by the dashed frame in FIG. 4. In this case, as shown in FIG. 5, an image G2 of the visual field range S1 is cut out from the captured image G1, and a perspective projection conversion process is performed on the cut-out image G2 so that it becomes an image that is visually recognized when facing the direction of the arrow D1 from the predicted position P1, and the converted image is set as the predicted captured image.

[0039] The control unit 64 controls the display unit 23 to display the predicted captured image generated by the generation unit 63 .

[0040] The calculation unit 62 calculates predicted states such as the predicted position and the predicted angle using a vehicle motion model that receives, for example, a steering signal as an input and outputs predicted states of the vehicle 30 such as the predicted position and the predicted angle of the vehicle 30. Specifically, for example, the vehicle motion model may be a model using a Smith predictor based on the Smith method.

[0041] The basic configuration of a feedback control system of the Smith method is shown in Figure 6. In the basic configuration shown in Figure 6, a controller C(s) controls a control target G(s) based on an input signal R(s). An output signal Y(s) output from the control target G(s) is fed back to the input side of the controller C(s). In addition, the control signal output from the controller C(s) is fed back to the input side of the controller C(s) via a Smith predictor P(s).

[0042] The linear state equation of the vehicle motion model using the Smith predictor P(s) is expressed by the following equation: Note that the linear state equation expressed by the following equations (1) and (2) is referred to as a first linear state equation.

[0043] X[t]=AX[t-1]+b(u[t-1]-u[(t-1)-(t1+t2)]) (1) Y[t] = cX[t] (2)

[0044] Here, X[t] in the above formula (1) indicates the state of the vehicle 30 at time t. The state of the vehicle 30 is the speed, angular velocity (yaw rate), position, angle, acceleration, and angular acceleration of the vehicle 30. Furthermore, X[t-1] indicates the state of the vehicle 30 at time t-1 immediately before time t. Furthermore, u[t-1] indicates the operation signal at time t-1, that is, the steering angle, acceleration, and deceleration. Furthermore, Y[t] in the above formula (2) indicates the predicted state of the vehicle at time t predicted at time t-1. The predicted state is the predicted speed, predicted angular velocity (yaw rate), predicted position, and predicted angle of the vehicle 30. Furthermore, A, b, and c are matrices or vectors including parameters, and are determined in advance. Furthermore, the first delay time t1 and the second delay time t2 are also determined in advance, but may be variable in consideration of fluctuations in the delay time.

[0045] The b(u[t-1]-u[(t-1)-(t1+t2)]) in the above equation (1) corresponds to the Smith predictor P(s).

[0046] u[t-1] corresponds to the movement of the vehicle 30 in response to the steering signal at time t-1. Also, u[(t-1)-(t1+t2)] corresponds to the captured image captured at the time when the vehicle 30 moves due to the steering signal at time t-1, that is, the captured image from which the predicted captured image is generated. Therefore, if the total delay time t3 (=t1+t2) is 0, that is, if there is no delay at all, b(u[t-1]-u[(t-1)-(t1+t2)]), which corresponds to the Smith predictor, is 0, and the captured image and the predicted captured image are the same.

[0047] By inputting u[t-1], which is the steering signal of the vehicle 30 at time t-1, into the above equation (1), it is possible to calculate Y[t], which is the predicted state of the vehicle 30 at time t.

[0048] Furthermore, when a captured image cannot be acquired for a predetermined period of time or longer, the generating unit 63 generates a predicted captured image based on the latest captured image.

[0049] The receiving unit 60 may also receive current position information indicating the current position of the vehicle together with the captured image, and the calculation unit 62 may correct the predicted position based on the current position information. In this case, the calculation unit 62 may correct the predicted position using a Kalman filter.

[0050] Furthermore, when a captured image includes a preceding vehicle, the generating unit 63 may generate a predicted captured image in which the preceding vehicle has been moved.

[0051] Next, the remote control processing executed by the CPU 21A of the remote control device 20 will be described with reference to Fig. 7. The remote control processing shown in Fig. 7 is processing that is repeatedly executed at a predetermined cycle by the remote control device 20 while the remote control device 20 is operating. It is assumed that the current position information acquired by the current position acquisition unit 33 of the vehicle 30 is successively transmitted to the remote control device 20 and successively received by the remote control device 20.

[0052] In step S100, the CPU 21A determines whether or not a captured image has been received from the vehicle 30. If the determination in step S100 is affirmative, the process proceeds to step S102, and if not, the process proceeds to step S112.

[0053] In step S102, the CPU 21A calculates a predicted position and a predicted angle of the current vehicle 30 using the first linear state equation. Specifically, the CPU 21A substitutes u[t-1], which is the steering signal at the immediately preceding time t-1, into the above equation (1), and calculates X[t-1], which represents the state of the vehicle 30 at time t-1, based on the current position of the vehicle 30 transmitted from the vehicle 30. Note that, of X[t-1], the speed, angular velocity, angle, acceleration, and angular acceleration of the vehicle 30 can be calculated by using the past positions of the vehicle 30.

[0054] Then, X[t-1] is substituted into the above formula (1) to calculate the state X[t] of the vehicle 30 at time t, and X[t] is substituted into the above formula (2) to calculate the predicted state Y[t] of the current vehicle 30. This provides the predicted position and predicted angle of the vehicle 30 at time t, i.e., the current time.

[0055] In step S104, the CPU 21A generates a predicted captured image at a predicted position obtained by converting the predicted position of the vehicle 30 calculated in step S102 into coordinates in the camera coordinate system of the camera 32 based on the captured image received in step S100. As described above, when the captured image acquired in step S100 is the captured image G1 as shown in FIG. 4, the predicted position obtained by converting the predicted position of the vehicle 30 calculated in step S102 into coordinates in the camera coordinate system is the predicted position P1 as shown in FIG. 5, and the direction indicated by the predicted angle of the vehicle 30 calculated in step S102 is the direction of the arrow D1, an image G2 of the visual field range S1 is cut out from the captured image G1 as shown in FIG. 5. Then, an image that has been subjected to perspective projection conversion processing on the cut-out image G2 so as to become an image that is visually recognized when facing the direction of the arrow D1 from the predicted position P1 is generated as a predicted captured image.

[0056] In step S106, the CPU 21A causes the display unit 23 to display the predicted captured image of the current vehicle 30 generated in step S104. The operator checks the predicted captured image displayed on the display unit 23 and operates the vehicle operation device 50 to remotely operate the vehicle 30.

[0057] In step S108, the CPU 21A determines whether or not a steering signal output from the vehicle operation device 50 as a result of the operator operating the vehicle operation device 50 has been input. If the determination in step S108 is affirmative, the process proceeds to step S110, and if the determination is negative, the present routine is terminated.

[0058] In step S110, the CPU 21A transmits the operation signal input in step S108 to the vehicle 30. As a result, the vehicle 30 is operated based on the operation signal transmitted from the remote control device 20.

[0059] On the other hand, in step S112, the CPU 21A determines whether or not a certain time has elapsed since the captured image was previously received in step S100. If the determination in step S112 is positive, that is, if the captured image has not been updated for a certain time or more, the process proceeds to step S114, and if the determination in step S112 is negative, the process proceeds to step S100.

[0060] In step S114, the CPU 21A calculates the predicted position and predicted angle of the vehicle 30 using the first linear state equation, that is, the above equations (1) and (2), in the same manner as in step S102.

[0061] In step S116, the CPU 21A calculates a predicted position and a predicted angle of the vehicle 30 according to the second linear state equation expressed by the following equation (3) and the above equation (2).

[0062] X[t]=AX[t-1]+bu[t-1] ···(3)

[0063] The above formula (3) is a formula obtained by removing u[(t-1)-(t1+t2)] from the above formula (1). That is, since the captured image cannot be received, the predicted position and predicted angle of the vehicle 30 are calculated using the above formula (3) by removing u[(t-1)-(t1+t2)], which corresponds to the captured image from which the predicted captured image is generated. As a result, even if the captured image cannot be received for a certain period of time or more and the captured image has not been updated, the calculation of the predicted position and predicted angle of the vehicle 30 using the first linear state equation and the second linear state equation based on the latest captured image is continued.

[0064] In step S118, the CPU 21A generates a predicted captured image similarly to step S104, but unlike step S104, the CPU 21A generates the predicted captured image based on a predicted position and a predicted angle calculated using the second linear state equation.

[0065] In this way, even if reception of captured images is interrupted, the predicted position and predicted angle of the vehicle 30 are calculated by the second linear state equation, and a predicted captured image is generated. Furthermore, even if reception of captured images is interrupted, calculation of the predicted position and predicted angle by the first linear state equation continues, so that when reception of captured images resumes and the predicted position and predicted angle used to generate the predicted captured image are switched to the predicted position and predicted angle calculated by the first linear state equation, it is possible to suppress the occurrence of an unnatural feeling in the predicted captured image.

[0066] However, for example, as shown in Fig. 8, in the captured image G3, when a preceding vehicle 70 traveling in front of the vehicle 30 is present, if an image of the visual field range S2 is cut out based on the predicted position and predicted angle of the vehicle 30 and generated as a predicted captured image G4 as shown in Fig. 9, the preceding vehicle 70 will appear to be stopped in the image even though it is actually moving forward, and will appear to be on the verge of colliding with the vehicle 30. This will cause problems with the operator's remote control.

[0067] Therefore, when the captured image includes a preceding vehicle, it is preferable to generate a predicted captured image in which the preceding vehicle has also been moved. Therefore, in generating the predicted captured image in steps S104 and S118, a predicted captured image in which the preceding vehicle has been moved may be generated and displayed. In this case, the vehicle 30 needs to be equipped with a ranging device, such as a Lidar (Light detection and ranging) device, capable of measuring the distance to the preceding vehicle.

[0068] For example, a case will be described in which the captured image received in step S100 is a captured image G3 including a preceding vehicle 70 as shown in Fig. 10. The vehicle 30 measures the distance to the preceding vehicle 70 using a Lidar device, and sequentially transmits the measured distance data to the remote control device 20.

[0069] First, the left and right ends of the preceding vehicle 70 relative to the vehicle 30 are detected based on the distance measurement data transmitted from the vehicle 30. For example, as shown in Fig. 10, a bounding box B1 of the preceding vehicle 70 is calculated based on the distance measurement data, and the lower left corner of the calculated bounding box B1 is detected as the left end L1, and the lower right corner of the bounding box B1 is detected as the right end R1. The bounding box B1 can be calculated using, for example, a known deep neural network (DNN) or the like.

[0070] In addition, the positions of the left and right rear wheels of the preceding vehicle 70 may be detected and used as the left end L1 and right end R1 of the preceding vehicle 70, respectively, or the left and right backlights of the preceding vehicle 70 may be detected and used as the left end L1 and right end R1 of the preceding vehicle 70, respectively.

[0071] Then, based on the time-series distance measurement data received from the vehicle 30, the speed of the preceding vehicle 70 is calculated.

[0072] Next, among the distance measurement data transmitted from the vehicle 30, the coordinates of the left end L1 and right end R1 of the preceding vehicle 70 in the image obtained by perspective projection transformation of the captured image G3 are stored in association with the distance measurement data of the closest coordinates. Note that it is assumed that the left end L1 and right end R1 of the preceding vehicle 70 are in contact with the road surface, but if the left and right backlights of the preceding vehicle 70 are set as the left end L1 and right end R1, the coordinates of the position of the backlights lowered vertically to the road surface are set as the left end L1 and right end R1 of the preceding vehicle 70.

[0073] Then, based on the speed of the preceding vehicle 70 calculated on the basis of the distance measurement data, the predicted positions of the left and right ends of the preceding vehicle 70 after a total delay time t3 has elapsed from the time the control signal is transmitted from the remote control device 20 are calculated on the basis of a constant velocity rectilinear motion model.

[0074] Next, the predicted positions of the left end L1 and right end R1 of the preceding vehicle 70 are converted into coordinates in the camera coordinate system by inverse perspective projection transformation, and the left end L2 and right end R2, which are the predicted positions of the left and right ends of the preceding vehicle 70 in the captured image G3, are calculated, for example, as shown in Fig. 11. Then, the size of the bounding box B1 is enlarged or reduced based on the calculated distance between the left end L2 and right end R2, i.e., the lateral length, to generate a bounding box B2 as shown in Fig. 11.

[0075] For example, as shown in FIG. 12, if the horizontal length of bounding box B1 is x1 and the vertical length is y1, and the horizontal length of bounding box B2 is x2 and the vertical length is y2, then the vertical length y2 of bounding box B2 can be calculated using the following formula based on the calculated horizontal length x2 of bounding box B2 and the horizontal length x1 and vertical length y1 of bounding box B1.

[0076] y2=(x2 / x1)×y1

[0077] Then, as shown in FIG. 11, an image of the field of view range S2 is cut out from an image obtained by synthesizing the image of the resized bounding box B2 at the predicted position of the captured image G3, and the cut-out image is subjected to perspective projection transformation processing to generate a predicted captured image.

[0078] This allows the display of a predicted captured image of the preceding vehicle 70 moving in accordance with its speed, enabling the operator to appropriately perform remote control.

[0079] In the example of FIG. 10, there is one preceding vehicle, but if multiple preceding vehicles are detected, the above processing is carried out for each of the detected multiple preceding vehicles.

[0080] Furthermore, an error occurs in the predicted position of the vehicle 30 calculated using the first linear state equation compared to the actual position of the vehicle 30. Therefore, the predicted position may be corrected using a so-called Kalman filter (see the references below).

[0081] (References) "Basics of Kalman Filters", Shuichi Adachi, Measurement and Control, Vol. 56, No. 9, pp. 632-637, September 2017

[0082] The Kalman filter is a filter for obtaining a current state estimate from a past state estimate and an observation value. Therefore, the predicted position of the vehicle 30 calculated using the first linear state equation is the past state estimate, and the actual position of the vehicle 30 in the current position information received from the vehicle 30 is the observation value. The difference between them is a prediction error, and the prediction error is multiplied by the Kalman gain and fed back to the Kalman filter. As shown in FIG. 1, the current position information transmitted from the vehicle 30 has a fourth delay time t4, so the Kalman filter is configured taking this into consideration. As a result, the predicted position of the vehicle 30 is corrected, and the predicted position can be calculated with higher accuracy than when the Kalman filter is not used.

[0083] According to the remote control device 20 of the present embodiment described above, a predicted position of the vehicle 30 is calculated based on a total delay time t3 obtained by adding up a first delay time t1 from when a control signal is transmitted to the vehicle 30 until the vehicle 30 receives the control signal, and a second delay time t2 from when the vehicle 30 transmits a captured image until the vehicle 30 receives the captured image, and a predicted captured image of the calculated predicted position is generated and displayed on the display unit 23. This enables appropriate remote control.

[0084] The present disclosure is not limited to the above-described embodiments, and various modifications and applications are possible without departing from the spirit and scope of the present invention.

[0085] In addition, the configuration of the remote control device 20 described in the above embodiment (see FIG. 2) is one example, and it goes without saying that unnecessary parts may be deleted and new parts may be added without departing from the spirit of the present invention.

[0086] Furthermore, the processing flow of the remote control program 25A described in the above embodiment (see FIG. 7) is also one example, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged within the scope of the present invention. [Explanation of symbols]

[0087] 10 Remote control system, 20 Remote control device, 25A Remote control program, 30 Vehicle, 31 Control unit, 32 Camera, 33 Current position acquisition unit, 50 Vehicle operation device, 51 Steering, 52 Accelerator, 53 Brake, 60 Receiving unit, 61 Transmitting unit, 62 Calculating unit, 63 Generating unit, 64 Control unit, 70 Leading vehicle

Claims

1. a receiving unit (60) for receiving an image captured by a camera (32) mounted on a vehicle (30); a transmitter (61) for transmitting a vehicle operation signal to the vehicle by remote control by an operator; a calculation unit (62) that calculates a predicted position of the vehicle based on a total delay time obtained by summing a first delay time from when the steering signal is transmitted to the vehicle until when the vehicle receives the steering signal, and a second delay time from when the vehicle transmits the captured image until when the captured image is received; A generation unit (63) that generates a predicted captured image by cutting out an image of a visual field range at a predicted position of the vehicle from the captured image; a control unit (64) that controls the generated predicted captured image to be displayed on a display unit (23); Equipped with The generating unit generates the predicted captured image based on the latest captured image when the captured image cannot be acquired for a predetermined period of time or longer. A remote control device (20).

2. The calculation unit calculates the predicted position using a vehicle motion model that receives the steering signal as an input and outputs the predicted position of the vehicle.

2. The remote control device according to claim 1.

3. The vehicle motion model is a model using a Smith predictor based on the Smith method.

3. The remote control device according to claim 2.

4. A receiving unit (60) that receives an image captured by a camera (32) mounted on a vehicle (30); a transmitter (61) for transmitting a vehicle operation signal to the vehicle by remote control by an operator; a calculation unit (62) that calculates a predicted position of the vehicle based on a total delay time obtained by summing a first delay time from when the steering signal is transmitted to the vehicle until when the vehicle receives the steering signal, and a second delay time from when the vehicle transmits the captured image until when the captured image is received; A generation unit (63) that generates a predicted captured image by cutting out an image of a visual field range at a predicted position of the vehicle from the captured image; a control unit (64) that controls the generated predicted captured image to be displayed on a display unit (23); Equipped with The receiving unit receives current position information indicating a current position of the vehicle together with the captured image, The calculation unit corrects the predicted position based on the current position information. Remote control device.

5. The calculation unit corrects the predicted position using a Kalman filter.

5. The remote control device according to claim 4.

6. A receiving unit (60) that receives an image captured by a camera (32) mounted on a vehicle (30); a transmitter (61) for transmitting a vehicle operation signal to the vehicle by remote control by an operator; a calculation unit (62) that calculates a predicted position of the vehicle based on a total delay time obtained by summing a first delay time from when the steering signal is transmitted to the vehicle until when the vehicle receives the steering signal, and a second delay time from when the vehicle transmits the captured image until when the captured image is received; A generation unit (63) that generates a predicted captured image by cutting out an image of a visual field range at a predicted position of the vehicle from the captured image; a control unit (64) that controls the generated predicted captured image to be displayed on a display unit (23); Equipped with When the captured image includes a preceding vehicle, the generating unit generates a predicted captured image in which the preceding vehicle has been moved. Remote control device.

7. A computer (21A) Receive images captured by a camera mounted on the vehicle, Transmitting a control signal for the vehicle by remote control of an operator to the vehicle; calculating a predicted position of the vehicle based on a total delay time obtained by adding up a first delay time from when the steering signal is transmitted to the vehicle until when the vehicle receives the steering signal and a second delay time from when the vehicle transmits the captured image until when the captured image is received; generating a predicted captured image by extracting an image of a visual field range at a predicted position of the vehicle from the captured image; Controlling the generated predicted captured image to be displayed on a display unit; When the captured image cannot be acquired for a predetermined period of time or longer, the predicted captured image is generated based on the latest captured image. A remote control method for performing a process including the steps of:

8. A computer (21A), Receive images captured by a camera mounted on the vehicle, Transmitting a control signal for the vehicle by remote control of an operator to the vehicle; calculating a predicted position of the vehicle based on a total delay time obtained by adding up a first delay time from when the steering signal is transmitted to the vehicle until when the vehicle receives the steering signal and a second delay time from when the vehicle transmits the captured image until when the captured image is received; generating a predicted captured image by extracting an image of a visual field range at a predicted position of the vehicle from the captured image; Controlling the generated predicted captured image to be displayed on a display unit; The receiving step receives current position information indicating a current position of the vehicle together with the captured image; The calculation corrects the predicted position based on the current position information. A remote control method for performing a process including the steps of:

9. A computer (21A), Receive images captured by a camera mounted on the vehicle, Transmitting a control signal for the vehicle by remote control of an operator to the vehicle; calculating a predicted position of the vehicle based on a total delay time obtained by adding up a first delay time from when the steering signal is transmitted to the vehicle until when the vehicle receives the steering signal and a second delay time from when the vehicle transmits the captured image until when the captured image is received; generating a predicted captured image by extracting an image of a visual field range at a predicted position of the vehicle from the captured image; Controlling the generated predicted captured image to be displayed on a display unit; When the captured image includes a preceding vehicle, a predicted captured image is generated by moving the preceding vehicle. A remote control method for performing a process including the steps of:

10. On the computer, Receive images captured by a camera mounted on the vehicle, Transmitting a control signal for the vehicle by remote control of an operator to the vehicle; calculating a predicted position of the vehicle based on a total delay time obtained by adding up a first delay time from when the steering signal is transmitted to the vehicle until when the vehicle receives the steering signal and a second delay time from when the vehicle transmits the captured image until when the captured image is received; generating a predicted captured image by extracting an image of a visual field range at a predicted position of the vehicle from the captured image; Controlling the generated predicted captured image to be displayed on a display unit; When the captured image cannot be acquired for a predetermined period of time or longer, the predicted captured image is generated based on the latest captured image. A remote control program (25A) for executing a process including the steps of:

11. A computer comprising: Receive images captured by a camera mounted on the vehicle, Transmitting a control signal for the vehicle by remote control of an operator to the vehicle; calculating a predicted position of the vehicle based on a total delay time obtained by adding up a first delay time from when the steering signal is transmitted to the vehicle until when the vehicle receives the steering signal and a second delay time from when the vehicle transmits the captured image until when the captured image is received; generating a predicted captured image by extracting an image of a visual field range at a predicted position of the vehicle from the captured image; Controlling the generated predicted captured image to be displayed on a display unit; The receiving step receives current position information indicating a current position of the vehicle together with the captured image; The calculation corrects the predicted position based on the current position information. A remote control program (25A) for executing a process including the steps of:

12. A computer comprising: Receive images captured by a camera mounted on the vehicle, Transmitting a control signal for the vehicle by remote control of an operator to the vehicle; calculating a predicted position of the vehicle based on a total delay time obtained by adding up a first delay time from when the steering signal is transmitted to the vehicle until when the vehicle receives the steering signal and a second delay time from when the vehicle transmits the captured image until when the captured image is received; generating a predicted captured image by extracting an image of a visual field range at a predicted position of the vehicle from the captured image; Controlling the generated predicted captured image to be displayed on a display unit; When the captured image includes a preceding vehicle, a predicted captured image is generated by moving the preceding vehicle. A remote control program (25A) for executing a process including the steps of:

Images