Remote control system and remote control device

Abstract

The present disclosure relates to a remote control system. The remote control system according to the present disclosure is a remote control system for remotely controlling a moving body from a remote control device, the moving body acquiring and outputting a first surrounding image of the moving body, the remote control device including a control device for controlling the moving body and a remote computer calculating a predicted image after a predetermined time based on the first surrounding image output by the moving body, the remote computer including a three-dimensional scene construction unit for constructing a three-dimensional scene based on the first surrounding image and a first depth image generated based on the first surrounding image, a moving body movement prediction unit for predicting the position and orientation of the moving body after the predetermined time based on operation information to the control device and a motion model of the moving body, and a predicted image generation unit for generating a predicted image based on the three-dimensional scene and the position and orientation after the predetermined time.

Description

[Technical field]

[0001] The present disclosure relates to a remote control system for a moving object, and more particularly to a remote control system that suppresses degradation of operability due to communication delays. [Background technology]

[0002] In remote control systems in which an operator operates a mobile object from a remote location based on images captured by a camera mounted on the mobile object, communication delays can cause a problem of reduced operability, but technology has been developed to improve operability by predicting images.

[0003] For example, Patent Document 1 discloses a technology that uses an environmental information sensor such as a laser range finder to recognize a moving obstacle from a camera image of a moving object, separates the moving obstacle image from a background image, predicts a future image of the moving obstacle based on the speed information of the moving obstacle, and presents an image in which the prediction result is combined with the camera image, thereby suppressing deterioration of operability due to communication delays. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6054513 Summary of the Invention [Problem to be solved by the invention]

[0005] In the technology disclosed in Patent Document 1, a moving body predicts future images of a moving obstacle based on the speed information of the moving obstacle acquired from a camera image, so there is a delay time until the speed information is obtained, which reduces the accuracy of the movement prediction, and therefore reduces the accuracy of the image prediction, resulting in reduced operability.

[0006] The present disclosure has been made to solve the above problems, and aims to provide a remote control system that suppresses deterioration of operability in a remote control device. [Means for solving the problem]

[0007] A remote control system according to the present disclosure is a remote control system for remotely controlling a moving body from a remote control device, the moving body acquiring and outputting a first surrounding image around the moving body, the remote control device including a control device for controlling the moving body and a remote computer calculating a predicted image after a predetermined time based on the first surrounding image output by the moving body, the remote computer including a three-dimensional scene construction unit for constructing a three-dimensional scene based on the first surrounding image and a first depth image generated based on the first surrounding image, a moving body movement prediction unit for predicting a position and orientation of the moving body after the predetermined time based on operation information to the control device and a motion model of the moving body, and a predicted image generation unit for generating the predicted image based on the three-dimensional scene and the position and orientation after the predetermined time. a dynamic target movement prediction unit that recognizes a dynamic target from the three-dimensional scene, estimates a speed of the dynamic target, and predicts a dynamic target movement prediction result including a position and an attitude of the dynamic target after the predetermined time; Have The predicted image generating unit generates the predicted image based on the dynamic target movement prediction result. do. Effect of the Invention

[0008] According to the remote control system of the present disclosure, since the operation information does not include a delay, by predicting the movement of the moving object based on the operation information and a motion model of the moving object, the prediction accuracy of the position after a specified time is improved, the accuracy of image prediction is also improved, and the operability of the remote control device is also improved. [Brief description of the drawings]

[0009] [Figure 1] 1 is a functional block diagram showing a configuration of a remote control system according to a first embodiment of the present disclosure. [Diagram 2] FIG. 1 is a system configuration diagram showing an example of a schematic configuration in which a moving body is an automobile. [Diagram 3] FIG. 2 is a diagram illustrating a vehicle coordinate system. [Figure 4] FIG. 11 is a diagram illustrating the relationship of communication delay between a remote control device and a mobile object. [Diagram 5] FIG. 11 is a schematic diagram showing an example of a procedure for generating a predicted image in a remote computer. [Figure 6] FIG. 11 is a functional block diagram showing a configuration of a remote control system according to a second embodiment of the present disclosure. [Figure 7] FIG. 11 is a functional block diagram showing a configuration of a remote control system according to a third embodiment of the present disclosure. [Figure 8] FIG. 13 is a functional block diagram showing a configuration of a remote control system according to a modified example of the third embodiment of the present disclosure. [Figure 9] FIG. 11 is a functional block diagram showing a configuration of a remote control system according to a fourth embodiment of the present disclosure. [Figure 10] FIG. 13 is a functional block diagram showing a configuration of a remote control system according to a fifth embodiment of the present disclosure. [Figure 11] FIG. 13 is a functional block diagram showing a configuration of a remote control system according to a sixth embodiment of the present disclosure. [Figure 12] FIG. 23 is a functional block diagram showing a configuration of a remote control system according to a seventh embodiment of the present disclosure. [Figure 13] FIG. 1 is a diagram illustrating a hardware configuration for implementing a remote control system according to first to seventh embodiments of the present disclosure. [Figure 14] FIG. 1 is a diagram illustrating a hardware configuration for implementing a remote control system according to first to seventh embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] <Embodiment 1> 1 is a functional block diagram showing a configuration of a remote control system 1000 according to a first embodiment of the present disclosure. As shown in FIG. 1, the remote control system 1000 includes a moving object 20 and a remote control device 30.

[0011] The moving body 20 is equipped with a camera 21 that acquires an image of the surroundings of the moving body 20 (a first surrounding image), an internal sensor 22 that acquires the state quantity of the moving body 20, a wireless device 26 for communicating with the remote control device 30, an actuator 24 for moving the moving body 20, and a control device 25 for controlling the moving body 20.

[0012] <Mobile object> The mobile body 20 is a movable object, and examples of the mobile body 20 include automobiles, personal mobility devices, mobile robots, drones, ships, airplanes, helicopters, etc., but the following description will be given using an automobile as an example.

[0013] The camera 21 includes a forward camera, and any of a monocular camera, a stereo camera, and a 360-degree camera can be used. When using a monocular camera, a depth image can be generated by known methods such as Visual SLAM (Simultaneous Localization and Mapping) and a neural network (NN). Compared with a stereo camera, images from multiple viewpoints are required, and the amount of movement of the camera between viewpoints also needs to be estimated, but the number of cameras can be reduced, which reduces costs. In addition, Visual SLAM using a camera is cheaper than LiDAR (Light Detection and Ranging), and with technological advances, its accuracy has improved, and it is rapidly becoming popular.

[0014] A method using a neural network is a method in which a monocular video is input and the depth of the image is estimated using a neural network. For example, the method described in "Casser, Vincent, et al. "Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019." can be used.

[0015] When using a stereo camera, a depth image is generated using a known method such as stereo matching. Although the number of cameras increases compared to a monocular camera, the depth can be estimated from an image from one viewpoint, and there is no need to estimate the amount of camera movement between viewpoints, so the estimation accuracy of the depth image is improved.

[0016] The internal sensor 22 includes a vehicle speed sensor, a steering angle sensor, an acceleration sensor, a yaw rate sensor, etc., and has a vehicle speed V, a steering angle δ, a longitudinal acceleration a, and the like as sensor values ​​z. x , acceleration in the left and right direction a y , yaw rate γ, and other vehicle state quantities are acquired.

[0017] The control device 25 controls the moving object 20 based on operation information u from the remote control device 30. The actuators 24 include an EPS motor, an engine, a brake, an EV motor, etc. The wireless device 26 is connected to the camera 21, the internal sensor 22, and the control device 25, and can transmit sensor information acquired by the camera 21 and the internal sensor 22 to the remote control device 30.

[0018] <Remote control device> The remote control device 30 includes an operation device 31 for operating the moving object 20, a display 32, a remote computer 33, and a wireless device 34 for communicating with the moving object 20. The remote operator operates the operation device 31 while watching the display 32 on which the first surrounding image is displayed, to remotely control the moving object 20. However, the communication delay (transmission delay) from the remote control device 30 to the moving object 20 is L. send The communication delay (reception delay) from the moving object 20 to the remote control device 30 is L recv Let us assume that.

[0019] The display 32 displays the predicted image at a constant cycle. The display cycle depends on the generation cycle of the predicted image and can be shorter than the reception cycle of the first surrounding image. This effectively improves the frame rate of the first surrounding image, improving operability. For example, if the first surrounding image is received at 30 fps (frames per second) and the predicted image is generated at 60 fps, the operator sees the scenery around the moving object changing at 60 fps, so that it appears as if the first surrounding image is being received at 60 fps.

[0020] The operation device 31 has a steering wheel, an accelerator pedal, a brake pedal, a joystick, etc. From the operation information u of the operator, a target value to be transmitted to the moving body 20 is calculated by a known method. For example, when the moving body 20 is a vehicle, the steering angle δ of the operation device 30 is calculated by a known method. s , accelerator pedal depression amount p g , brake pedal depression amount p b Based on the above, the target steering angle δ t , target vehicle speed V t Alternatively, the target yaw rate γ t , target acceleration a t etc. are calculated.

[0021] When the operation device 31 is a joystick, the target value is calculated from the inclination of the joystick, etc. Although the target value can be calculated from the operation information u by either the remote operation device 30 or the moving body 20, hereinafter, it is assumed that the calculation is performed by the remote operation device 30. When the calculation is performed by the remote operation device 30, the calculated target value is also referred to as the operation information u in the broad sense, or simply as the operation information u, and the operation information u not including the target value is referred to as the operation information u in the narrow sense. The important point here is that the operation information u does not include a communication delay. Therefore, when the moving body 20 calculates the target value from the operation information u in the narrow sense, it is necessary to perform a movement prediction of the moving body 20 using the operation information u in the narrow sense that does not include a delay.

[0022] In the following description, the operation device 31 is a steering wheel, an accelerator pedal, and a brake pedal, and the moving object 20 is set to a target steering angle δ t , target vehicle speed V t The steering angle δ of the operation device 31 is s , accelerator pedal depression amount p g , brake pedal depression amount p b From the above, the target steering angle δ t and target vehicle speed V t is calculated, for example, using the following formula (1).

[0023]

number

[0024] The upper part of the formula (1) is the target steering angle δ t is the steering angle δ s This shows that the steering angle δ of the operation device 31 can be obtained by substituting s The target steering angle δ t This indicates that it can be used as such.

[0025] The middle equation of equation (1) is the target acceleration a t This is the formula for calculating k g and k b is the control gain.

[0026] In the lower part of formula (1), T s is the control period, and the previous target vehicle speed V t and a t T s From this target vehicle speed V t The operation information u is calculated as u=[δ t ,V t ]

[0027] The remote computer 33 includes a depth image generator 331 , a three-dimensional scene constructor 332 , a moving object movement predictor 333 , a predicted image generator 334 , and an output unit 335 .

[0028] The depth image generator 331 generates a first depth image based on the first surrounding image, and the three-dimensional scene composer 332 composes a three-dimensional scene based on the first surrounding image and the first depth image.

[0029] The moving body movement prediction unit 333 predicts the position and posture of the moving body 20 after a predetermined time based on operation information to the operation device 31 and a motion model of the moving body 20, the predicted image generation unit 334 generates a predicted image after a predetermined time based on the three-dimensional scene and the position and posture after the predetermined time, and the output unit 335 outputs the predicted image to the display 32.

[0030] The function of the remote computer 33 is the operation information u, which is the information available at the current time t=t0. 0:t0 , sensor value z 0:t0-Lrecv and Image I 0:t0-Lrecv From time t0-L recv than the given time L pred The aim of this study is to predict future images only and present them to the remote operator.

[0031] The depth image generating unit 331 calculates the communication delay L from the past (t=0) to the present. recv Only in the past (t=t0-L recv ) Time series data of images up to I 0:t0-Lrecv Using this, a depth image D is generated using a known method such as Visual SLAM or NN. 0:t0-Lrecv Generate.

[0032] The three-dimensional scene construction unit 332 converts a point (u, v) on the first ambient image I into a point (X img ,Y img ,Z img ) to construct a three-dimensional scene. Here, the definition of three-dimensional space is arbitrary, but in the following, we will use the time t=t0-L recv Consider a coordinate system in which the center of gravity of the moving body 20 in is the origin, the front is the positive X-axis, the left is the positive Y-axis, and the upper side is the positive Z-axis.

[0033] It is also possible to store a 3D map of the surroundings of the moving object as a database on the remote control device 30 or on the cloud, and to construct a 3D scene based on that information. In particular, for static targets such as roads and buildings, the accuracy can be improved and blind spots can be reduced by using the 3D map in the database.

[0034] The mobile object movement prediction unit 333 calculates the time series data u of the operation information from the past (t=0) to the present (t=t0). 0:t0 and the communication delay from the past (t=0) to the present, L recv Only in the past (t=t0-L recv ) time series data of state quantity x 0:t0-Lrecv Based on the motion model dx / dt=f(x,u) of the moving object 20, at time t=t0-Lrecv From a given time L pred After t=t0-L recv +L pred The state quantity x including the position and orientation at t0-Lrecv+Lpred Predict.

[0035] A known model is used as the motion model. In the case of a vehicle, a geometric model or a two-wheel model can be used, but the motion model can also be expressed using a NN.

[0036] When using NN, if a part of the state quantity x is estimated from the sensor value, the operation information u and the state quantity x are collected, and the operation model can be learned offline or online based on the time series data. This improves the accuracy of the operation model and also improves the accuracy of the movement prediction. In particular, if the mobile object 20 is equipped with a GNSS (Global Navigation Satellite System) and LiDAR, and the sensor value z includes information on the position and orientation, the accuracy of the position and orientation of the state quantity x is improved, so that if the NN of the operation model is learned using the information as teacher data, the accuracy of the movement prediction is greatly improved.

[0037] The operation information u is a target value to be transmitted to the moving body 20. In the case of a vehicle, for example, a target steering angle δ t and target vehicle speed V t Alternatively, the target yaw rate γ t and the target acceleration a t etc.

[0038] The state quantity x is the horizontal position (X g ,Y g ), including the yaw angle θ. Here, the origin of the XY coordinate system is at time t=t0-L recv The center of gravity of the moving body 20 is defined as the position of the center of gravity of the moving body 20 at the time of the control cycle, and the front side is defined as the positive X-axis direction and the left side is defined as the positive Y-axis direction. g,t0-Lrecv ,Yg, t0-Lrecv ,θ t0-Lrecv Set to 0. If you want to consider three-dimensional movement such as a drone, include the three-dimensional position and roll, pitch, and yaw.

[0039] The state quantity x can use a sensor value acquired by the internal sensor 22, but can also be estimated by a known estimation method such as a Kalman filter. The Kalman filter is a method for identifying the position of a moving object with a smaller error by combining a sensor value and an equation of motion.

[0040] A target steering angle δ is set to the moving body 20. t and target vehicle speed V t When transmitting the target acceleration a to the moving body 20, if the steering angle tracking ability and the speed tracking ability of the moving body 20 are known, the steering angle δ and the vehicle speed V can be estimated with high accuracy only from the operation information u without using the sensor value acquired by the internal sensor 22. t When transmitting the vehicle speed V, it is desirable to also use the vehicle speed V acquired by the internal sensor 22, since an integration error will be accumulated if an attempt is made to estimate the vehicle speed V from only the operation information u.

[0041] Predetermined time L pred For example, L pred =L recv +L send As will be explained later with reference to FIG. t The time when this is reflected is t=t0+L send and time t=t0+L send This is because it is easier to operate if the image in the image is presented to the remote operator, and the information available at the current time is 0:t0 ,x 0:t0-Lrecv and x 0:t0-Lrecv From x t0+Lsend This takes into consideration the need to predict the

[0042] The point here is that the operation information 0:t0 The prediction is made using the operation information u 0:t0 does not include communication delays, so x t0+Lsend However, for a given time L pred can be other values, and the predetermined time L predSince there is a trade-off between the length and accuracy of , it can be set smaller.

[0043] Communication Delay recv and L send If is constant, it can be set to a constant value, and if it varies, the communication delay L recv and L send The delay in receiving the sensor value L can be estimated as a variable. recv,sens and image reception delay L recv,img If it is different from L, the purpose is to compensate for the image delay. pred ≦L recv,img +L send and L recv,img Set the predicted time based on the

[0044] When using a geometric model as the motion model, for example, the state quantity is [X g ,Y g ,θ,V,δ]. In this case, u=[δ t ,V t ], the state equation (operation model) is given by the following equation (2).

[0045]

number

[0046] In formula (2), T d,V and T d,δ is the time constant when the speed tracking ability and steering angle tracking ability are modeled as a first-order lag system. β is the sideslip angle, and γ is the yaw rate, which are given by the following equations (3) and (4), respectively.

[0047]

number

[0048]

number

[0049] However, f and l r are the distances from the center of gravity of the vehicle to the front and rear axles, respectively, and l is the wheelbase, then l = l f +l r It is.

[0050] x t0-Lrecv+Lpred The prediction of is to set the initial state value x t0-Lrecv Let u be the time series of operation information. t0-(Lrecv+Lsend):t0-(Lrecv+Lsend)+Lpred That is, it is calculated by integrating the state equation using the following formula (5).

[0051]

number

[0052] However, L pred ≦L recv +L send In addition, in formula (5), x t0-Lrecv Although X t0-Lrecv ,Y t0-Lrecv ,θ t0-Lrecv is 0 in every control cycle, and the steering angle δ and the vehicle speed V can be estimated from the operation information u, so the state quantity x t0-Lrecv+Lpred It is possible to estimate

[0053] The predicted image generating unit 334 generates the predicted state quantity x of the moving object 20. t0-Lrecv+Lpred The viewpoint p of the camera 21 is moved based on the above, and the three-dimensional scene S is projected from the viewpoint p after the movement by a known method such as perspective projection. t0-Lrecv+Lpred Predicted image as seen from l t0-Lrecv+Lpred Generate.

[0054] As explained above, the generation cycle of the predicted image can be made shorter than the reception cycle of the first surrounding image. In other words, the operation cycle of the predicted image generation unit 334 can be upsampled. This allows the frame rate of the image presented to the remote operator to be upsampled, improving operability. At that time, the operation cycle of the moving object movement prediction unit 333 can also be upsampled, and the prediction time can be adjusted to be consistent with the upsampling. The predicted image generated by the predicted image generation unit 334 can be upsampled. t0-Lrecv+Lpred is displayed on the display 32 via the output unit 335.

[0055] Fig. 2 is a system configuration diagram showing an example of a schematic configuration of a vehicle 1 when a moving body 20 is an automobile. As shown in Fig. 2, the vehicle 1 includes a steering wheel 2, a steering shaft 3, a steering unit 4, an EPS (Electric Power Steering) motor 5, a power train unit 6, and a brake unit 7 as a drive system.

[0056] The sensor system also includes a forward camera 111, a radar sensor 112, a GNSS sensor 121, a steering angle sensor 131, a steering torque sensor 132, a yaw rate sensor 133, a speed sensor 134, and an acceleration sensor 135.

[0057] In addition to these, the vehicle includes a navigation device 122, a vehicle control unit 200, an EPS controller 311, a powertrain controller 312, and a brake controller 313.

[0058] A steering wheel 2, which is installed so that a driver can operate the vehicle 1, is coupled to a steering shaft 3. A steering unit 4 is connected to the steering shaft 3. The steering unit 4 rotatably supports two front tires as steering wheels, and is supported on a vehicle body frame so that it can be steered. Therefore, the torque generated by the driver's operation of the steering wheel 2 rotates the steering shaft 3, and the steering unit 4 steers the front wheels to the left and right. This allows the driver to control the amount of lateral movement of the vehicle 1 when the vehicle 1 moves forward and backward. The steering shaft 3 can also be rotated by an EPS motor 5, and the front wheels can be steered freely independent of the driver's operation of the steering wheel 2 by controlling the current flowing through the EPS motor 5 with an EPS controller 311.

[0059] The vehicle control unit 200 is an integrated circuit such as a microprocessor, and includes an A / D (Analog / Digital) conversion circuit, a D / A (Digital / Analog) conversion circuit, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

[0060] Connected to the vehicle control unit 200 are a forward camera 111, a radar sensor 112, a GNSS sensor 113, a navigation device 122, a steering angle sensor 131 that detects the steering angle, a steering torque sensor 132 that detects the steering torque, a yaw rate sensor 133 that detects the yaw rate, a speed sensor 134 that detects the speed of the vehicle 1, an acceleration sensor 135 that detects the acceleration of the vehicle 1, an EPS controller 311, a powertrain controller 312 and a brake controller 313.

[0061] The vehicle control unit 200 processes information input from the connected sensors according to a program stored in a ROM, transmits a target steering angle to the EPS controller 311, and transmits a target acceleration to the powertrain controller 312 and the brake controller 313.

[0062] The front camera 111 is installed in a position where it can detect the lane markings in front of the vehicle as an image, and detects the environment in front of the vehicle 1, such as lane information and the location of obstacles, based on the image information. Note that, although the present embodiment has been described as an example in which only a camera that detects the environment in front of the vehicle 1 is used, cameras that detect the environment behind and to the sides may also be installed.

[0063] The radar sensor 112 emits radar and detects the reflected waves, thereby outputting the relative distance and relative speed to an obstacle present around the vehicle 1. As this distance measuring sensor, a well-known sensor such as a millimeter wave radar, LiDAR, laser range finder, or ultrasonic radar can be used.

[0064] The GNSS sensor 121 receives radio waves from positioning satellites with an antenna, performs positioning calculations, and outputs the absolute position and absolute direction of the vehicle 1.

[0065] The navigation device 122 has a function of calculating the optimal driving route for the destination set by the driver, and stores road information on the driving route. The road information is map node data that expresses the road linearity, and each map node data incorporates information on latitude, longitude, and altitude that indicate the absolute position at each node, lane width, cant angle, inclination angle information, etc.

[0066] The EPS controller 311 controls the EPS motor 5 based on the target steering angle transmitted from the vehicle control unit 200, thereby controlling the traveling trajectory of the vehicle 1.

[0067] The powertrain controller 312 controls the powertrain unit 6 so as to achieve the target acceleration transmitted from the vehicle control unit 200 .

[0068] In this embodiment, a vehicle that uses an engine as its only driving force source is given as an example, but this embodiment can also be applied to vehicles that use an electric motor as their only driving force source, vehicles that use both an engine and an electric motor as their driving force sources, etc.

[0069] The brake controller 313 controls the brake unit 7 so as to realize the target acceleration transmitted from the vehicle control unit 200 .

[0070] Fig. 3 is a diagram showing a schematic representation of the vehicle coordinate system used in this embodiment. That is, the X-axis and Y-axis in Fig. 3 represent the inertial coordinate system, and g ,Y g , θ represent the center of gravity position and vehicle body direction of vehicle 1 in the inertial coordinate system. The x-axis and y-axis in Fig. 3 are the vehicle coordinate system with the center of gravity of vehicle 1 as the origin, the x-axis pointing forward, and the y-axis pointing to the left. The angle θ is set with the positive direction of the x-axis as the reference direction, and counterclockwise is positive. Here, the center of gravity position X of vehicle 1 is calculated for each execution cycle. g ,Y g and the vehicle direction θ is initialized to 0. In other words, the inertial coordinate system and the host vehicle coordinate system are made to coincide with each execution cycle.

[0071] 4 is a schematic diagram showing the relationship of communication delay between the remote operation device 30 and the moving object 20, where the horizontal axis is the time axis. As shown in FIG. 4, the remote operation device 30 transmits operation information u to the moving object 20, and receives a sensor value z and an image I from the moving object 20.

[0072] The communication delay from the remote control device 30 to the mobile unit 20 is L send The communication delay in receiving the signal at the remote control device 30 is L recv It is.

[0073] There is a communication delay L send and communication delay when receiving L recv By L send +L recv This causes delays.

[0074] The information available at the current time t=t0 is the operation information u 0:t0 , sensor value z 0:t0-Lrecv , state quantity x 0:t0-Lrecv and Image I 0:t0-Lrecv It is.

[0075] Latest operation information t0 is later t=t0+L send The latest image I currently available is reflected in the moving object 20. t0-Lrecv From I t0-Lrecv+Lsend It is desirable to predict the time L and present it to the remote operator. pred is at most L send +L recv Let us assume that.

[0076] FIG. 5 is a schematic diagram showing an example of a procedure for generating a predicted image in the remote computer 33 (FIG. 1). A forward image, which is a first surrounding image captured by the camera 21 of the moving body 20, is input to the depth image generating unit 331 from the wireless device 26 of the moving body 20 via the wireless device 34 of the remote control device 30. As described with reference to FIG. 4, the communication delay in reception at the remote control device 30 is L recv Therefore, the forward image is the time series data I 0:t0-Lrecv The depth image generation unit 331 generates a depth image D as a first depth image by a known method such as Visual SLAM or NN, as described above. t0-Lrecv Generate and output.

[0077] Output depth image D t0-Lrecv is input to the 3D scene construction unit 332, and a 3D scene S is constructed using the original front image and depth image. t0-Lrecv The 3D scene construction unit 332 constructs (reconstructs) and outputs a 3D scene S using the corresponding depth image data and forward image data, using a known 3D scene generation method. t0-Lrecv Configure.

[0078] The output 3D scene S t0-Lrecv is input to a predicted image generating unit 334, and the viewpoint of the camera 21 is moved based on the state quantity including the predicted position and orientation of the moving object 20, and the three-dimensional scene S is generated by a known method such as perspective projection. t0-Lrecv A predicted image is generated as it would appear when viewed from the viewpoint after the movement.

[0079] 5, in order to generate a predicted image for a three-dimensional scene around a moving object represented by a cube, the viewpoint represented by a triangular pyramid is moved from the left to the right according to the movement prediction. This movement of the moving object is calculated by the moving object movement prediction unit 333 using the state quantity x 0:t0-Lrecv and operation information 0:t0 Using the above, a state quantity x including the position and orientation of the moving body 20 after a predetermined time is calculated based on the motion model of the moving body 20. t0-Lrecv+Lpred The state quantity obtained by the moving object movement prediction unit 333 is input to the predicted image generation unit 334.

[0080] As described above, in the remote operation system 1000 of the first embodiment, state quantities including the position and attitude of the moving object 20 after a predetermined time are predicted based on the motion model of the moving object 20 using operation information of the operator via the operation device 31 of the remote operation device 30 and state quantities past the communication delay. Since the operation information does not include communication delay, the movement prediction based on the motion model improves the prediction accuracy of the movement position and also improves the accuracy of image prediction. This improves the operability of the remote operation device 30.

[0081] <Embodiment 2> Fig. 6 is a functional block diagram showing a configuration of a remote control system 2000 according to a second embodiment of the present disclosure. As shown in Fig. 6, the remote control system 2000 includes a moving object 20A and a remote control device 30A. In Fig. 6, the same components as those in the remote control system 1000 described using Fig. 1 are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0082] The moving body 20A further includes an external sensor 23 and a moving body computer 27 in addition to the configuration of the moving body 20 of the first embodiment shown in FIG.

[0083] The external sensor 23 is a sensor that acquires distance information (first distance information) to targets around the moving body 20A, and is a sensor capable of measuring the distance to a target, such as a LiDAR or a millimeter wave radar.

[0084] The mobile computer 27 has a depth image generator 271 that generates a first depth image based on a first surrounding image input from the camera 21 and first distance information input from the external sensor 23. The depth image generator 271 uses the first surrounding image and the first distance information to generate a first depth image by a known method such as coordinate transformation and perspective projection.

[0085] The accuracy of the depth image is improved by using the first distance information acquired by the external sensor 23. However, since the point cloud information acquired from LiDAR or the like has a larger amount of data than the depth image, the amount of communication can be reduced by generating the depth image on the mobile body side and sending it to the remote control device side compared to sending the first distance information acquired by the external sensor 23 to the remote control device side.

[0086] The remote operation device 30A has a remote computer 33A, but the remote computer 33A does not have a depth image generating unit 331 like the remote operation device 30 of the first embodiment shown in Fig. 1. A first depth image generated by the moving object 20A is input to a three-dimensional scene constructing unit 332 of the remote computer 33A from the wireless device 26 of the moving object 20A via the wireless device 34 of the remote operation device 30A, and a three-dimensional scene is constructed in the three-dimensional scene constructing unit 332.

[0087] As described above, in the remote operation system 2000 of the second embodiment, the operability of the remote operation device 30A is improved as in the remote operation system 1000 of the first embodiment, and the accuracy of the depth image is improved by using the first distance information acquired by the external sensor 23. Furthermore, by generating a depth image on the moving body side and sending it to the remote operation device side, the amount of communication can be reduced compared to the case where the first distance information is sent to the remote operation device side.

[0088] <Embodiment 3> Fig. 7 is a functional block diagram showing a configuration of a remote control system 3000 according to a third embodiment of the present disclosure. As shown in Fig. 7, the remote control system 3000 includes a moving object 20, a remote control device 30A, and a roadside unit 40. In Fig. 7, the same components as those in the remote control system 1000 described using Fig. 1 are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0089] The roadside unit 40 includes a roadside computer 41, a roadside sensor 42 that acquires an image of the surroundings of the roadside unit 40 (a second surrounding image) and distance information (second distance information) to surrounding landmarks of the roadside unit 40, and a wireless device 43 for communicating with the mobile body 20 and the remote control device 30A.

[0090] The roadside computer 41 is equipped with a depth image generation unit 411 that generates a first depth image and a second depth image corresponding to the first surrounding image and the second surrounding image, respectively, based on the first surrounding image, the second surrounding image, and the second distance information input from the wireless device 26 of the mobile body 20 via the wireless device 43.

[0091] The roadside sensor 42 is a sensor that acquires images of the surroundings of the roadside unit 40 and information on the distance to targets in the vicinity of the roadside unit 40, and includes cameras such as a monocular camera, a stereo camera, and a 360-degree camera, and sensors that measure the distance to targets such as LiDAR and millimeter-wave radar.

[0092] The roadside unit 40 is installed at the side of a road (roadside) on which vehicles such as the mobile object 20 travel, and is an advanced device that links vehicles with camera information and the like using V2I (Vehicle to Infrastructure) technology, and is linked not only with the vehicle but also with the remote control device 30A via a wireless device 43. When the mobile object 20 enters the detection range of the roadside sensor 42 of the roadside unit 40, the roadside unit 40 acquires an image of the surroundings of the roadside unit 40 and distance information to the surrounding objects of the roadside unit 40, and generates a first depth image and a second depth image.

[0093] The remote operation device 30A has a remote computer 33A, but the remote computer 33A does not have a depth image generating unit 331 like the remote operation device 30 of the first embodiment shown in Fig. 1. The first depth image and the second depth image generated by the depth image generating unit 411 of the roadside unit 40 are input to a three-dimensional scene constructing unit 332 of the remote computer 33A from the wireless device 43 of the roadside unit 40 via the wireless device 34 of the remote operation device 30A, and a three-dimensional scene is constructed in the remote computer 33A.

[0094] As described above, in the remote operation system 3000 of the third embodiment, like the remote operation system 1000 of the first embodiment, the operability of the remote operation device 30A is improved, and by using the surrounding images and distance information of the roadside unit 40, a three-dimensional scene of an area that is a blind spot of the camera 21 of the moving object 20 can be constructed, improving the accuracy of image prediction and the operability of the remote operation device 30A.

[0095] <Modification> Fig. 8 is a functional block diagram showing a configuration of a remote control system 3001 according to a modification of the third embodiment of the present disclosure. As shown in Fig. 8, the remote control system 3001 includes a moving object 20, a remote control device 30, and a roadside unit 40A. In Fig. 8, the same components as those in the remote control system 1000 described using Fig. 1 are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0096] The roadside unit 40A of the remote operation system 3001 shown in FIG. 8 includes a roadside sensor 42 that acquires a surrounding image (second surrounding image) of the roadside unit 40A and distance information (second distance information) to surrounding objects of the roadside unit 40A, and a wireless device 43 for communicating with the moving body 20 and the remote operation device 30.

[0097] The roadside sensor 42 is a sensor that acquires images of the surroundings of the roadside unit 40A and information on distances to targets in the vicinity of the roadside unit 40A, and includes cameras such as a monocular camera, a stereo camera, and a 360-degree camera, and sensors that measure distances to targets such as LiDAR and millimeter-wave radar.

[0098] The roadside unit 40 outputs the second surrounding image and the second distance information acquired by the roadside sensor 42 from the wireless device 43 to the depth image generator 331 via the wireless device 34 of the remote control device 30. The depth image generator 331 generates a first depth image and a second depth image corresponding to the first surrounding image and the second surrounding image, respectively, based on the first surrounding image input from the wireless device 26 of the moving object 20 via the wireless device 34, and the second surrounding image and the second distance information input from the roadside sensor 42 via the wireless device 34.

[0099] The three-dimensional scene construction unit 332 of the remote computer 33 receives the first depth image and the second depth image from the depth image generation unit 331 and constructs a three-dimensional scene.

[0100] As described above, in the remote operation system 3001 according to the modification of the third embodiment, the roadside unit 40A only outputs the second surrounding image and the second distance information acquired by the roadside sensor 42 to the remote operation device 30. Unlike the roadside unit 40 of the remote operation system 3000 according to the third embodiment, the roadside unit 40A does not have the roadside computer 41. Therefore, a known roadside unit can be used, and the cost of building the system can be reduced.

[0101] <Fourth embodiment> Fig. 9 is a functional block diagram showing a configuration of a remote control system 4000 according to a fourth embodiment of the present disclosure. As shown in Fig. 9, the remote control system 4000 includes a moving object 20A, a remote control device 30A, and a roadside unit 40. In Fig. 9, the same components as those in the remote control system 2000 described using Fig. 6 are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0102] The roadside unit 40 includes a roadside computer 41, a roadside sensor 42 that acquires an image of the surroundings of the roadside unit 40 (a second surrounding image) and distance information (second distance information) to surrounding landmarks of the roadside unit 40, and a wireless device 43 for communicating with the mobile body 20A and the remote control device 30A.

[0103] The roadside computer 41 includes a depth image generator 411 that generates a second depth image corresponding to the second surrounding image based on the second surrounding image and the second distance information.

[0104] The roadside sensor 42 is a sensor that acquires images of the surroundings of the roadside unit 40 and information on the distance to targets in the vicinity of the roadside unit 40, and includes cameras such as a monocular camera, a stereo camera, and a 360-degree camera, and sensors that measure the distance to targets such as LiDAR and millimeter-wave radar.

[0105] The depth image generating unit 271 of the moving object 20A generates a first depth image based on the first surrounding image input from the camera 21 and the first distance information input from the external sensor .

[0106] The remote operation device 30A has a remote computer 33A, but the remote computer 33A does not have a depth image generating unit 331 like the remote operation device 30 of the first embodiment shown in Fig. 1. A first depth image generated by the moving object 20A is input to a three-dimensional scene constructing unit 332 of the remote computer 33A from the wireless device 26 of the moving object 20A via the wireless device 34 of the remote operation device 30A, and a three-dimensional scene is constructed in the three-dimensional scene constructing unit 332.

[0107] In addition, the second depth image generated by the roadside unit 40 is input to the three-dimensional scene construction unit 332 from the wireless device 43 of the roadside unit 40 via the wireless device 34 of the remote control device 30A, and a three-dimensional scene is constructed in the three-dimensional scene construction unit 332.

[0108] As described above, in the remote operation system 4000 of the fourth embodiment, the operability of the remote operation device 30A is improved as in the remote operation system 1000 of the first embodiment, and the accuracy of the first depth image is improved by using the first distance information acquired by the external sensor 23. Furthermore, by generating a depth image on the moving body side and sending it to the remote operation device side, the amount of communication can be reduced compared to the case where the first distance information is sent to the remote operation device side.

[0109] In addition, by using the surrounding images and distance information from the roadside unit 40, it is possible to construct a three-dimensional scene of the area that is in the blind spot of the camera 21 of the moving object 20A, improving the accuracy of image prediction and the operability of the remote control device 30A.

[0110] <Fifth Preferred Embodiment> Fig. 10 is a functional block diagram showing a configuration of a remote control system 5000 according to a fifth embodiment of the present disclosure. As shown in Fig. 10, the remote control system 5000 includes a moving object 20 and a remote control device 30B. In Fig. 10, the same components as those in the remote control system 1000 described using Fig. 1 are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0111] The remote control device 30B includes an operating device 31, a display 32, a remote computer 33B, and a wireless device 34. The remote computer 33B further includes a dynamic target movement prediction unit 336 in addition to the configuration of the remote computer 33 of the remote control device 30 shown in FIG.

[0112] The dynamic target movement prediction unit 336 recognizes a dynamic target using the three-dimensional scene output from the three-dimensional scene construction unit 332, estimates the speed of the dynamic target, predicts the dynamic target movement prediction result, which is the position and attitude of the dynamic target after a predetermined time, and inputs it to the predicted image generation unit 334.

[0113] Note that a dynamic target can also be recognized using the first depth image output from the depth image generating unit 331 instead of the three-dimensional scene, and a dynamic target can also be recognized using the first surrounding image.

[0114] Dynamic targets are objects that may move and include, for example, vehicles, people, animals, etc. Roads, buildings, etc. that do not move are static targets.

[0115] Dynamic target recognition can use known techniques such as image processing and neural networks. For example, the position and type of a target in an image can be estimated using YOLO (You Only Look Once).

[0116] YOLO is one of the most representative image analysis methods that utilize artificial intelligence (AI), and uses a Convolutional Neural Network (CNN) to simultaneously detect and identify multiple targets in an image with a single image scan.

[0117] The position and attitude of the dynamic target in the dynamic target movement prediction unit 336 can be predicted by a known method, such as predicting that the recognized dynamic target is moving at a constant speed while maintaining the current speed. In addition, when the dynamic target moves on the road, it can be predicted that the dynamic target moves along the road, and it can be predicted that the dynamic target is turning from the change in the azimuth θ when the vehicle turns.

[0118] In addition, if the moving body 20 is equipped with a millimeter wave radar or the like, the speed of the dynamic target can be estimated from the millimeter wave radar, and the estimated result can be input from the wireless device 26 of the moving body 20 to the dynamic target movement prediction unit 336 via the wireless device 34 of the remote control device 30B, thereby determining the speed of the dynamic target.

[0119] The predicted image generating unit 334 generates a predicted image after a predetermined time based on the dynamic target movement prediction result input from the dynamic target movement predicting unit 336 .

[0120] As described above, in the remote operation system 5000 of the fifth embodiment, the predicted image generating unit 334 generates a predicted image after a predetermined time based on the input dynamic target movement prediction result. That is, the predicted image is generated by moving the dynamic target in the three-dimensional scene based on the dynamic target movement prediction result. By moving the dynamic target, the accuracy of image prediction is improved, and the operability of the remote operation device 30B is improved.

[0121] <Sixth embodiment> Fig. 11 is a functional block diagram showing a configuration of a remote control system 6000 according to a sixth embodiment of the present disclosure. As shown in Fig. 11, the remote control system 6000 includes a moving body 20B and a remote control device 30. In Fig. 11, the same components as those in the remote control system 1000 described using Fig. 1 are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0122] The moving body 20B further includes a moving body calculator 28 in addition to the configuration of the moving body 20 of the first embodiment shown in Fig. 1. The moving body calculator 28 has a target state estimation unit 281 that estimates the state of surrounding targets based on information of the first surrounding image input from the camera 21.

[0123] The state of the surrounding targets includes the position, type, and color of the target, and the state of the surrounding targets can be estimated by known methods such as image processing and NN. For example, the position, type, and color of the target in the image can be estimated by YOLO described above. In addition, by using YOLO, multiple targets in the image can be simultaneously detected and estimated by one image scan.

[0124] The data (second data) including the states of the surrounding targets estimated by the target state estimation unit 281 is transmitted to the remote control device 30 via the wireless device 26 so as to reduce communication delay compared to the data (first data) including the first surrounding image acquired by the camera 21 of the moving body 20. In other words, the second data including the states of the surrounding targets is sent quickly.

[0125] A method for shortening the communication delay of the second data includes a method for separating packets and lines, in which the first data and the second data are transmitted in separate packets or over separate lines.

[0126] The second data including the states of the surrounding targets estimated by the target state estimation unit 281 is input to a predicted image generation unit 334 of the remote computer 33 via the wireless device 34, and the predicted image generation unit 334 generates a predicted image after a predetermined time based on the states of the surrounding targets as well.

[0127] As described above, in the remote control system 6000 of the sixth embodiment, the predicted image generating unit 334 generates a predicted image after a predetermined time based on the state of the surrounding targets estimated by the target state estimating unit 281 of the moving body 20B. Therefore, it is possible to reflect targets not shown in the surrounding image in the predicted image, and to reflect discontinuously changing information such as traffic signal displays in the predicted image.

[0128] Furthermore, by quickly transmitting only information on the estimated state of surrounding targets to the remote control device 30, discontinuous changes in targets not captured in the first surrounding image, such as the appearance of a new obstacle or a change in the color of a traffic light, can be reflected in the predicted image, improving safety.

[0129] Note that transmitting the second data not including the first surrounding image via the wireless device 26 so that the communication delay is shorter than that of the first data including the first surrounding image is not limited to the remote control system 6000 of the sixth embodiment, but can be applied to any of the remote control systems of the first to fifth embodiments. Also, the second data is not limited to data including the state of the surrounding targets.

[0130] <Embodiment 7> Fig. 12 is a functional block diagram showing a configuration of a remote control system 7000 according to a seventh embodiment of the present disclosure. As shown in Fig. 12, the remote control system 7000 includes a moving object 20 and a remote control device 30C. In Fig. 12, the same components as those in the remote control system 1000 described using Fig. 1 are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0131] The remote control device 30C includes an operation device 31, a display 32, a remote computer 33C, a wireless device 34, and a database 35. The remote computer 33C further includes a target type recognition unit 337 in addition to the configuration of the remote computer 33 of the remote control device 30 shown in FIG.

[0132] The database 35 stores data related to the texture of the target in general. The texture includes CG, polygons, images, etc. Images can be referenced from sites such as Google (registered trademark) Street View.

[0133] The target type recognition unit 337 recognizes the type of target in the surrounding image or the image present in the three-dimensional scene. The type can be a rough classification such as "vehicle," "pedestrian," or "building," or a proper noun-level classification such as the vehicle type and the name of the building.

[0134] The type of target in an image can be recognized by known methods such as image processing and NN. For example, the type of target in an image can be recognized by YOLO, which has been described above. In addition, by using YOLO, multiple targets in an image can be simultaneously detected and estimated by scanning the image once.

[0135] The predicted image generation unit 334 assigns textures stored in the database 35 to the three-dimensional scene constructed by the three-dimensional scene construction unit 332 based on the type of target input from the target type recognition unit 337, thereby complementing blind spots in the three-dimensional scene and generating a predicted image.

[0136] For example, textures such as CG are applied to vehicles, roads, guardrails, buildings, etc., classified by the target type recognition unit 337. Also, instead of CG, a published texture of the entire target is applied.

[0137] When the target classified by the target type recognition unit 337 is a vehicle and only the rear part of the vehicle is visible in the three-dimensional scene, CG is assigned to the other parts of the vehicle that are blind spots to complement the view so that the entire vehicle is visible.

[0138] In addition, if the target classified by the target type recognition unit 337 is a vehicle and there is a part of the three-dimensional scene that is hidden by the vehicle and cannot be seen, the hidden part is complemented with an image of Google (registered trademark) Street View.

[0139] The display of the completed areas can also be changed to show them in grayscale or hatched to show that they are less certain.

[0140] In addition, when the predicted image includes a blind spot, the predicted image generation unit 334 can input the predicted image including the blind spot into a trained neural network model without using a database and a target type recognition unit, etc., to complement the blind spot, thereby simplifying the configuration of the remote control device 30.

[0141] As described above, in the remote control system 7000 of embodiment 7, targets are classified by the target type recognition unit 337, and textures stored in the database 35 are assigned to the three-dimensional scene constructed by the three-dimensional scene construction unit 332. This makes it possible to generate predicted images for areas that would be blind spots using only image information, thereby improving prediction accuracy and the operability of the remote control device 30C.

[0142] <Hardware configuration> Each of the components of the remote computers 33-33C, the mobile computer 27 and the roadside computer 41 in the above-described first to seventh embodiments can be configured using a computer, and is realized by the computer executing a program. That is, the remote computers 33-33C, the mobile computer 27 and the roadside computer 41 are realized by, for example, a processing circuit 500 shown in Fig. 13. A processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor) is applied to the processing circuit 500, and the function of each part is realized by executing a program stored in a storage device.

[0143] Dedicated hardware may be applied to the processing circuit 500. When the processing circuit 500 is dedicated hardware, the processing circuit 500 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of these.

[0144] The functions of the components of the remote computers 33 to 33C, the mobile computers 27 and 28, and the roadside computer 41 may be realized by individual processing circuits, or the functions may be realized collectively by a single processing circuit.

[0145] 14 shows a hardware configuration in the case where the processing circuit 500 is configured using a processor. In this case, the functions of each part of the remote computers 33-33C, the mobile computers 27, 28, and the roadside computer 41 are realized by a combination of software, etc. (software, firmware, or software and firmware). The software, etc. are described as a program and stored in the memory 520. The processor 510 functioning as the processing circuit 500 realizes the functions of each part by reading and executing the program stored in the memory 520 (storage device). In other words, it can be said that this program causes a computer to execute the procedure and method of operation of the components of the remote computers 33-33C, the mobile computers 27, 28, and the roadside computer 41.

[0146] Here, memory 520 may be, for example, a non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), HDD (Hard Disk Drive), magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc) and its drive device, or any storage medium to be used in the future.

[0147] The above describes a configuration in which the functions of the components of the remote computers 33-33C, the mobile computers 27, 28, and the roadside computer 41 are realized by either hardware or software. However, the present invention is not limited to this, and some of the components of the remote computers 33-33C, the mobile computers 27, 28, and the roadside computer 41 may be realized by dedicated hardware, and other components may be realized by software, etc. For example, it is possible that the functions of some of the components are realized by the processing circuit 500 as dedicated hardware, and the functions of other components are realized by the processing circuit 500 as the processor 510 reading and executing a program stored in the memory 520.

[0148] As described above, the remote computers 33 to 33C, the mobile computers 27 and 28, and the roadside computer 41 can realize the above-mentioned functions by hardware, software, or a combination of these.

[0149] Although the present disclosure has been described in detail, the above description is illustrative in all respects and is not intended to limit the present disclosure. It is understood that countless variations not illustrated can be envisioned without departing from the scope of the present disclosure.

[0150] In addition, within the scope of the present disclosure, it is possible to freely combine the respective embodiments, and to appropriately modify or omit the respective embodiments.

Claims

1. A remote control system for remotely controlling a moving object from a remote control device, The moving body is acquiring and outputting a first surrounding image of the surroundings of the moving object; The remote control device is an operating device for operating the moving body; a remote computer that calculates a predicted image after a predetermined time based on the first surrounding image output by the moving body, The remote computer a three-dimensional scene construction unit that constructs a three-dimensional scene based on the first ambient image and a first depth image generated based on the first ambient image; a moving object movement prediction unit that predicts a position and an attitude of the moving object after the predetermined time based on operation information to the operation device and a motion model of the moving object; a predicted image generation unit that generates the predicted image based on the three-dimensional scene and the position and orientation after the predetermined time; a dynamic target movement prediction unit that recognizes a dynamic target from the three-dimensional scene, estimates a velocity of the dynamic target, and predicts a dynamic target movement prediction result including a position and an attitude of the dynamic target after the predetermined time, The predicted image generation unit and generating the predicted image based also on the dynamic target movement prediction result.

2. A remote control system for remotely controlling a moving object from a remote control device, The moving body is a target state estimation unit that estimates a state of a target around the moving object based on a first surrounding image of the moving object; acquiring and outputting first data including the first ambient image; outputting second data that does not include the first surrounding image and is in the state of the surrounding target so that a communication delay is shorter than that of the first data; The remote control device is an operating device for operating the moving body; a remote computer that calculates a predicted image after a predetermined time based on the first surrounding image output by the moving body, The remote computer a three-dimensional scene construction unit that constructs a three-dimensional scene based on the first ambient image and a first depth image generated based on the first ambient image; a moving object movement prediction unit that predicts a position and an attitude of the moving object after the predetermined time based on operation information to the operation device and a motion model of the moving object; a predicted image generation unit that generates the predicted image based on the three-dimensional scene and the position and orientation after the predetermined time.

3. A remote control system for remotely controlling a moving object from a remote control device, The moving body is acquiring and outputting a first surrounding image of the surroundings of the moving object; The remote control device is an operating device for operating the moving body; a remote computer that calculates a predicted image after a predetermined time based on the first surrounding image output by the moving object; a database relating to the texture of the target; The remote computer a three-dimensional scene construction unit that constructs a three-dimensional scene based on the first ambient image and a first depth image generated based on the first ambient image; a moving object movement prediction unit that predicts a position and an attitude of the moving object after the predetermined time based on operation information to the operation device and a motion model of the moving object; a predicted image generation unit that generates the predicted image based on the three-dimensional scene and the position and orientation after the predetermined time; a target type recognition unit that recognizes a type of target in the first surrounding image or an image present in the three-dimensional scene, The predicted image generation unit A remote control system that assigns the texture to a blind spot of a target in the image in the three-dimensional scene based on the type of target in the image and the database.

4. The remote computer a depth image generator that generates the first depth image based on the first ambient image; The remote control system according to claim 1 , wherein the first depth image is input to the three-dimensional scene construction unit.

5. The moving body is a mobile object calculator, acquiring first distance information to the surrounding targets of the mobile object and inputting the first distance information to the mobile object calculator; The mobile computer a depth image generator that generates the first depth image based on the first surrounding image and the first distance information; The remote control system according to claim 1 , wherein the first depth image is transmitted to the remote control device.

6. the remote control system includes a roadside unit; The roadside unit comprises: acquiring a second surrounding image of the surroundings of the roadside unit; The three-dimensional scene construction unit The remote control system of claim 1 , wherein the three-dimensional scene is constructed based on the second ambient image and also based on a second depth image generated based on the second ambient image.

7. the remote control system includes a roadside unit; the roadside unit includes a roadside computer; acquiring a second surrounding image of the surroundings of the roadside unit and inputting it into the roadside computer; The roadside computer a depth image generation unit that generates the first depth image and the second depth image corresponding to the first surrounding image and the second surrounding image, respectively, based on the first surrounding image and the second surrounding image output by the moving object; the first depth image and the second depth image are transmitted to the remote control device; The three-dimensional scene construction unit The remote control system according to claim 1 , wherein the three-dimensional scene is constructed also based on the second depth image.

8. The moving body is a mobile object calculator, acquiring first distance information to the surrounding targets of the mobile object and inputting the first distance information to the mobile object calculator; The mobile computer a depth image generator that generates the first depth image based on the first surrounding image and the first distance information; the roadside unit includes a roadside computer; The roadside computer The remote control system according to claim 6 , further comprising a depth image generator that generates the second depth image based on the second surrounding image.

9. The roadside unit comprises: acquiring second distance information to a target object in the vicinity of the roadside unit; The remote control system according to claim 7 , wherein the depth image generator generates the second depth image based also on the second distance information.

10. The dynamic target movement prediction unit The teleoperation system according to claim 1 , wherein the dynamic target is recognized by image processing or a neural network.

11. The dynamic target movement prediction unit The remote operation system according to claim 1 , wherein the dynamic target movement prediction result is predicted based on the second surrounding image or the second depth image.

12. The moving body is The remote control system according to claim 1 or 3, wherein the second data not including the first surrounding image is transmitted so that the communication delay is shorter than that of the first data including the first surrounding image.

13. The moving body is 13. The remote control system according to claim 12, wherein the first data and the second data are transmitted in separate packets or over separate lines so that the communication delay for the second data is shorter.

14. The moving body is a target state estimation unit that estimates a state of a target around the moving object based on the first surrounding image; the moving body transmits the state of the surrounding target as the second data; The predicted image generation unit The remote operation system according to claim 12 , wherein the predicted image is generated based on the state of the surrounding objects.

15. The target state estimation unit The remote operation system according to claim 14, wherein the state of the surrounding target is estimated by image processing or a neural network.

16. The remote control device is Further comprising a database relating to the texture of the target; The remote computer a target type recognition unit that recognizes a type of target in the first surrounding image or an image present in the three-dimensional scene; The predicted image generation unit 3. The remote control system according to claim 1, wherein the texture is assigned to a blind spot of the target in the image in the three-dimensional scene based on the type of the target in the image and the database.

17. The target type recognition unit 17. The remote control system according to claim 16, wherein the type of target in the image is recognized by image processing or a neural network.

18. The predicted image generation unit If the predicted image includes a blind spot, The remote control system according to claim 1 , wherein the blind spots are complemented by a trained neural network model.

19. The remote control device is a database of a three-dimensional map of the surroundings of the moving object; The three-dimensional scene construction unit 4. The remote control system according to claim 1, wherein the three-dimensional scene is constructed based also on the database of three-dimensional maps.

20. The predicted image generation unit The remote control system according to claim 1 , wherein the predicted image is generated at a cycle shorter than a cycle of receiving the first surrounding image.

21. The moving object movement prediction unit 4. The remote control system according to claim 1, wherein the operation model is expressed by a neural network.

22. The moving object movement prediction unit 22. The remote control system according to claim 21, wherein the neural network is trained based on the operation information and a state quantity of the moving object.

23. A remote control device for remotely controlling a moving object, The moving body is acquiring and outputting a first surrounding image of the surroundings of the moving object; The remote control device is an operating device for operating the moving body; a remote computer that calculates a predicted image after a predetermined time based on the first surrounding image output by the moving body, The remote computer a three-dimensional scene construction unit that constructs a three-dimensional scene based on the first ambient image and a first depth image generated based on the first ambient image; a moving object movement prediction unit that predicts a position and an attitude of the moving object after the predetermined time based on operation information to the operation device and a motion model of the moving object; a predicted image generation unit that generates the predicted image based on the three-dimensional scene and the position and orientation after the predetermined time; a dynamic target movement prediction unit that recognizes a dynamic target from the three-dimensional scene, estimates a velocity of the dynamic target, and predicts a dynamic target movement prediction result including a position and an attitude of the dynamic target after the predetermined time, The predicted image generation unit A remote control device that generates the predicted image based also on the dynamic target movement prediction result.

24. A remote control device for remotely controlling a moving object, The moving body is a target state estimation unit that estimates a state of a target around the moving object based on a first surrounding image of the moving object; acquiring and outputting first data including the first ambient image; outputting second data that does not include the first surrounding image and is in the state of the surrounding target so that a communication delay is shorter than that of the first data; The remote control device is an operating device for operating the moving body; a remote computer that calculates a predicted image after a predetermined time based on the first surrounding image output by the moving body, The remote computer a three-dimensional scene construction unit that constructs a three-dimensional scene based on the first ambient image and a first depth image generated based on the first ambient image; a moving object movement prediction unit that predicts a position and an attitude of the moving object after the predetermined time based on operation information to the operation device and a motion model of the moving object; a predicted image generating unit that generates the predicted image based on the three-dimensional scene and the position and orientation after the predetermined time.

25. A remote control device for remotely controlling a moving object, The moving body is acquiring and outputting a first surrounding image of the surroundings of the moving object; The remote control device is an operating device for operating the moving body; a remote computer that calculates a predicted image after a predetermined time based on the first surrounding image output by the moving object; a database relating to the texture of the target; The remote computer a three-dimensional scene construction unit that constructs a three-dimensional scene based on the first ambient image and a first depth image generated based on the first ambient image; a moving object movement prediction unit that predicts a position and an attitude of the moving object after the predetermined time based on operation information to the operation device and a motion model of the moving object; a predicted image generation unit that generates the predicted image based on the three-dimensional scene and the position and orientation after the predetermined time; a target type recognition unit that recognizes a type of target in the first surrounding image or an image present in the three-dimensional scene, The predicted image generation unit a remote control device that assigns the texture to a blind spot of the target in the image in the three-dimensional scene based on the type of the target in the image and the database;

Images