Remote operating system

Abstract

Provided is a remote operation system that appropriately performs delay compensation in remote operation of a moving body. A remote operation system for remote operation of a moving body executes a delay compensation process that compensates for a communication delay between the moving body and a remote operator terminal. The delay compensation processing includes a first delay compensation processing for an image captured by a camera mounted on the moving body, and a second delay compensation processing for remote operation information reflecting an operation amount of a remote operator. The first delay compensation process generates a second image by applying a projective conversion process to a first image captured by a camera, and displays the second image on a display device of a remote operator terminal. The second delay compensation process acquires delay compensation operation information by performing delay compensation on the remote operation information, and controls the moving body in accordance with the delay compensation operation information.

Description

Technical Field

[0001] The present disclosure relates to remote operation of a mobile object by a remote operator. Background Art

[0002] In vehicle remote operation, images (videos) captured by an onboard camera are transmitted from the vehicle to a remote operator terminal and displayed on the remote operator terminal's display device. In this case, it is desirable to compensate for communication delays between the vehicle and the remote operator terminal.

[0003] Patent Document 1 discloses a remote video output device. The device receives video from an autonomous vehicle and estimates changes in the autonomous vehicle's viewpoint based on communication delay time. Taking this change into account, the device crops a portion of the received video frame and displays the cropped image.

[0004] Non-Patent Document 1 discloses a delay compensation technique using projection transformation. More specifically, this delay compensation technique performs a projection transformation on images received from a vehicle, changing the viewpoint based on the amount of vehicle movement corresponding to the delay time, thereby compensating for the delay. In other words, this delay compensation technique visually compensates for the delay by anticipating the camera image as it would be viewed from a viewpoint that is ahead by the delay time. Prior art literature Patent Literature

[0005] Patent Document 1: International Publication No. 2018 / 155159 Non-patent literature

[0006] Non-Patent Literature 1: Matsubara Kodai and Ohmae Manabu, "Study on Delay Compensation of Camera Images for Remote Operation of Automobiles Based on Projection Transformation," 19th ITS Symposium 2021, 4-A-12, December 2021 Summary of the Invention Problems to be solved by the invention

[0007] Consider the possibility of compensating for delay in the image displayed to the remote operator during remote operation of a mobile object. For example, delay compensation can be performed using the technology disclosed in Non-Patent Document 1. However, as the displacement of the vehicle increases, the distortion of the transformed image obtained through the projective transformation also increases. If the distortion of the transformed image is too great, it becomes difficult for the remote operator to understand the situation around the mobile object based on the transformed image. Technical means to solve problems

[0008] One aspect of the present disclosure relates to a remote operation system for remote operation of a mobile object by a remote operator. The remote operation system includes one or more control devices that execute a delay compensation process for compensating for a communication delay between the mobile object and a remote operator terminal on the remote operator side. The delay compensation process includes a first delay compensation process performed on an image captured by a camera mounted on the mobile object, and a second delay compensation process performed on remote operation information reflecting an operation amount of a remote operator. The first delay compensation processing includes: obtaining a first image captured by a camera at a first timing; generating a second image observed from the camera's viewpoint at a second timing that is later than the first timing by a first delay compensation time by applying a projection transformation processing to the first image; and displaying the second image on a display device of a remote operator terminal. The second delay compensation process includes: acquiring delay compensation operation information by performing delay compensation on the remote operation information according to a second delay compensation time; and controlling the moving object according to the delay compensation operation information. Effects of the Invention

[0009] The combination of the first and second delay compensation processes allows the strengths and weaknesses of each to complement each other. As a result, the accuracy and stability of the overall delay compensation process are improved. This contributes to improved accuracy and stability in remote operations performed by remote operator O. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a schematic diagram showing a configuration example of a remote operation system. Figure 2 This is a conceptual diagram for explaining an outline of the first delay compensation process. Figure 3 This is a conceptual diagram used to explain projective transformation based on perspective projection transformation. Figure 4 This is a conceptual diagram for explaining an example of the second delay compensation process. Figure 5 Various examples of combinations of the first delay compensation process and the second delay compensation process are shown. Figure 6 Various examples of combinations of the first delay compensation process and the second delay compensation process are shown. DETAILED DESCRIPTION

[0011] 1. Overview of Remote Operation System Consider remote operation (remote driving) of a mobile object. Examples of mobile objects to be remotely operated include vehicles, robots, and flying objects. Vehicles can be autonomous vehicles or vehicles driven by a driver. Examples of robots include logistics robots and work robots. Examples of flying objects include drones. As an example, in the following description, consider a case where the mobile object is a vehicle. For generalization, the term "vehicle" in the following description is replaced with "mobile object."

[0012] Figure 1 This is a schematic diagram illustrating an example configuration of a remote operation system 1 according to this embodiment. The remote operation system 1 includes a vehicle 100, a remote operator terminal 200, and a management device 300. The vehicle 100 is the target of remote operation. The remote operator terminal 200 is a terminal device used by a remote operator O to remotely operate the vehicle 100. The management device 300 manages the remote operation system 1. Typically, the management device 300 is a management server on the cloud. The management server may also be composed of multiple servers performing distributed processing.

[0013] The vehicle 100, the remote operator terminal 200, and the management device 300 can communicate with each other via the communication network. The vehicle 100 and the remote operator terminal 200 can communicate with each other via the management device 300. Alternatively, the vehicle 100 and the remote operator terminal 200 can communicate directly without intermediary of the management device 300.

[0014] 1-1. Example of vehicle configuration Vehicle 100 includes a communication device 110 , a sensor group 120 , a travel device 130 , and a control device 150 .

[0015] The communication device 110 communicates with the remote operator terminal 200 and the management device 300 .

[0016] The sensor group 120 includes an identification sensor, a vehicle state sensor, a position sensor, and the like. The identification sensor identifies (detects) the surrounding conditions of the vehicle 100. Examples of the identification sensor include a camera CAM, a LIDAR, and a radar. The camera CAM captures the surroundings of the vehicle 100 and obtains an image (video) IMG showing the surrounding conditions of the vehicle 100. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, and the like. The position sensor detects the position and orientation of the vehicle 100. For example, the position sensor includes a GNSS sensor.

[0017] The travel device 130 includes a steering device, a drive device, and a brake device. The steering device steers the wheels. For example, the steering device includes an electric power steering (EPS) device. The drive device is a power source that generates driving force. Examples of drive devices include an engine, an electric motor, and an in-wheel motor. The brake device generates braking force.

[0018] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors and one or more storage devices. The processor performs various processes. Examples of the processor include a CPU, a GPU, an ASIC, and an FPGA. The processor may also be referred to as a circuit or a processing circuit. The storage device stores various information. Examples of the storage device include a volatile memory, a non-volatile memory, an HDD, and an SSD. The functions of the control device 150 may also be realized through the collaboration between the processor that executes the control program and the storage device. The control program is stored in the storage device. The control program may also be recorded in a computer-readable recording medium.

[0019] The control device 150 uses the sensor group 120 to obtain sensor detection information SEN. The sensor detection information SEN includes an image IMG, vehicle state information, position information, object information, etc. The image IMG is captured by the camera CAM. The vehicle state information shows the state of the vehicle 100 (for example, speed, steering angle, etc.) detected by the vehicle state sensor. The position information shows the position and orientation of the vehicle 100 detected by the position sensor. Object information is information related to objects around the vehicle 100 (for example, pedestrians, other vehicles, road structures, traffic lights, signs, etc.). The control device 150 can identify objects around the vehicle 100 by using an identification sensor. The object information includes the relative position and relative speed of the object relative to the vehicle 100.

[0020] The control device 150 performs vehicle travel control for controlling the travel of the vehicle 100. Vehicle travel control includes steering control, drive control, and brake control. The control device 150 performs vehicle travel control by controlling the travel device 130 (steering device, drive device, and brake device).

[0021] The control device 150 can also perform autonomous driving control based on the sensor detection information SEN. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the sensor detection information SEN. Furthermore, based on the sensor detection information SEN, the control device 150 generates a target trajectory for the vehicle 100 to follow the driving plan. The target trajectory includes a target position and a target speed. The control device 150 then controls vehicle driving so that the vehicle 100 follows the target trajectory.

[0022] During remote operation of the vehicle 100, the control device 150 communicates with the remote operator terminal 200 via the communication device 110. The control device 150 transmits at least a portion of the sensor detection information SEN to the remote operator terminal 200. The sensor detection information SEN transmitted to the remote operator terminal 200 includes at least the image IMG captured by the camera CAM. Furthermore, the control device 150 obtains remote operation information OPE, described later, from the remote operator terminal 200. The remote operation information OPE reflects the amount of operation performed by the remote operator O. The control device 150 controls vehicle travel according to the obtained remote operation information OPE.

[0023] 1-2. Configuration example of remote operator terminal The remote operator terminal 200 includes a communication device 210 , a display device 220 , an input device 230 , and a control device 250 .

[0024] The communication device 210 communicates with the vehicle 100 and the management device 300 .

[0025] The display device 220 displays various information for the remote operator O who performs remote operation. In other words, the display device 220 presents various information to the remote operator O by displaying the various information. Examples of the display device 220 include a display and a touch panel.

[0026] The input device 230 is a component that the remote operator O operates when remotely operating the vehicle 100. For example, the input device 230 includes remote operation components. The remote operation components include a steering wheel, an accelerator pedal, a brake pedal, a direction indicator, and the like.

[0027] The control device 250 is a computer that controls the remote operator terminal 200. The control device 250 includes one or more processors and one or more storage devices. The processor performs various processes. Examples of the processor include a CPU, a GPU, an ASIC, and an FPGA. The processor may also be referred to as a circuit or a processing circuit. The storage device stores various information. Examples of the storage device include a volatile memory, a non-volatile memory, an HDD, and an SSD. The functions of the control device 250 may also be realized through the collaboration between the processor that executes the control program and the storage device. The control program is stored in the storage device. The control program may also be recorded in a computer-readable recording medium.

[0028] During remote operation of vehicle 100, control device 250 communicates with vehicle 100 via communication device 210. Control device 250 receives sensor detection information SEN transmitted from vehicle 100. Control device 250 presents necessary information from the received sensor detection information SEN to remote operator O. For example, control device 250 presents image IMG to remote operator O by displaying it on display device 220. Remote operator O can recognize the status of vehicle 100 and surrounding conditions based on the presented information.

[0029] Remote operator O operates input device 230. The amount of operation on input device 230 is detected by a sensor provided on input device 230. Control device 250 generates remote operation information OPE reflecting the amount of operation (steering operation amount, acceleration operation amount, brake operation amount) on input device 230 by remote operator O. Control device 250 then transmits this remote operation information OPE to vehicle 100 via communication device 210. This enables remote operation of vehicle 100.

[0030] 2. Delay compensation processing Remote operation of vehicle 100 involves communication delays between vehicle 100 and remote operator terminal 200. This communication delay can destabilize the behavior of vehicle 100 during remote operation. Furthermore, communication delays can reduce the accuracy of remote operation of vehicle 100. Therefore, performing "delay compensation processing" to compensate for communication delays during remote operation of vehicle 100 is crucial.

[0031] In this embodiment, two types of delay compensation processing are discussed. The "first delay compensation processing" is performed on the image IMG presented to the remote operator O. Meanwhile, the "second delay compensation processing" is performed on the remote operation information OPE reflecting the amount of operation performed by the remote operator O. The first and second delay compensation processing are described below.

[0032] 2-1. First Delay Compensation Process In the first delay compensation process, the remote operation system 1 takes into account the communication delay and performs visual delay compensation on the image IMG displayed on the display device 220. In particular, the remote operation system 1 according to this embodiment uses "projection transformation" to perform visual delay compensation on the image IMG.

[0033] Figure 2This is a conceptual diagram for explaining the outline of the first delay compensation process performed by the remote operation system 1. A first image IMG1 is an image IMG actually captured by the camera CAM mounted on the vehicle 100 at a first timing T1. The first image IMG1 is transmitted from the vehicle 100 to the remote operator terminal 200. The remote operator terminal 200 obtains the first image IMG1 after the first timing T1. If it is possible to estimate (forecast) images IMG to be captured in the future based on this first image IMG1, delay compensation can be performed.

[0034] The second timing T2 is the timing that becomes the foreseeable target, and is a timing that is later than the first timing T1. The difference between the second timing T2 and the first timing T1 is equivalent to the "delay compensation time". The remote operation system 1 may also set at least a part of the round-trip communication delay time between the vehicle 100 and the remote operator terminal 200 as the delay compensation time. The communication delay time between the vehicle 100 and the remote operator terminal 200 can be estimated by a known technique. The delay compensation time may also be set to the communication time from when the image IMG is sent from the vehicle 100 to when it arrives at the remote operator terminal 200. Alternatively, the delay compensation time may also be set to a constant value. In either case, the remote operation system 1 sets the second timing T2 to a delay compensation time that is later than the first timing T1.

[0035] For convenience, the camera CAM at the first timing T1 is referred to as the first camera CAM1, and the camera CAM at the second timing T2 is referred to as the second camera CAM2. The first viewpoint is the viewpoint of the first camera CAM1 and is defined by the combination of the position and orientation of the first camera CAM1 at the first timing T1. The second viewpoint is the viewpoint of the second camera CAM2 and is defined by the combination of the position and orientation of the second camera CAM2 expected at the second timing T2.

[0036] The remote operation system 1 obtains camera information related to the camera CAM mounted on the vehicle 100. The camera information includes the camera CAM's installation information and performance information. The installation information indicates the camera CAM's installation position and installation orientation in the vehicle coordinate system. The performance information indicates the camera CAM's focal length, field of view, and other information. The camera information is provided to the remote operator terminal 200 from the vehicle 100, for example. Since the camera CAM is fixed to the vehicle 100, the camera CAM's installation information can be used to convert the vehicle 100's movement direction and movement amount into the camera CAM's movement direction and movement amount in the camera coordinate system. In other words, based on the camera CAM's installation information and the vehicle 100's movement direction and movement amount, changes in the camera CAM's viewpoint can be estimated.

[0037] More specifically, the remote operation system 1 estimates the direction and amount of movement of the vehicle 100 during the period from the first timing T1 to the second timing T2 (i.e., the delay compensation time). For example, the remote operation system 1 estimates the direction and amount of movement of the vehicle 100 during the period from the first timing T1 to the second timing T2 based on the speed and steering angle of the vehicle 100 at the first timing T1 and the delay compensation time. The information on the speed and steering angle of the vehicle 100 is obtained from the sensor detection information SEN provided by the vehicle 100. Alternatively, the steering angle in the steering operation performed by the remote operator O can also be regarded as the steering angle of the vehicle 100. It can also be assumed that the vehicle 100 performs a stable circular turn. In addition, the remote operation system 1 calculates the difference between the first viewpoint and the second viewpoint based on the above-mentioned camera information (setting information) and the movement amount and movement direction of the vehicle 100 during the delay compensation time.

[0038] The first image IMG1 can be said to be an image IMG taken from the first viewpoint, that is, an image IMG observed from the first viewpoint. Hereinafter, the image IMG expected to be taken from the second viewpoint, that is, the image IMG expected to be observed from the second viewpoint, will be referred to as the "second image IMG2." The remote operation system 1 transforms the first image IMG1 observed from the first viewpoint into the second image IMG2 observed from the second viewpoint based on the difference between the first viewpoint and the second viewpoint. In other words, the remote operation system 1 anticipates (foresees) the second image IMG2 observed from the second viewpoint based on the first image IMG1 observed from the first viewpoint. Projective transformation is used in this foresight.

[0039] Figure 3 This is a conceptual diagram for explaining the projective transformation. The projective transformation is performed based on the perspective projection transformation. The perspective projection transformation is a drawing method for depicting an object in a three-dimensional space on a two-dimensional plane as viewed from the camera CAM. Therefore, the perspective projection transformation takes into account the viewpoint of the camera CAM and projects the points in the three-dimensional space onto the projection plane P. The projection plane P is associated with the camera CAM. For example, the projection plane P is a plane that is orthogonal to the optical axis of the camera CAM. In addition, the points in the three-dimensional space are defined in the three-dimensional world coordinate system (absolute coordinate system). On the other hand, the points projected onto the projection plane P are defined in the two-dimensional image coordinate system.

[0040] For example, N virtual points are virtually set in a three-dimensional world coordinate system. N is an integer greater than or equal to 4. These N virtual points, when viewed from the first camera CAM1 (first viewpoint), are projected onto a first projection plane P1 associated with the first camera CAM1 through a perspective projection transformation. Furthermore, the N virtual points, when viewed from the second camera CAM2 (second viewpoint), are projected onto a second projection plane P2 associated with the second camera CAM2 through a perspective projection transformation. The second viewpoint is obtained based on the difference between the first and second viewpoints. The image coordinates of the virtual points on the first projection plane P1 when viewed from the first camera CAM1 (first viewpoint) are given by [x, y]. On the other hand, the image coordinates of the virtual points on the second projection plane P2 when viewed from the second camera CAM2 (second viewpoint) are given by [x', y']. Based on a comparison of the two, a projective transformation matrix H is calculated for the transformation from the first viewpoint to the second viewpoint. This projective transformation matrix H is then applied to the entire first image IMG1 actually captured by the first camera CAM1, thereby generating a second image IMG2 that is expected to be observed from the second viewpoint.

[0041] As another example, the method described in Non-Patent Document 1 can also be used. Specifically, each image coordinate point on the first image IMG1 (projection plane P) is transformed into a world coordinate point in the world coordinate system through the inverse transformation of the perspective projection transformation. Based on the difference between the first viewpoint and the second viewpoint, the world coordinate point observed from the first viewpoint is transformed into the world coordinate point observed from the second viewpoint. Then, through the perspective projection transformation, the world coordinate point observed from the second viewpoint is returned to the projection plane P. In this way, a second image IMG2 is generated that is expected to be observed from the second viewpoint.

[0042] The first delay compensation process is performed primarily by the remote operator terminal 200, for example. However, the first delay compensation process is not limited to the remote operator terminal 200. At least a portion of the first delay compensation process may also be performed by the vehicle 100 or the management device 300. In either case, the remote operator terminal 200 ultimately obtains the second image IMG2. The remote operator terminal 200 then displays the second image IMG2 on the display device 220. This visually compensates for the communication delay.

[0043] 2-2. Second Delay Compensation Process Remote operation information OPE is information reflecting the amount of operation performed by remote operator O and is transmitted from remote operator terminal 200 to vehicle 100. Vehicle 100 acquires remote operation information OPE at a time at least the communication delay after the remote operation timing of remote operator O. If delay compensation is not performed, the amount of operation performed by remote operator O will be reflected in the behavior of vehicle 100 at a time at least the communication delay after the remote operation timing.

[0044] Therefore, during the second delay compensation process, the remote operation system 1 takes into account the communication delay and performs delay compensation on the remote operation information OPE. More specifically, the remote operation system 1 corrects the remote operation information OPE in such a way as to shorten the time it takes for the operation amount performed by the remote operator O to be reflected in the behavior of the vehicle 100. For convenience, the remote operation information OPE that has undergone this second delay compensation process is referred to as "delay-compensated operation information OPE'." That is, the remote operation system 1 obtains the delay-compensated operation information OPE' by performing delay compensation on the remote operation information OPE according to the delay compensation time. The delay compensation time can also be set to at least a portion of the round-trip communication delay time between the vehicle 100 and the remote operator terminal 200. The communication delay time between the vehicle 100 and the remote operator terminal 200 can be estimated using known techniques. Alternatively, the delay compensation time can be set to a constant value. The remote operation system 1 then controls the vehicle 100 according to the delay-compensated operation information OPE' obtained through the second delay compensation process.

[0045] Figure 4 This is a conceptual diagram illustrating an example of the second delay compensation process. In this example, remote operation information OPE is corrected using model predictive control (MPC). First, based on remote operation information OPE reflecting the operation amount of remote operator O, a target value representing the target vehicle behavior set by remote operator O is calculated. The MPC controller obtains this target value. In addition, the MPC controller obtains a state quantity (output) representing the actual vehicle behavior of vehicle 100, which is the control target. The MPC controller includes an optimization calculator and a prediction model. The prediction model is a model of vehicle 100, which is the control target, and includes, for example, the equation of motion of vehicle 100. The MPC controller calculates the optimal operation quantity for causing the state quantity to follow the target value by using predictions from the prediction model and optimization from the optimization calculator. In particular, in this embodiment, the MPC controller calculates the optimal operation quantity for causing the state quantity to follow the target value while taking into account the hysteresis of the vehicle behavior corresponding to the delay compensation time. The delay compensation operation information OPE' shows the optimal operation quantity thus calculated. The vehicle 100 is then controlled according to this delay compensation operation information OPE'.

[0046] Furthermore, the subject of the second delay compensation process may be the remote operator terminal 200 or the vehicle 100. In other words, the MPC controller may be included in the remote operator terminal 200 or the vehicle 100.

[0047] For example, the remote operator terminal 200 includes an MPC controller and performs a second delay compensation process. In this case, the remote operator terminal 200 obtains remote operation information OPE and performs the second delay compensation process on the remote operation information OPE to obtain delay compensation operation information OPE'. The remote operator terminal 200 transmits the delay compensation operation information OPE' to the vehicle 100. The vehicle 100 receives the delay compensation operation information OPE' from the remote operator terminal 200 and controls the vehicle according to the delay compensation operation information OPE'.

[0048] As another example, vehicle 100 includes an MPC controller and performs a second delay compensation process. In this case, remote operator terminal 200 transmits remote operation information OPE to vehicle 100. Vehicle 100 receives remote operation information OPE from remote operator terminal 200 and performs the second delay compensation process on remote operation information OPE to obtain delay compensation operation information OPE'. Vehicle 100 then controls the vehicle according to the delay compensation operation information OPE'.

[0049] 3. Combination of delay compensation processing 3-1. Overview As described above, the first delay compensation process performs delay compensation on the first image IMG1 based on the projective transformation process. A characteristic of this projective transformation process is that as the displacement of the vehicle 100 increases, the amount of transformation from the first viewpoint to the second viewpoint increases, and the distortion of the second image IMG2 displayed on the display device 220 tends to increase. For example, as the steering angle of the steering operation performed by the remote operator O increases, the distortion of the second image IMG2 increases. As another example, as the speed of the vehicle 100 increases, the distortion of the second image IMG2 increases. If the distortion of the second image IMG2 is excessive, it becomes difficult for the remote operator O to understand the situation around the vehicle 100 based on the second image IMG2.

[0050] Meanwhile, the second delay compensation process is performed on the remote operation information OPE. If the remote operation information OPE changes very rapidly, the second delay compensation process's followability deteriorates. For example, during slalom driving, the steering frequency increases, but the followability of the second delay compensation process in this case is not necessarily high. This deterioration in the followability of the second delay compensation process (e.g., steering hysteresis) reduces the accuracy of the delay compensation operation information OPE'.

[0051] As described above, the first delay compensation process and the second delay compensation process each have their strengths and weaknesses. The first delay compensation process, which utilizes a projection transformation process, may not be suitable in situations where the distortion of the second image IMG2 becomes extremely large (e.g., large steering angles, high vehicle speeds). On the other hand, the second delay compensation process can accurately calculate the delay compensation operation information OPE' even in situations such as large steering angles and high vehicle speeds. However, the second delay compensation process may not be suitable in situations where the followability deteriorates at high steering frequencies. On the other hand, in situations where the distortion of the second image IMG2 does not increase at high steering frequencies, the first delay compensation process is suitable.

[0052] Based on the above insights, the remote operation system 1 of this embodiment performs delay compensation by combining a first delay compensation process with a second delay compensation process. This combination of the first and second delay compensation processes complements each other's strengths and weaknesses. As a result, the overall accuracy and stability of the delay compensation process are improved. This contributes to improved accuracy and stability in remote operations performed by the remote operator O.

[0053] Furthermore, the first and second delay compensation processes can be executed by the same or different entities. For example, both the first and second delay compensation processes can be executed by remote operator terminal 200. As another example, the first delay compensation process can be executed by remote operator terminal 200, while the second delay compensation process can be executed by vehicle 100. In summary, the first and second delay compensation processes are executed by one or more control devices (150, 250, 150, and 250). The first and second delay compensation processes can also be said to be executed by processing circuitry.

[0054] 3-2. Various Examples The delay compensation time in the first delay compensation process will be referred to as the "first delay compensation time" hereinafter. Meanwhile, the delay compensation time in the second delay compensation process will be referred to as the "second delay compensation time" hereinafter. The sum of the first delay compensation time and the second delay compensation time is the "total delay compensation time." The total delay compensation time can be set to at least a portion of the round-trip communication delay time between the vehicle 100 and the remote operator terminal 200. The total delay compensation time can also be set to the round-trip communication delay time between the vehicle 100 and the remote operator terminal 200. The round-trip communication delay time can be estimated using known techniques. Alternatively, the total delay compensation time can be set to a constant value.

[0055] Figure 5: is a conceptual diagram showing various examples of combinations of the first delay compensation process and the second delay compensation process. Figure 5 In the figure, the vertical axis represents the delay compensation time, and the horizontal axis represents various parameters. In addition, "projection transformation" means the first delay compensation process, and "MPC" means the second delay compensation process. Figure 5 In the example shown, the ratio of the first delay compensation time to the second delay compensation time is dynamically changed based on a parameter. More specifically, the first and second delay compensation times are set so that as the first delay compensation time decreases, the second delay compensation time increases. In other words, the first and second delay compensation times are set so that as the first delay compensation time increases, the second delay compensation time decreases.

[0056] Figure 5 [A] in the figure shows the case where the parameter is the steering angle of the steering operation performed by the remote operator O. The steering angle is obtained based on the remote operation information OPE. As the steering angle increases, the first delay compensation time decreases and the second delay compensation time increases (first processing). The change (decrease, increase) of each delay compensation time can be either a monotonic change or a staged change. Since the first delay compensation time decreases as the steering angle increases, the distortion of the second image IMG2 is suppressed. Therefore, the remote operator O can well grasp the situation around the vehicle 100 based on the second image IMG2.

[0057] Figure 5 [B] in the figure shows a case where the parameter is the speed of vehicle 100. The speed of vehicle 100 is obtained based on sensor detection information SEN. As the speed increases, the first delay compensation time decreases and the second delay compensation time increases (second process). The change (decrease, increase) in each delay compensation time can be either monotonic or step-by-step. Since the first delay compensation time decreases as the speed increases, distortion of the second image IMG2 is suppressed. Therefore, remote operator O can better understand the conditions around vehicle 100 based on the second image IMG2.

[0058] Figure 5 [C] in the figure shows the case where the parameter is the steering frequency of the steering operation performed by remote operator O. The steering frequency is obtained based on remote operation information OPE. As the steering frequency increases, the second delay compensation time decreases and the first delay compensation time increases (third process). The change (decrease, increase) in each delay compensation time can be monotonically or in stages. Since the second delay compensation time decreases with increasing steering frequency, deterioration in followability (steering hysteresis) is suppressed.

[0059] Figure 6: is a conceptual diagram showing various examples of combinations of the first delay compensation process and the second delay compensation process. Figure 6 In the figure, the vertical axis represents the delay compensation time, and the horizontal axis represents time. Figure 6 In the illustrated example, at least one of the first delay compensation time and the second delay compensation time is fixed.

[0060] Figure 6 [A] in the figure shows a case where the first delay compensation time is fixed to a first constant value and the second delay compensation time is not fixed. When the first delay compensation time in the first delay compensation process (projection transformation process) changes frequently, "flickering" of the second image IMG2 may occur. By fixing the first delay compensation time, such "flickering" can be prevented. The second delay compensation time can vary according to the actual communication delay time (for example, the actual round-trip communication delay time). For example, the second delay compensation time can also be set to the difference between the actual communication delay time and the first delay compensation time (first constant value).

[0061] As a variation, consider a case where the second delay compensation time is fixed to a second constant value while the first delay compensation time is not fixed. If the second delay compensation time in the second delay compensation process changes frequently, the computational load may be increased. By fixing the second delay compensation time, the computational load can be reduced. The first delay compensation time can vary based on the actual communication delay time (e.g., the actual round-trip communication delay time). For example, the first delay compensation time can also be set to the difference between the actual communication delay time and the second delay compensation time (the second constant value).

[0062] Figure 6 [B] in the figure shows a case where the first delay compensation time is fixed to a first constant value and the second delay compensation time is fixed to a second constant value. The total delay compensation time, the sum of the first and second delay compensation times, can be set to be longer than the normal communication delay time, with a margin.

[0063] Figure 6 [C] in Figure 1 shows a variation of [B]. Here, multiple modes are prepared for the total delay compensation time. Furthermore, the total delay compensation time switches between these modes in conjunction with the actual communication delay time. For example, the mode that is longer than the actual communication delay time and the shortest is preferentially selected. If the actual communication delay time exceeds the total delay compensation time for a certain period of time, the total delay compensation time switches to a new mode. Description of Reference Numerals

[0064] 1…remote operation system, 100…vehicle, 200…remote operator terminal, 300…management device.

Claims

1. A remote operation system for a remote operator to remotely operate a mobile object. One or more control devices are provided for executing a delay compensation process for compensating for a communication delay between the mobile object and the remote operator terminal on the remote operator side, The delay compensation process includes: a first delay compensation process performed on an image captured by a camera mounted on the mobile object; and The second delay compensation process is performed on the remote operation information reflecting the operation amount of the remote operator, wherein the first delay compensation process includes: Obtaining a first image captured by the camera at a first timing; generating a second image observed from the viewpoint of the camera at a second timing later than the first timing by a first delay compensation time by applying a projective transformation process to the first image; and displaying the second image on the display device of the remote operator terminal, The second delay compensation process includes: Obtaining delay compensation operation information by performing delay compensation on the remote operation information according to a second delay compensation time; and The moving object is controlled according to the delay compensation operation information.

2. The remote operation system according to claim 1, wherein: The one or more control devices are further configured to dynamically change a ratio of the first delay compensation time to the second delay compensation time.

3. The remote operation system according to claim 2, wherein: The one or more control devices are further configured to set the first delay compensation time and the second delay compensation time in such a manner that the second delay compensation time increases as the first delay compensation time decreases.

4. The remote operation system according to claim 2, wherein: The one or more control devices are further configured to perform at least one of the following processes: a first process of decreasing the first delay compensation time and increasing the second delay compensation time as the steering angle of the steering operation performed by the remote operator increases; a second process of reducing the first delay compensation time and increasing the second delay compensation time as the speed of the moving object increases; as well as The third process is to increase the first delay compensation time and decrease the second delay compensation time as the steering frequency of the steering operation performed by the remote operator increases.

5. The remote operation system according to claim 1, wherein: The one or more control devices are further configured to fix at least one of the first delay compensation time and the second delay compensation process.

Images