Remote operation system

Abstract

To more properly perform a delay compensation in a remote operation of a moving object.SOLUTION: A remote operation system for remote operation of a moving object performs a delay compensation process so as to compensate for a communication delay between the moving object and a remote operator terminal. The delay compensation process includes: a first delay compensation process which is performed on an image captured by a camera mounted on the moving object; and a second delay compensation process which is executed for remote operation information that reflects an operation amount by a remote operator. The first delay compensation process generates a second image by applying a projective transformation process to a first image taken by the camera, and displays the second image on a display device on the remote operator terminal. The second delay compensation process acquires delay compensation operation information by performing delay compensation on the remote operation information, and controls the moving object according to the delay compensation operation information.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to remote operation of a mobile object by a remote operator. [Background technology]

[0002] In remote control of a vehicle, images (video) captured by an on-board camera are transmitted from the vehicle to a remote operator terminal and displayed on a display device of the remote operator terminal. At this time, it is desirable to compensate for communication delays between the vehicle and the remote operator terminal.

[0003] Patent Document 1 discloses a remote video output device. The remote video output device receives video transmitted from an autonomous vehicle and estimates a change in the viewpoint position of the autonomous vehicle according to a communication delay time. Taking into account the change in the viewpoint position of the autonomous vehicle, the remote video output device cuts out a portion of the frame of the received video and displays the video of the cut-out portion.

[0004] Non-Patent Document 1 discloses a delay compensation technique that uses projective transformation. More specifically, this delay compensation technique performs projective transformation on images received from a vehicle and compensates for the delay by changing the viewpoint according to the amount of vehicle movement equivalent to the delay time. In other words, this delay compensation technique visually compensates for the delay by looking ahead at camera images seen from a viewpoint that is the delay time ahead. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2018 / 155159 [Non-patent literature]

[0006] [Non-Patent Document 1] Kodai Matsubara and Manabu Ohmae, "Research on delay compensation of camera images for remotely controlled automobiles using projective transformation," 19th ITS Symposium 2021, 4-A-12, December 2021 Summary of the Invention [Problem to be solved by the invention]

[0007] In remote control of a moving object, it is possible to perform delay compensation on the image presented to the remote operator. For example, delay compensation is possible using the technology disclosed in Non-Patent Document 1. However, as the displacement of the vehicle increases, the distortion of the transformed image obtained by projective transformation also increases. If the distortion of the transformed image is too large, it becomes difficult for the remote operator to grasp the situation around the moving object based on the transformed image. [Means for solving the problem]

[0008] One aspect of the present disclosure relates to a remote operation system for remotely operating a mobile object by a remote operator. The remote operation system includes one or more control devices that execute a delay compensation process to compensate for a communication delay between a mobile object and a remote operator terminal on the remote operator side. The delay compensation process includes a first delay compensation process performed on an image captured by a camera mounted on a moving body, and a second delay compensation process performed on remote operation information that reflects the amount of operation by a remote operator. The first delay compensation process includes acquiring a first image captured by a camera at a first timing, applying a projective transformation process to the first image to generate a second image seen from the viewpoint of the camera at a second timing that is a first delay compensation time later than the first timing, and displaying the second image on a display device of a remote operator terminal. The second delay compensation process includes acquiring delay compensation operation information by performing delay compensation for the remote operation information by the second delay compensation time, and controlling the moving object in accordance with the delay compensation operation information. [Effects of the Invention]

[0009] By combining the first delay compensation process and the second delay compensation process, it is possible to compensate for the respective advantages and disadvantages of the first delay compensation process and the second delay compensation process. As a result, the accuracy and stability of the delay compensation process as a whole are improved. This leads to improvements in the accuracy and stability of remote operation by the remote operator O. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of the configuration of a remote control system. [Figure 2] FIG. 10 is a conceptual diagram for explaining an overview of a first delay compensation process. [Figure 3] FIG. 1 is a conceptual diagram for explaining a projective transformation based on a perspective projection transformation. [Figure 4] FIG. 10 is a conceptual diagram for explaining an example of a second delay compensation process. [Figure 5] 10 shows various examples of combinations of the first delay compensation process and the second delay compensation process. [Figure 6] 10 shows various examples of combinations of the first delay compensation process and the second delay compensation process. DETAILED DESCRIPTION OF THE INVENTION

[0011] 1. Overview of the remote control system Consider the remote control (remote driving) of a moving object. Examples of moving objects that can be remotely controlled include vehicles, robots, flying objects, etc. The vehicle may be an autonomous vehicle or a vehicle driven by a driver. Examples of robots include logistics robots and work robots. Examples of flying objects include drones, etc. As an example, in the following explanation, consider a case where the moving object is a vehicle. When generalizing, "vehicle" in the following explanation should be read as "moving object."

[0012] FIG. 1 is a schematic diagram showing an example of the configuration of a remote operation system 1 according to this embodiment. The remote operation system 1 includes a vehicle 100, a remote operator terminal 200, and a management device 300. The vehicle 100 is the target of remote operation. The remote operator terminal 200 is a terminal device used by a remote operator O to remotely operate the vehicle 100. The management device 300 manages the remote operation system 1. Typically, the management device 300 is a management server on the cloud. The management server may be composed of multiple servers that perform distributed processing.

[0013] The vehicle 100, the remote operator terminal 200, and the management device 300 can communicate with each other via a communication network. The vehicle 100 and the remote operator terminal 200 can communicate with each other via the management device 300. Alternatively, the vehicle 100 and the remote operator terminal 200 may communicate directly without going through the management device 300.

[0014] 1-1. Example of vehicle configuration The vehicle 100 includes a communication device 110 , a sensor group 120 , a driving device 130 , and a control device 150 .

[0015] The communication device 110 communicates with the remote operator terminal 200 and the management device 300 .

[0016] The sensor group 120 includes a recognition sensor, a vehicle state sensor, a position sensor, etc. The recognition sensor recognizes (detects) the situation around the vehicle 100. Examples of the recognition sensor include a camera CAM, LIDAR, radar, etc. The camera CAM photographs the surroundings of the vehicle 100 and acquires an image (video) IMG showing the situation around the vehicle 100. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, etc. The position sensor detects the position and orientation of the vehicle 100. For example, the position sensor includes a GNSS sensor.

[0017] The traveling device 130 includes a steering device, a drive device, and a braking device. The steering device steers the wheels. For example, the steering device includes an electric power steering (EPS) device. The drive device is a power source that generates driving force. Examples of the drive device include an engine, an electric motor, and an in-wheel motor. The braking device generates braking force.

[0018] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors and one or more storage devices. The processor executes various processes. Examples of the processor include a CPU, a GPU, an ASIC, and an FPGA. The processor can also be called a circuitry or a processing circuitry. The storage device stores various information. Examples of the storage device include a volatile memory, a non-volatile memory, an HDD, and an SSD. The functions of the control device 150 may be realized by cooperation between the processor that executes a control program and the storage device. The control program is stored in the storage device. The control program may be recorded on a computer-readable recording medium.

[0019] The control device 150 acquires sensor detection information SEN using the sensor group 120. The sensor detection information SEN includes an image IMG, vehicle state information, position information, object information, etc. The image IMG is captured by a camera CAM. The vehicle state information indicates the state of the vehicle 100 (e.g., speed, steering angle, etc.) detected by the vehicle state sensor. The position information indicates the position and orientation of the vehicle 100 detected by the position sensor. The object information is information about objects around the vehicle 100 (e.g., pedestrians, other vehicles, road structures, traffic lights, signs, etc.). The control device 150 can recognize objects around the vehicle 100 by using a recognition sensor. The object information includes the relative position and relative speed of the object with respect to the vehicle 100.

[0020] The control device 150 executes vehicle driving control to control the driving of the vehicle 100. The vehicle driving control includes steering control, drive control, and braking control. The control device 150 executes vehicle driving control by controlling the driving device 130 (steering device, drive device, and brake device).

[0021] The control device 150 may perform automatic driving control based on the sensor detection information SEN. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the sensor detection information SEN. Furthermore, the control device 150 generates a target trajectory required for the vehicle 100 to drive according to the driving plan based on the sensor detection information SEN. The target trajectory includes a target position and a target speed. Then, the control device 150 performs vehicle driving control so that the vehicle 100 follows the target trajectory.

[0022] During remote operation of the vehicle 100, the control device 150 communicates with the remote operator terminal 200 via the communication device 110. The control device 150 transmits at least a portion of the sensor detection information SEN to the remote operator terminal 200. The sensor detection information SEN transmitted to the remote operator terminal 200 includes at least an image IMG captured by the camera CAM. The control device 150 also receives remote operation information OPE, which will be described later, from the remote operator terminal 200. The remote operation information OPE is information that reflects the amount of operation by the remote operator O. The control device 150 performs vehicle driving control in accordance with the received remote operation information OPE.

[0023] 1-2. Example of remote operator terminal configuration The remote operator terminal 200 includes a communication device 210 , a display device 220 , an input device 230 , and a control device 250 .

[0024] The communication device 210 communicates with the vehicle 100 and the management device 300 .

[0025] The display device 220 displays various information for the remote operator O who performs remote operation. In other words, the display device 220 displays various information to present the information to the remote operator O. Examples of the display device 220 include a display and a touch panel.

[0026] The input device 230 includes members that the remote operator O operates when remotely operating the vehicle 100. For example, the input device 230 includes remote operation members, such as a steering wheel, an accelerator pedal, a brake pedal, and turn signals.

[0027] The control device 250 is a computer that controls the remote operator terminal 200. The control device 250 includes one or more processors and one or more storage devices. The processor executes various processes. Examples of the processor include a CPU, a GPU, an ASIC, an FPGA, etc. The processor can also be called a circuitry or a processing circuitry. The storage device stores various information. Examples of the storage device include a volatile memory, a non-volatile memory, an HDD, an SSD, etc. The functions of the control device 250 may be realized by cooperation between the processor that executes a control program and the storage device. The control program is stored in the storage device. The control program may be recorded on a computer-readable recording medium.

[0028] During remote operation of the vehicle 100, the control device 250 communicates with the vehicle 100 via the communication device 210. The control device 250 receives sensor detection information SEN transmitted from the vehicle 100. The control device 250 presents necessary information from the received sensor detection information SEN to the remote operator O. For example, the control device 250 presents an image IMG to the remote operator O by displaying the image IMG on the display device 220. The remote operator O can recognize the state of the vehicle 100 and the surrounding situation based on the presented information.

[0029] The remote operator O operates the input device 230. The amount of operation of the input device 230 is detected by a sensor installed on the input device 230. The control device 250 generates remote operation information OPE that reflects the amount of operation of the input device 230 by the remote operator O (amount of steering operation, amount of accelerator operation, amount of brake operation). Then, the control device 250 transmits the remote operation information OPE to the vehicle 100 via the communication device 210. In this way, remote operation of the vehicle 100 is realized.

[0030] 2. Delay compensation processing Remote operation of the vehicle 100 involves a communication delay between the vehicle 100 and the remote operator terminal 200. The communication delay may destabilize the behavior of the vehicle 100 during remote operation. The communication delay may also result in a decrease in the accuracy of the remote operation of the vehicle 100. Therefore, it is important to perform a "delay compensation process" that compensates for the communication delay during remote operation of the vehicle 100.

[0031] In this embodiment, two types of delay compensation processing will be considered. The "first delay compensation processing" is a delay compensation processing performed on the image IMG presented to the remote operator O. On the other hand, the "second delay compensation processing" is a delay compensation processing performed on the remote operation information OPE that reflects the amount of operation by the remote operator O. The first delay compensation processing and the second delay compensation processing will be described below.

[0032] 2-1. First delay compensation process In the first delay compensation process, the remote operation system 1 takes into consideration the communication delay and performs visual delay compensation on the image IMG displayed on the display device 220. In particular, the remote operation system 1 according to this embodiment performs visual delay compensation on the image IMG by using "projective transformation."

[0033] 2 is a conceptual diagram for explaining an outline of the first delay compensation process by the remote operation system 1. The first image IMG1 is an image IMG that is actually captured at a first timing T1 by a camera CAM mounted on the vehicle 100. The first image IMG1 is transmitted from the vehicle 100 to the remote operator terminal 200. The remote operator terminal 200 acquires the first image IMG1 after the first timing T1. If it is possible to estimate (predict) an image IMG that will be captured in the future from the first image IMG1, it becomes possible to perform delay compensation.

[0034] The second timing T2 is a target timing for look-ahead and is later than the first timing T1. The difference between the second timing T2 and the first timing T1 corresponds to the "delay compensation time." The remote operation system 1 may set at least a portion of the round-trip communication delay time between the vehicle 100 and the remote operator terminal 200 as the delay compensation time. The communication delay time between the vehicle 100 and the remote operator terminal 200 can be estimated using well-known techniques. The delay compensation time may be set to the communication time required for the image IMG to be transmitted from the vehicle 100 and reach the remote operator terminal 200. Alternatively, the delay compensation time may be set to a constant value. In either case, the remote operation system 1 sets the second timing T2 to be later than the first timing T1 by the delay compensation time.

[0035] For convenience, the camera CAM at the first timing T1 will be referred to as the first camera CAM1, and the camera CAM at the second timing T2 will be referred to as the second camera CAM2. The first viewpoint is the viewpoint of the first camera CAM1 and is defined by the combination of the position and orientation of the first camera CAM1 at the first timing T1. The second viewpoint is the viewpoint of the second camera CAM2 and is defined by the combination of the predicted position and orientation of the second camera CAM2 at the second timing T2.

[0036] The remote operation system 1 acquires camera information about the camera CAM mounted on the vehicle 100. The camera information includes installation information and performance information about the camera CAM. The installation information indicates the installation position and installation orientation of the camera CAM in the vehicle coordinate system. The performance information indicates the focal length, angle of view, etc. of the camera CAM. The camera information is provided, for example, from the vehicle 100 to the remote operator terminal 200. Because the camera CAM is fixed to the vehicle 100, by using the installation information about the camera CAM, the direction and amount of movement of the vehicle 100 can be converted into the direction and amount of movement of the camera CAM in the camera coordinate system. In other words, a change in the viewpoint of the camera CAM can be estimated based on the installation information about the camera CAM and the direction and amount of movement of the vehicle 100.

[0037] More specifically, the remote control system 1 estimates the direction and amount of movement of the vehicle 100 during the period from the first timing T1 to the second timing T2 (i.e., the delay compensation time). For example, the remote control system 1 estimates the direction and amount of movement of the vehicle 100 during the period from the first timing T1 to the second timing T2 based on the speed and steering angle of the vehicle 100 at the first timing T1 and the delay compensation time. Information on the speed and steering angle of the vehicle 100 is obtained from sensor detection information SEN provided by the vehicle 100. Alternatively, the steering angle in the steering operation by the remote operator O may be considered to be the steering angle of the vehicle 100. The vehicle 100 may be assumed to make a steady circular turn. Then, the remote control system 1 calculates the difference between the first viewpoint and the second viewpoint based on the above-mentioned camera information (installation information) and the amount and direction of movement of the vehicle 100 during the delay compensation time.

[0038] The first image IMG1 can be said to be an image IMG captured from the first viewpoint, i.e., an image IMG viewed from the first viewpoint. An image IMG expected to be captured from the second viewpoint, i.e., an image IMG expected to be seen from the second viewpoint, will hereinafter be referred to as a "second image IMG2." The remote control system 1 converts the first image IMG1 viewed from the first viewpoint into a second image IMG2 viewed from the second viewpoint based on the difference between the first and second viewpoints. In other words, the remote control system 1 predicts (reads ahead) the second image IMG2 viewed from the second viewpoint based on the first image IMG1 viewed from the first viewpoint. Projective transformation is used for this look-ahead.

[0039] FIG. 3 is a conceptual diagram for explaining projective transformation. Projective transformation is performed based on perspective projection transformation. Perspective projection transformation is a rendering technique for rendering an object in three-dimensional space on a two-dimensional plane as seen from the camera CAM. To achieve this, perspective projection transformation projects points in three-dimensional space onto a projection plane P, taking into account the viewpoint of the camera CAM. The projection plane P is associated with the camera CAM. For example, the projection plane P is a plane perpendicular to the optical axis of the camera CAM. Note that points in three-dimensional space are defined in a three-dimensional world coordinate system (absolute coordinate system). On the other hand, points projected onto the projection plane P are defined in a two-dimensional image coordinate system.

[0040] For example, N virtual points are virtually set in a three-dimensional world coordinate system. N is an integer equal to or greater than 4. The N virtual points as viewed from a first camera CAM1 (first viewpoint) are projected onto a first projection plane P1 associated with the first camera CAM1 by perspective projection transformation. The N virtual points as viewed from a second camera CAM2 (second viewpoint) are projected onto a second projection plane P2 associated with the second camera CAM2 by perspective projection transformation. The second viewpoint is obtained from the difference between the first viewpoint and the second viewpoint. The image coordinates of the virtual points on the first projection plane P1 as viewed from the first camera CAM1 (first viewpoint) are given by [x, y]. On the other hand, the image coordinates of the virtual points on the second projection plane P2 as viewed from the second camera CAM2 (second viewpoint) are given by [x', y']. Based on a comparison of the two, a projection transformation matrix H for converting from the first viewpoint to the second viewpoint is calculated. Then, the projective transformation matrix H is applied to the entire first image IMG1 actually captured by the first camera CAM1, thereby generating a second image IMG2 that is expected to be seen from the second viewpoint.

[0041] As another example, the method described in Non-Patent Document 1 may be used. Specifically, by inverse transformation of the perspective projection transformation, each image coordinate point on the first image IMG1 (projection plane P) is transformed into a world coordinate point in the world coordinate system. Based on the difference between the first viewpoint and the second viewpoint, the world coordinate point as seen from the first viewpoint is transformed into a world coordinate point as seen from the second viewpoint. Then, by perspective projection transformation, the world coordinate point as seen from the second viewpoint is returned onto the projection plane P. This generates a second image IMG2 that is expected to be seen from the second viewpoint.

[0042] The subject of the first delay compensation process is, for example, the remote operator terminal 200. However, the subject of the first delay compensation process is not limited to the remote operator terminal 200. At least a part of the first delay compensation process may be executed by the vehicle 100 or the management device 300. In either case, the remote operator terminal 200 ultimately acquires the second image IMG2. Then, the remote operator terminal 200 displays the second image IMG2 on the display device 220. This visually compensates for the communication delay.

[0043] 2-2. Second delay compensation process The remote operation information OPE is information reflecting the amount of operation by the remote operator O, and is transmitted from the remote operator terminal 200 to the vehicle 100. The vehicle 100 receives the remote operation information OPE at least a communication delay after the timing of the remote operation by the remote operator O. If delay compensation is not performed, the amount of operation by the remote operator O will be reflected in the behavior of the vehicle 100 at least a communication delay time after the timing of the remote operation.

[0044] Therefore, in the second delay compensation process, the remote operation system 1 performs delay compensation on the remote operation information OPE, taking the communication delay into consideration. More specifically, the remote operation system 1 corrects the remote operation information OPE so as to shorten the time until the operation amount by the remote operator O is reflected in the behavior of the vehicle 100. For convenience, the remote operation information OPE that has undergone the second delay compensation process is referred to as "delay-compensated operation information OPE'." That is, the remote operation system 1 acquires the delay-compensated operation information OPE' by performing delay compensation on the remote operation information OPE by the delay compensation time. The delay compensation time may be set to at least a portion of the round-trip communication delay time between the vehicle 100 and the remote operator terminal 200. The communication delay time between the vehicle 100 and the remote operator terminal 200 can be estimated using well-known techniques. Alternatively, the delay compensation time may be set to a constant value. Then, the remote operation system 1 controls the vehicle 100 according to the delay-compensated operation information OPE' obtained by the second delay compensation process.

[0045] FIG. 4 is a conceptual diagram illustrating an example of the second delay compensation process. In this example, the remote operation information OPE is corrected by using model predictive control (MPC). First, a target value representing the target vehicle behavior targeted by the remote operator O is calculated from the remote operation information OPE reflecting the operation amount by the remote operator O. The MPC controller receives the target value. The MPC controller also receives a state variable (output) representing the actual vehicle behavior of the vehicle 100, which is the control target. The MPC controller has an optimization calculator and a prediction model. The prediction model is a model of the vehicle 100, which is the control target, and includes, for example, the equation of motion of the vehicle 100. The MPC controller calculates an optimal operation variable for the state variable to track the target value through prediction using the prediction model and optimization using the optimization calculator. Particularly in this embodiment, the MPC controller calculates an optimal operation variable for the state variable to track the target value, taking into account the delay in the vehicle behavior corresponding to the delay compensation time. Delay compensation operation information OPE' indicates the optimal operation variable calculated in this manner. Then, the vehicle 100 is controlled in accordance with the delay compensation operation information OPE'.

[0046] The subject of the second delay compensation process may be the remote operator terminal 200 or the vehicle 100. In other words, the MPC controller may be included in the remote operator terminal 200 or the vehicle 100.

[0047] For example, the remote operator terminal 200 has an MPC controller and performs the second delay compensation processing. In this case, the remote operator terminal 200 acquires remote operation information OPE and acquires delay compensation operation information OPE' by performing the second delay compensation processing on the remote operation information OPE. The remote operator terminal 200 transmits the delay compensation operation information OPE' to the vehicle 100. The vehicle 100 receives the delay compensation operation information OPE' from the remote operator terminal 200 and controls the vehicle in accordance with the delay compensation operation information OPE'.

[0048] As another example, the vehicle 100 has an MPC controller and performs the second delay compensation processing. In this case, the remote operator terminal 200 transmits remote operation information OPE to the vehicle 100. The vehicle 100 receives the remote operation information OPE from the remote operator terminal 200 and performs the second delay compensation processing on the remote operation information OPE to obtain delay compensation operation information OPE'. Then, the vehicle 100 performs vehicle control according to the delay compensation operation information OPE'.

[0049] 3. Combination of delay compensation processes Overview As described above, the first delay compensation process performs delay compensation on the first image IMG1 based on the projective transformation process. One feature of this projective transformation process is that as the displacement of the vehicle 100 increases, the amount of transformation from the first viewpoint to the second viewpoint also increases, and the distortion of the second image IMG2 displayed on the display device 220 tends to increase. For example, as the steering angle of the steering operation by the remote operator O increases, the distortion of the second image IMG2 increases. As another example, as the speed of the vehicle 100 increases, the distortion of the second image IMG2 increases. If the distortion of the second image IMG2 is too large, it becomes difficult for the remote operator O to grasp the situation around the vehicle 100 based on the second image IMG2.

[0050] On the other hand, the second delay compensation process is performed on the remote operation information OPE. When the remote operation information OPE changes very quickly, the tracking ability of the second delay compensation process deteriorates. For example, in the case of slalom driving, the steering frequency becomes high, but in that case the tracking ability of the second delay compensation process is not necessarily high. Deterioration in the tracking ability of the second delay compensation process (e.g., steering delay) leads to a decrease in the accuracy of the delay compensation operation information OPE'.

[0051] As described above, the first delay compensation process and the second delay compensation process each have their advantages and disadvantages. The first delay compensation process, which uses projective transformation processing, is not necessarily appropriate in situations where the distortion of the second image IMG2 becomes very large (e.g., large steering angles, high vehicle speeds). On the other hand, the second delay compensation process can accurately calculate the delay compensation operation information OPE' even in situations such as large steering angles and high vehicle speeds. However, the second delay compensation process is not necessarily appropriate in situations where the steering frequency is high and tracking performance deteriorates. On the other hand, in situations where the steering frequency is high, the distortion of the second image IMG2 does not become large, so the first delay compensation process is appropriate.

[0052] Based on the above findings, the remote operation system 1 according to this embodiment performs delay compensation processing by combining the first delay compensation processing and the second delay compensation processing. By combining the first delay compensation processing and the second delay compensation processing, it becomes possible to compensate for the respective advantages and disadvantages of the first delay compensation processing and the second delay compensation processing. As a result, the accuracy and stability of the delay compensation processing as a whole are improved. This leads to improvements in the accuracy and stability of remote operation by the remote operator O.

[0053] The first delay compensation process and the second delay compensation process may be performed by the same or different entities. For example, both the first delay compensation process and the second delay compensation process may be performed by the remote operator terminal 200. As another example, the first delay compensation process may be performed by the remote operator terminal 200, and the second delay compensation process may be performed by the vehicle 100. In general, the first delay compensation process and the second delay compensation process are performed by one or more control devices (150, 250, 150 and 250). It can also be said that the first delay compensation process and the second delay compensation process are performed by processing circuitry.

[0054] 3-2.Various examples The delay compensation time in the first delay compensation process is hereinafter referred to as the "first delay compensation time." On the other hand, the delay compensation time in the second delay compensation process is hereinafter referred to as the "second delay compensation time." The sum of the first delay compensation time and the second delay compensation time is the "total delay compensation time." The total delay compensation time may be set to at least a portion of the round-trip communication delay time between the vehicle 100 and the remote operator terminal 200. The total delay compensation time may be set to the round-trip communication delay time between the vehicle 100 and the remote operator terminal 200. The round-trip communication delay time can be estimated using well-known techniques. Alternatively, the total delay compensation time may be set to a constant value.

[0055] FIG. 5 is a conceptual diagram showing various examples of combinations of the first delay compensation process and the second delay compensation process. In FIG. 5, the vertical axis represents the delay compensation time, and the horizontal axis represents various parameters. Furthermore, "projective transformation" refers to the first delay compensation process, and "MPC" refers to the second delay compensation process. In the example shown in FIG. 5, the ratio between the first delay compensation time and the second delay compensation time is dynamically changed according to the parameters. More specifically, the first delay compensation time and the second delay compensation time are set so that the second delay compensation time increases as the first delay compensation time decreases. In other words, the first delay compensation time and the second delay compensation time are set so that the second delay compensation time decreases as the first delay compensation time increases.

[0056] [A] in FIG. 5 shows a case where the parameter is the steering angle of the steering operation by the remote operator O. The steering angle is obtained from the remote operation information OPE. As the steering angle increases, the first delay compensation time decreases and the second delay compensation time increases (first process). The change (decrease, increase) in each delay compensation time may be a monotonic change or a step-by-step change. Since the first delay compensation time decreases as the steering angle increases, distortion of the second image IMG2 is suppressed. Therefore, the remote operator O can clearly grasp the situation around the vehicle 100 based on the second image IMG2.

[0057] [B] in FIG. 5 shows a case where the parameter is the speed of the vehicle 100. The speed of the vehicle 100 is obtained from the sensor detection information SEN. As the speed increases, the first delay compensation time decreases and the second delay compensation time increases (second process). The change (decrease, increase) in each delay compensation time may be a monotonic change or a step-by-step change. Since the first delay compensation time decreases as the speed increases, distortion of the second image IMG2 is suppressed. Therefore, the remote operator O can clearly grasp the situation around the vehicle 100 based on the second image IMG2.

[0058] [C] in Figure 5 shows a case where the parameter is the steering frequency of the steering operation by the remote operator O. The steering frequency is obtained from the remote operation information OPE. As the steering frequency increases, the second delay compensation time decreases and the first delay compensation time increases (third process). The change (decrease, increase) in each delay compensation time may be a monotonic change or a step-by-step change. As the second delay compensation time decreases as the steering frequency increases, deterioration of tracking ability (steering delay) is suppressed.

[0059] Fig. 6 is a conceptual diagram showing various examples of combinations of the first delay compensation process and the second delay compensation process. In Fig. 6, the vertical axis represents the delay compensation time, and the horizontal axis represents time. In the example shown in Fig. 6, at least one of the first delay compensation time and the second delay compensation time is fixed.

[0060] [A] in FIG. 6 shows a case where the first delay compensation time is fixed to a first constant value and the second delay compensation time is not fixed. If the first delay compensation time in the first delay compensation process (projection transformation process) changes frequently, flickering of the second image IMG2 may occur. Fixing the first delay compensation time makes it possible to prevent such flickering. The second delay compensation time may vary depending on the actual communication delay time (e.g., the actual round-trip communication delay time). For example, the second delay compensation time may be set to the difference between the actual communication delay time and the first delay compensation time (first constant value).

[0061] As a modified example, the second delay compensation time may be fixed to a second constant value, and the first delay compensation time may not be fixed. If the second delay compensation time in the second delay compensation process changes frequently, the calculation processing may become heavy. By fixing the second delay compensation time, it is possible to reduce the load of the calculation processing. The first delay compensation time may vary depending on the actual communication delay time (e.g., the actual round-trip communication delay time). For example, the first delay compensation time may be set to the difference between the actual communication delay time and the second delay compensation time (second constant value).

[0062] [B] in Fig. 6 shows a case where the first delay compensation time is fixed to a first constant value and the second delay compensation time is fixed to a second constant value. The total delay compensation time, which is the sum of the first delay compensation time and the second delay compensation time, may be set with a margin so as to be longer than a typical communication delay time.

[0063] [C] in Figure 6 shows a modified example of [B]. Here, multiple patterns are prepared for the total delay compensation time. The total delay compensation time is switched among the multiple patterns in conjunction with the actual communication delay time. For example, from the multiple patterns, one that is longer than the actual communication delay time and has the shortest length is selected with priority. If the actual communication delay time exceeds the total delay compensation time for a certain period of time, the total delay compensation time is switched to a new pattern. [Explanation of symbols]

[0064] 1... remote operation system, 100... vehicle, 200... remote operator terminal, 300... management device

Claims

1. A remote operation system for remotely operating a mobile object by a remote operator, comprising: one or more control devices that execute a delay compensation process to compensate for a communication delay between the mobile object and a remote operator terminal on the remote operator side; The delay compensation process includes: a first delay compensation process performed on an image captured by a camera mounted on the moving object; a second delay compensation process performed on the remote operation information reflecting the operation amount by the remote operator; Including, The first delay compensation process includes: acquiring a first image captured by the camera at a first timing; generating a second image seen from a viewpoint of the camera at a second timing that is a first delay compensation time later than the first timing by applying a projective transformation process to the first image; displaying the second image on a display device of the remote operator terminal; Including, The second delay compensation process includes: acquiring delay-compensated operation information by performing delay compensation for the remote operation information by a second delay compensation time; controlling the moving body according to the delay compensation operation information; Contains Remote control system.

2. The one or more control devices are further configured to dynamically change a ratio between the first delay compensation time and the second delay compensation time. The remote control system according to claim 1 .

3. The one or more control devices are further configured to set the first delay compensation time and the second delay compensation time such that the second delay compensation time increases as the first delay compensation time decreases. The remote control system according to claim 2 .

4. The one or more control devices may further include: a first process of decreasing the first delay compensation time and increasing the second delay compensation time as the steering angle of the steering operation by the remote operator increases; a second process of decreasing the first delay compensation time and increasing the second delay compensation time as the speed of the moving object increases; a third process of increasing the first delay compensation time and decreasing the second delay compensation time in accordance with an increase in a steering frequency of the steering operation by the remote operator; configured to perform at least one of The remote control system according to claim 2 .

5. The one or more control devices are further configured to fix at least one of the first delay compensation time and the second delay compensation process. The remote control system according to claim 1 .

Images