Control device, control method, and computer-readable recording medium

Abstract

The present disclosure relates to a control device, a control method, and a computer-readable recording medium, and provides a technology by which a remote operator can more smoothly and safely perform remote operation of a mobile body. The control device controls a mobile body that is an object of remote operation performed by a remote operator. During remote operation of the mobile body, the control device communicates with a remote operator terminal on the remote operator side, and receives remote operation information including a first operation amount performed by the remote operator. The control device acquires a delay amount of communication from the remote operator terminal to the mobile body, based on a result of reception of the remote operation information. Furthermore, the control device performs delay compensation processing of calculating a first corrected operation amount by compensating for delay of the first operation amount based on the delay amount. Moreover, the control device controls the mobile body based on the first corrected operation amount.

Description

Technical Field

[0001] The present disclosure relates to a technology for controlling a mobile object that is a target of remote operation by a remote operator. Background Art

[0002] Patent Document 1 discloses a driving control system for remotely controlling a vehicle. A remote control device communicates wirelessly with a vehicle-side driving control device. The driving control device calculates the delay time of wireless communication with the remote control device. If the delay time exceeds a threshold, the driving control device changes the vehicle's driving control from normal control to safety control. For example, safety control reduces the vehicle speed compared to normal control.

[0003] Furthermore, Patent Documents 2 and 3 disclose technologies related to remote control of a vehicle.

[0004] In the field of measurement, Kalman filters are generally known. Kalman filters estimate the state of a system based on measurement data. However, there is a deviation between the timing of data measurement and the timing of Kalman filter operation. As this deviation increases, the estimation accuracy decreases.

[0005] Non-Patent Documents 1 and 2 disclose a “Delayed Kalman Filter” that can also be applied to measurement data containing delays. The Delayed Kalman Filter is also called OOSM (Out-of-Sequence Measurement).

[0006] Prior art literature

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Application Laid-Open No. 2016-071585

[0009] Patent Document 2: Japanese Patent Application Laid-Open No. 2004-295360

[0010] Patent Document 3: Japanese Patent Application Laid-Open No. 2021-061516

[0011] Non-patent literature

[0012] Non-patent literature 1: Yaakov Bar-Shalom, "Update with Out-of-SequenceMeasurements in Tracking: Exact Solution," IEEE Transactions on Aerospace and Electronic Systems, VOL.38, No.3, pp.769-778, July 2002.

[0013] Non-patent literature 2: Keshu Zhang et al., "Optimal Update with Out-of-SequenceMeasurements," IEEE Transactions on Signal Processing, Vol.53, No.6, pp.1992-2004, June 2005. Summary of the Invention

[0014] Consider remote operation of a mobile object (e.g., a vehicle or robot) by a remote operator. The remote operation system consists of the mobile object (the target of the remote operation) and a remote operator terminal on the remote operator's side. During remote operation, the mobile object and the remote operator terminal communicate with each other. The mobile object receives information about the amount of operation performed by the remote operator from the remote operator terminal and is controlled accordingly.

[0015] There is a delay in communication between the remote operator terminal and the mobile object. Therefore, it takes time for the remote operator's operations to be reflected in the mobile object's control. As a result, there is a possibility that the remote operator's remote operation of the mobile object may become unsmooth. In some cases, the mobile object may meander. This is undesirable from the perspective of the safety of remote operation of the mobile object.

[0016] One object of the present disclosure is to provide a technology that enables a remote operator to perform remote control of a mobile object more smoothly and safely.

[0017] A first aspect relates to a control device that controls a moving object that is the target of remote operation by a remote operator.

[0018] The control device includes one or more processors.

[0019] One or more processors configured to perform:

[0020] During remote operation of the mobile object, a process of communicating with a remote operator terminal on a remote operator side to receive remote operation information including a first operation amount performed by the remote operator;

[0021] a process for obtaining a delay amount of communication from a remote operator terminal to a mobile object based on a result of receiving remote operation information;

[0022] a delay compensation process of calculating a first correction operation amount by compensating for a delay of the first operation amount according to the delay amount; and

[0023] A process of controlling a movable body based on a first correction operation amount.

[0024] The second aspect relates to a method of controlling a moving object that is the target of remote operation by a remote operator.

[0025] Control methods include:

[0026] During remote operation of the mobile object, a process of communicating with a remote operator terminal on a remote operator side to receive remote operation information including a first operation amount performed by the remote operator;

[0027] a process for obtaining a delay amount of communication from a remote operator terminal to a mobile object based on a result of receiving remote operation information;

[0028] a delay compensation process of calculating a first correction operation amount by compensating for a delay of the first operation amount according to the delay amount; and

[0029] A process of controlling a movable body based on a first correction operation amount.

[0030] The third aspect relates to a control program for controlling a mobile object that is the target of remote operation by a remote operator.

[0031] The control program causes the computer to:

[0032] During remote operation of the mobile object, a process of communicating with a remote operator terminal on a remote operator side to receive remote operation information including a first operation amount performed by the remote operator;

[0033] a process for obtaining a delay amount of communication from a remote operator terminal to a mobile object based on a result of receiving remote operation information;

[0034] a delay compensation process of calculating a first correction operation amount by compensating for a delay of the first operation amount according to the delay amount; and

[0035] A process of controlling a movable body based on a first correction operation amount.

[0036] According to the present disclosure, a delay compensation process is performed on a mobile object to compensate for the delay of a first operation variable performed by a remote operator. The mobile object is controlled based on the first corrected operation variable obtained through the delay compensation process. This allows the remote operator to perform remote operation of the mobile object more smoothly and safely. It also suppresses snaking of the mobile object. In other words, according to the present disclosure, the safety of remote operation of the mobile object is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram showing a configuration example of a remote operation system according to an embodiment of the present disclosure.

[0038] Figure 2 This is a block diagram showing a configuration example of a vehicle according to an embodiment of the present disclosure.

[0039] Figure 3 This is a conceptual diagram used to illustrate the topic.

[0040] Figure 4 This is a conceptual diagram for explaining a delay compensation process in a vehicle according to an embodiment of the present disclosure.

[0041] Figure 5 This is a conceptual diagram for explaining a delay compensation process and an upsampling process in a vehicle according to an embodiment of the present disclosure.

[0042] Figure 6 This is a block diagram illustrating a functional configuration example related to delay compensation processing in a vehicle according to an embodiment of the present disclosure.

[0043] Figure 7 This is a block diagram showing a configuration example of a delay compensation unit according to an embodiment of the present disclosure.

[0044] Figure 8 This is a conceptual diagram for explaining an abnormality determination process and an operation amount arbitration process in a vehicle according to an embodiment of the present disclosure.

[0045] Figure 9 This is a block diagram illustrating a functional configuration example related to abnormality determination processing and operation amount arbitration processing in a vehicle according to an embodiment of the present disclosure.

[0046] Figure 10 This is a flowchart illustrating an example of communication abnormality determination processing according to an embodiment of the present disclosure.

[0047] Figure 11 This is a flowchart illustrating an example of communication abnormality determination processing according to an embodiment of the present disclosure.

[0048] Figure 12 This is a flowchart illustrating an example of terminal abnormality determination processing according to an embodiment of the present disclosure.

[0049] Figure 13 1 is a flowchart illustrating an example of vehicle abnormality determination processing according to an embodiment of the present disclosure.

[0050] Figure 14 It is a diagram for explaining an example of operation amount determination processing according to an embodiment of the present disclosure.

[0051] Figure 15 This is a block diagram illustrating an example of a functional configuration associated with various processes in a vehicle according to an embodiment of the present disclosure.

[0052] Figure 16 This is a block diagram showing a configuration example of a remote operator terminal according to an embodiment of the present disclosure.

[0053] (Explanation of Symbols)

[0054] 1: Remote operation system; 100: Vehicle; 101: Communication device; 102: Sensor group; 103: Driving device; 105: Control device; 106: Processor; 107: Storage device; 110: Receiving unit; 120: Delay compensation unit; 130: Control unit; 140: Automatic driving control unit; 150: Operation amount determination unit; 160: Communication anomaly determination unit; 170: Terminal anomaly determination unit; 180: Vehicle anomaly determination unit; 200: Remote operator terminal; 300: Management device; DL: Delay amount; FL-C: Communication anomaly flag; FL-V: Vehicle anomaly flag; FL-T: Terminal anomaly flag; OPE: Remote operation information; VCL: Vehicle information; X: Operation amount; X1: First operation amount; X1C: First correction operation amount; X2: Second operation amount. DETAILED DESCRIPTION

[0055] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0056] 1. Overview of Remote Operation System

[0057] Consider remote operation (remote driving) of mobile objects. Examples of mobile objects that can be remotely operated include vehicles, robots, and aircraft. Vehicles can be either autonomous vehicles or vehicles driven by a human operator. Examples of robots include logistics robots and work robots. Examples of aircraft include airplanes and drones.

[0058] As an example, in the following description, a case where the mobile object to be remotely operated is a vehicle is considered. In general, the term "vehicle" in the following description is replaced with "mobile object".

[0059] Figure 1: is a schematic diagram showing a structural example of the remote operation system 1 of the present embodiment. The remote operation system 1 includes a vehicle 100, a remote operator terminal 200, and a management device 300. The vehicle 100 is the object of remote operation. The remote operator terminal 200 is a terminal device used when the remote operator O remotely operates the vehicle 100. The remote operator terminal 200 can also be referred to as a remote operation HMI (Human Machine Interface). The management device 300 manages the remote operation system 1. The management of the remote operation system 1 includes, for example, assigning a remote operator O to a vehicle 100 that requires remote operation. The management device 300 can communicate with the vehicle 100 and the remote operator terminal 200 via a communication network. Typically, the management device 300 is a management server on the cloud. The management server can also be composed of multiple servers that perform distributed processing.

[0060] The vehicle 100 is equipped with various sensors, including a camera. The camera captures the surrounding conditions of the vehicle 100, acquiring image information representing the surrounding conditions. Vehicle information VCL is information acquired by the various sensors, including image information acquired by the camera. The vehicle 100 transmits the vehicle information VCL to the remote operator terminal 200 via the management device 300. Specifically, the vehicle 100 transmits the vehicle information VCL to the management device 300, and the management device 300 transmits the received vehicle information VCL to the remote operator terminal 200.

[0061] The remote operator terminal 200 receives the vehicle information VCL transmitted from the vehicle 100. The remote operator terminal 200 presents the vehicle information VCL to the remote operator O. Specifically, the remote operator terminal 200 includes a display device, and displays image information and the like on the display device. The remote operator O observes the displayed information, recognizes the conditions around the vehicle 100, and performs remote operation of the vehicle 100. The remote operation information OPE is information related to the remote operation performed by the remote operator O. For example, the remote operation information OPE includes the amount of operation performed by the remote operator O. The remote operator terminal 200 transmits the remote operation information OPE to the vehicle 100 via the management device 300. That is, the remote operator terminal 200 transmits the remote operation information OPE to the management device 300, and the management device 300 transmits the received remote operation information OPE to the vehicle 100.

[0062] The vehicle 100 receives the remote operation information OPE transmitted from the remote operator terminal 200. The vehicle 100 controls the vehicle's travel in accordance with the received remote operation information OPE. In this way, remote operation of the vehicle 100 is achieved.

[0063] 2. Vehicle Example

[0064] 2-1. Structural example

[0065] Figure 2 1 is a block diagram showing a configuration example of a vehicle 100 . The vehicle 100 includes a communication device 101 , a sensor group 102 , a travel device 103 , and a control device 105 .

[0066] The communication device 101 communicates with the outside of the vehicle 100 . For example, the communication device 101 communicates with the remote operator terminal 200 and the management device 300 .

[0067] The sensor group 102 includes an identification sensor, a vehicle state sensor, a position sensor, and the like. The identification sensor identifies (detects) the surrounding conditions of the vehicle 100. Examples of the identification sensor include a camera, LIDAR (Laser Imaging Detection and Ranging), and a radar. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, and the like. The position sensor detects the position and orientation of the vehicle 100. For example, the position sensor includes a GNSS (Global Navigation Satellite System).

[0068] The travel device 103 includes a steering device, a drive device, and a brake device. The steering device steers the wheels. For example, the steering device includes an Electric Power Steering (EPS) device. The drive device is a power source that generates driving force. Examples of drive devices include an engine, an electric motor, and an in-wheel motor. The brake device generates braking force.

[0069] The control device 105 is a computer that controls the vehicle 100. The control device 105 includes one or more processors 106 (hereinafter referred to as the processor 106) and one or more storage devices 107 (hereinafter referred to as the storage device 107). The processor 106 performs various processes. For example, the processor 106 includes a CPU (Central Processing Unit). The storage device 107 stores various information required for the processes implemented by the processor 106. Examples of the storage device 107 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), and the like. The control device 105 may also include one or more ECUs (Electronic Control Units).

[0070] The vehicle control program PROG1 is a computer program executed by the processor 106. The functions of the control device 105 are realized by executing the vehicle control program PROG1 by the processor 106. The vehicle control program PROG1 is stored in the storage device 107. Alternatively, the vehicle control program PROG1 may be recorded on a computer-readable recording medium.

[0071] 2-2. Driving environment information

[0072] The control device 105 uses the sensor group 102 to obtain driving environment information ENV indicating the driving environment of the vehicle 100 . The driving environment information ENV is stored in the storage device 107 .

[0073] Driving environment information ENV includes surrounding condition information representing the recognition results obtained by the recognition sensor. For example, surrounding condition information includes image information captured by a camera. Surrounding condition information may also include object information related to objects surrounding vehicle 100. Examples of objects surrounding vehicle 100 include pedestrians, other vehicles (such as preceding vehicles and parked vehicles), white lines, signals, signs, and roadside structures. Object information indicates the relative position and speed of the object relative to vehicle 100.

[0074] In addition, the driving environment information ENV includes vehicle state information indicating a vehicle state detected by a vehicle state sensor.

[0075] Furthermore, the driving environment information ENV includes vehicle position information indicating the position and orientation of the vehicle 100. The vehicle position information is obtained by a position sensor. High-precision vehicle position information can also be obtained through localization using map information and surrounding situation information (object information).

[0076] 2-3. Vehicle driving control

[0077] The control device 105 performs vehicle travel control for controlling the travel of the vehicle 100. Vehicle travel control includes steering control, drive control, and brake control. The control device 105 performs vehicle travel control by controlling the travel device 103 (steering device, drive device, and brake device).

[0078] The control device 105 can also perform autonomous driving control based on the driving environment information ENV. More specifically, the control device 105 generates a driving plan for the vehicle 100 based on the driving environment information ENV. Furthermore, the control device 105 generates a target trajectory based on the driving environment information ENV for the vehicle 100 to follow the driving plan. The target trajectory includes a target position and a target speed. Furthermore, the control device 105 controls vehicle driving so that the vehicle 100 tracks the target trajectory.

[0079] 2-4. Processing related to remote operation

[0080] Hereinafter, a case where the vehicle 100 is remotely controlled will be described. The control device 105 communicates with the remote operator terminal 200 via the communication device 101 .

[0081] The control device 105 transmits vehicle information VCL to the remote operator terminal 200. Vehicle information VCL is information required for remote operation by the remote operator O and includes at least a portion of the aforementioned driving environment information ENV. For example, vehicle information VCL includes surrounding condition information (particularly image information). Vehicle information VCL may also include vehicle status information and vehicle location information.

[0082] Furthermore, the control device 105 receives remote operation information OPE from the remote operator terminal 200. The remote operation information OPE is information regarding remote operations performed by the remote operator O. For example, the remote operation information OPE includes the amount of operation performed by the remote operator O. The control device 105 performs vehicle driving control according to the received remote operation information OPE.

[0083] Hereinafter, improvement of safety in remote operation of vehicle 100 will be examined in detail.

[0084] 3. Delay compensation processing and upsampling processing

[0085] 3-1. Overview

[0086] First, refer to Figure 3 The comparative example shown illustrates the topic studied in this section. The horizontal axis represents time, and the vertical axis represents the amount of operation performed by remote operator O. Hereinafter, the amount of operation performed by remote operator O will be referred to as "first operation amount X1." The first operation amount X1 is included in the remote operation information OPE transmitted from the remote operator terminal 200 to the vehicle 100. The control device 105 of the vehicle 100 controls vehicle driving according to the first operation amount X1 included in the received remote operation information OPE.

[0087] Figure 3The upper section in FIG. 1 shows the temporal variation of the first manipulated variable X1 by the remote operator O. The circled symbols in the upper section indicate the first manipulated variable X1 transmitted to the vehicle 100 and the timing of its transmission. Meanwhile, the lower section shows the temporal variation of the first manipulated variable X1 used for control in the vehicle 100. The circled symbols in the lower section indicate the first manipulated variable X1 received by the vehicle 100 and the timing of its reception.

[0088] The first issue is the delay in communication from the remote operator terminal 200 to the vehicle 100. The delay DL of this communication is not constant but varies depending on the communication status. Figure 3 As shown, the first operation variable X1 on the vehicle 100 side is delayed by the delay amount DL from the first operation variable X1 on the remote operator O side. In other words, it takes at least the delay amount DL for the first operation variable X1 implemented by the remote operator O to be actually reflected in the vehicle control. As a result, there is a possibility that the remote operation of the vehicle 100 by the remote operator O may become unsmooth. In some cases, the vehicle 100 may even meander. This is not desirable from the perspective of the safety of remote operation of the vehicle 100.

[0089] The second topic is the difference between the communication cycle and the control cycle. Figure 3 As shown, the communication cycle of the remote operation information OPE (the first operation amount X1) is longer than the control cycle for controlling the vehicle 100. In other words, the control frequency for controlling the vehicle 100 is higher than the communication frequency of the remote operation information OPE. Therefore, the first operation amount X1 used for control in the vehicle 100 deviates from the first operation amount X1 actually performed by the remote operator O. The greater the difference between the communication cycle and the control cycle, the greater the deviation of the first operation amount X1 becomes. In addition, the greater the communication delay DL, the greater the deviation of the first operation amount X1 becomes. Due to such a deviation of the first operation amount X1, there is also the possibility that the remote operation performed by the remote operator O on the vehicle 100 will become jerky. Depending on the situation, the vehicle 100 sometimes snakes. From the perspective of the safety of the remote operation of the vehicle 100, this is not preferred.

[0090] The vehicle 100 of the present embodiment is configured to solve at least one of the above-mentioned problems.

[0091] Figure 4This is a conceptual diagram for explaining the "delay compensation process" in the vehicle 100 of the present embodiment. The control device 105 of the vehicle 100 is configured to compensate for the delay of the first operation variable X1 on the vehicle 100 side. More specifically, the control device 105 obtains information on the communication delay amount DL based on the result of receiving the remote operation information OPE from the remote operator terminal 200. Furthermore, the control device 105 compensates for the delay of the first operation variable X1 based on the delay amount DL. This process is the delay compensation process. The control device 105 controls the vehicle 100 based on the first operation variable X1 after the delay compensation process. The first problem is solved by such a delay compensation process. In addition, a specific example of the delay compensation process will be described later.

[0092] Figure 5 This is a conceptual diagram for explaining the "upsampling process" in the vehicle 100 of the present embodiment. The control device 105 of the vehicle 100 is configured to increase the sampling frequency (sampling rate) of the first operation quantity X1 by inferring (predicting) the first operation quantity X1 during a period without sampling. This process is an upsampling process. In particular, the control device 105 performs the upsampling process in such a manner that the difference between the sampling period (sampling frequency) and the control period (control frequency) of the first operation quantity X1 is smaller than the difference between the communication period (communication frequency) and the control period (control frequency). For example, the control device 105 performs the upsampling process in such a manner that the sampling period (sampling frequency) and the control period (control frequency) of the first operation quantity X1 are consistent. The second problem is solved by such an upsampling process. In addition, a specific example of the upsampling process will be described later.

[0093] In this embodiment, the control device 105 of the vehicle 100 is configured to perform at least one of delay compensation and upsampling. Alternatively, the control device 105 may be configured to perform both. This allows the remote operator O to more smoothly and safely remotely operate the vehicle 100. This also suppresses snaking behavior in the vehicle 100. In other words, this embodiment improves the safety of remote operation of the vehicle 100.

[0094] 3-2. Functional structure example

[0095] Figure 6 1 is a block diagram showing an example of a functional configuration related to delay compensation processing in vehicle 100. Vehicle 100 includes a receiving unit 110, a delay compensation unit 120, and a control unit 130 as functional blocks. These functional blocks are implemented by communication device 101 and control device 105.

[0096] During remote operation of the vehicle 100, the receiving unit 110 receives remote operation information OPE transmitted from the remote operator terminal 200. The remote operation information OPE includes information about the first operation amount X1 performed by the remote operator O. Based on the result of receiving the remote operation information OPE, the receiving unit 110 understands the communication status with the remote operator terminal 200. Examples of the communication status include the presence or absence of data reception, the delay DL, the communication speed, and the radio wave reception strength.

[0097] The delay compensation unit 120 performs a delay compensation process. Specifically, the delay compensation unit 120 obtains information about the first operation quantity X1 and the delay quantity DL from the receiving unit 110. The delay quantity DL is not constant, but varies according to the communication state. The delay compensation unit 120 compensates for the delay of the first operation quantity X1 according to the delay quantity DL. Hereinafter, the first operation quantity X1 after delay compensation is referred to as the "first corrected operation quantity X1C". It can be said that the delay compensation unit 120 calculates the first corrected operation quantity X1C by compensating for the delay of the first operation quantity X1 according to the delay quantity DL. The delay compensation unit 120 can also perform the delay compensation process while performing the upsampling process. In the following section 3-3, a specific example of the delay compensation unit 120 is described.

[0098] The control unit 130 receives the first corrected operation amount X1C after the delay compensation process and performs vehicle travel control according to the first corrected operation amount X1C.

[0099] 3-3. Example of Delay Compensation Unit

[0100] Figure 7 : is a block diagram showing a configuration example of the delay compensation unit 120 . Figure 7 The delay compensating unit 120 shown is configured to perform delay compensation processing simultaneously with upsampling processing. More specifically, the delay compensating unit 120 includes a correcting unit 121 , an estimating unit 122 , and an output unit 123 .

[0101] The correction unit 121 receives information about the first operation variable X1 performed by the remote operator O from the remote operator terminal 200. Furthermore, the correction unit 121 receives information about the delay amount DL. The delay amount DL is not constant but fluctuates depending on the communication status. The correction unit 121 performs delay compensation processing based on the first operation variable X1 and the delay amount DL to calculate a first corrected operation variable X1C. The output unit 123 outputs the first corrected operation variable X1C.

[0102] The estimation unit 122 estimates (predicts) the first correction manipulated variable X1C during the non-sampling period based on the first correction manipulated variable X1C output from the output unit 123. In other words, the estimation unit 122 performs upsampling. Specifically, the estimation unit 122 performs upsampling so that the difference between the sampling frequency of the first correction manipulated variable X1C and the control frequency is smaller than the difference between the communication frequency and the control frequency.

[0103] For example, the estimation unit 122 holds an equation of motion representing the motion of the vehicle 100. The estimated value of the first correction operation variable X1C output from the estimation unit 122 is fed back to the estimation unit 122 via the output unit 123. Based on the equation of motion and the estimated value of the first correction operation variable X1C, the estimation unit 122 estimates (predicts) the first correction operation variable X1C at the next timing. However, the estimated value of the first correction operation variable X1C contains an error. This error is corrected by the correction unit 121.

[0104] Specifically, the estimated value of the first correction manipulated variable X1C output from the estimation unit 122 is also input to the correction unit 121. Meanwhile, the first manipulated variable X1 (actual value) performed by the remote operator O, received from the remote operator terminal 200, is also input to the correction unit 121. As described above, the communication cycle is longer than the control cycle, and the control frequency is higher than the communication frequency. Therefore, the first manipulated variable X1 (actual value) is input to the correction unit 121 at a lower frequency than the estimated value of the first correction manipulated variable X1C. The correction unit 121 calculates the first correction manipulated variable X1C by correcting the estimated value of the first correction manipulated variable X1C based on the first manipulated variable X1 (actual value), while performing delay compensation based on the delay amount DL. The first correction manipulated variable X1C calculated by the correction unit 121 is fed back to the estimation unit 122 via the output unit 123. The estimation unit 122 performs upsampling based on the first correction manipulated variable X1C output from the output unit 123.

[0105] so, Figure 7 The delay compensating unit 120 shown in FIG. performs delay compensation simultaneously with upsampling. The delay compensating unit 120 feeds back the estimated value of the first corrected manipulated variable X1C obtained through upsampling and the first corrected manipulated variable X1C obtained through delay compensation, thereby estimating the first corrected manipulated variable X1C during periods without sampling. Furthermore, while performing delay compensation based on the delay amount DL, the delay compensating unit 120 corrects the estimated value of the first corrected manipulated variable X1C based on the first manipulated variable X1 (actual value) to calculate the first corrected manipulated variable X1C.

[0106] The delay compensator 120 that performs upsampling and delay compensation can be implemented, for example, by utilizing a delayed Kalman filter. A delayed Kalman filter is also known as an OOSM (Out-of-Sequence Measurement). For details on the delayed Kalman filter, please refer to Non-Patent Documents 1 and 2. The delayed Kalman filter can also be applied to the first manipulated variable X1 with varying delay amount DL. The delay compensator 120 performs upsampling and delay compensation by applying the delayed Kalman filter to the first manipulated variable X1 with varying delay amount DL.

[0107] In order to improve the estimation accuracy of the first correction operation amount X1C estimated by the estimation unit 122, supplementary information SUP can also be used. For example, the supplementary information SUP includes the shape of the road on which the vehicle 100 moves. For example, when the vehicle 100 is traveling on a curve, information on the shape of such a curve is used to estimate the future operation amount. The road shape is obtained from, for example, surrounding condition information. Alternatively, the road shape can also be obtained from vehicle position information and map information. The supplementary information SUP can also include the control amount of the driving device 103 (actuator) of the vehicle 100, that is, the vehicle driving control amount. The vehicle driving control amount is also useful for the estimation processing performed by the estimation unit 122. Such supplementary information SUP is provided to the delay compensation unit 120 from the control unit 130. The estimation unit 122 also estimates (predicts) the first correction operation amount X1C taking into account the supplementary information SUP. As a result, the estimation accuracy of the first correction operation amount X1C is further improved.

[0108] 3-4. Effect

[0109] As described above, the control device 105 of the vehicle 100 in this embodiment is configured to perform at least one of delay compensation and upsampling. Alternatively, the control device 105 may be configured to perform both. This allows the remote operator O to more smoothly and safely remotely operate the vehicle 100. This also suppresses snaking behavior in the vehicle 100. In other words, according to this embodiment, the safety of remote operation of the vehicle 100 is improved.

[0110] 4. Abnormal determination and operation amount determination

[0111] 4-1. Overview

[0112] Next, consider "abnormalities" that may occur in the remote operation system 1. Abnormality encompasses failures, malfunctions, abnormalities, and malfunctions. Examples of abnormalities in the remote operation system 1 include abnormalities in the communication between the vehicle 100 and the remote operator terminal 200 (i.e., "communication abnormalities"); abnormalities in the vehicle 100 (i.e., "vehicle abnormalities"); and abnormalities in the remote operator terminal 200 (i.e., "terminal abnormalities").

[0113] Figure 8 This is a conceptual diagram used to illustrate abnormality-related processing in remote operation system 1. In this embodiment, control device 105 of vehicle 100 performs "abnormality determination processing" to determine whether an abnormality has occurred in remote operation system 1. Specifically, control device 105 distinguishes between communication abnormalities, vehicle abnormalities, and terminal abnormalities when making these determinations. In other words, control device 105 determines whether at least one of these abnormalities has occurred.

[0114] If an abnormality occurs during remote operation of vehicle 100, it is desirable to appropriately control vehicle 100 to ensure safety. In this case, controlling vehicle 100 based on the first manipulated variable X1 performed by remote operator O may not be appropriate. For example, if a terminal abnormality occurs, the first manipulated variable X1 transmitted from remote operator terminal 200 may also be abnormal. In some cases, it may be safer to delegate driving to the automatic driving function (autonomous driving function) of vehicle 100.

[0115] Therefore, the control device 105 of the vehicle 100 of this embodiment performs an "operation amount determination process" to appropriately determine the final operation amount X based on the abnormal situation. The first operation amount X1 is the operation amount performed by the remote operator O and is included in the remote operation information OPE. On the other hand, the second operation amount X2 is the operation amount determined (requested) by the automatic driving function of the vehicle 100. In the operation amount determination process, the control device 105 determines the "final operation amount X" based on at least one of the first operation amount X1 and the second operation amount X2 according to the result of the abnormality determination process (i.e., the abnormal situation). Then, the control device 105 controls the vehicle 100 according to the determined operation amount X.

[0116] For example, if a terminal abnormality occurs, vehicle 100 is controlled according to the second operation variable X2 instead of the first operation variable X1, thereby improving safety. As another example, even if a communication abnormality occurs, vehicle 100 is controlled according to the second operation variable X2 instead of the first operation variable X1, thereby improving safety.

[0117] Thus, according to this embodiment, the control device 105 of the vehicle 100 is configured to perform an abnormality determination process and an operation variable determination process. In the operation variable determination process, the control device 105 determines a "final operation variable X" based on at least one of the first operation variable X1 and the second operation variable X2, based on the results of the abnormality determination process. The vehicle 100 is then controlled in accordance with the determined operation variable X. In this way, if an abnormality occurs in the remote operation system 1, the vehicle 100 can be appropriately controlled taking into account the abnormal situation.

[0118] 4-2. Functional structure example

[0119] Figure 9 This is a block diagram illustrating an example of the functional configuration associated with abnormality determination processing and operation amount arbitration processing in vehicle 100. Vehicle 100 includes a receiving unit 110, a control unit 130, a communication abnormality determination unit 160, a terminal abnormality determination unit 170, and a vehicle abnormality determination unit 180 as functional blocks. These functional blocks are implemented by communication device 101 and control device 105.

[0120] During remote operation of the vehicle 100, the receiving unit 110 receives remote operation information OPE transmitted from the remote operator terminal 200. The remote operation information OPE includes information about the first operation amount X1 performed by the remote operator O. Based on the result of receiving the remote operation information OPE, the receiving unit 110 understands the communication status with the remote operator terminal 200. Examples of the communication status include the presence or absence of data reception, the delay DL, the communication speed, and the radio wave reception strength.

[0121] The control unit 130 receives information regarding the first manipulated variable X1 via the receiving unit 110. Furthermore, the control unit 130 includes an automatic driving control unit 140 and an manipulated variable determination unit 150. The automatic driving control unit 140 performs the aforementioned automatic driving control based on the driving environment information ENV. The manipulated variable requested by this automatic driving control is the second manipulated variable X2. The manipulated variable determination unit 150 performs a manipulated variable determination process. Specifically, the manipulated variable determination unit 150 determines a "final manipulated variable X" based on at least one of the first manipulated variable X1 and the second manipulated variable X2.

[0122] The communication abnormality determination unit 160 performs "communication abnormality determination processing". More specifically, the communication abnormality determination unit 160 obtains information related to the communication status from the receiving unit 110. Moreover, the communication abnormality determination unit 160 determines whether an abnormality has occurred in the communication from the remote operator terminal 200 to the vehicle 100 based on the communication status. A specific example of the communication abnormality determination processing will be described later (see Section 4-3). The communication abnormality flag FL-C is information indicating the result of the communication abnormality determination processing. The communication abnormality determination unit 160 outputs the communication abnormality flag FL-C to the operation amount determination unit 150.

[0123] The terminal abnormality determination unit 170 performs "terminal abnormality determination processing". More specifically, the terminal abnormality determination unit 170 determines whether an abnormality has occurred in the remote operator terminal 200 based on the first operation amount X1 included in the remote operation information OPE. A specific example of the terminal abnormality determination processing will be described later (see Section 4-4). The terminal abnormality flag FL-T is information indicating the result of the terminal abnormality determination processing. The terminal abnormality determination unit 170 outputs the terminal abnormality flag FL-T to the operation amount determination unit 150.

[0124] The vehicle abnormality determination unit 180 performs "vehicle abnormality determination processing". In more detail, the vehicle abnormality determination unit 180 receives information on the vehicle driving control amount calculated by the control unit 130. Moreover, the vehicle abnormality determination unit 180 determines whether an abnormality has occurred in the vehicle driving control based on the vehicle driving control amount. A specific example of the vehicle abnormality determination processing will be described later (refer to Chapter 4-5). The vehicle abnormality flag FL-V is information indicating the result of the vehicle abnormality determination processing. The vehicle abnormality determination unit 180 outputs the vehicle abnormality flag FL-V to the operation amount determination unit 150.

[0125] The manipulated variable determination unit 150 executes manipulated variable determination processing based on the various abnormality flags FL-C, FL-T, and FL-V. Specifically, the manipulated variable determination unit 150 executes manipulated variable determination processing based on the results of the various abnormality determination processes. A specific example of the manipulated variable determination processing will be described later (see Sections 4-6).

[0126] The control unit 130 controls the vehicle 100 according to the final operation amount X output from the operation amount determination unit 150 .

[0127] 4-3. Example of communication abnormality determination processing

[0128] Figure 10 1 is a flowchart showing an example of the communication abnormality determination process performed by the communication abnormality determination unit 160 .

[0129] In step S110, the communication abnormality determination unit 160 determines whether the receiving unit 110 has received data. If the receiving unit 110 has received data (step S110: Yes), the process proceeds to step S120. Otherwise (step S110: No), the process proceeds to step S130.

[0130] In step S120, the communication anomaly determination unit 160 determines whether the reception status is good. The reception status is indicated by parameters such as communication speed and radio wave reception strength. If the parameter is above a predetermined threshold (step S120: "Yes"), the reception status is determined to be good, and the process proceeds to step S160. Otherwise (step S120: "No"), the process proceeds to step S130.

[0131] In step S130, the communication anomaly determination unit 160 determines whether the state of no data being received or the poor reception state has continued for Ta seconds. If such a poor state has continued for Ta seconds (step S130: "Yes"), the process proceeds to step S140. On the other hand, if such a poor state has not continued for Ta seconds (step S130: "No"), the process proceeds to step S150.

[0132] In step S140 , the communication abnormality determination unit 160 determines (judgments) that a communication abnormality has occurred.

[0133] In step S150 , the communication abnormality determination unit 160 does not finalize the determination and sets the current state to “communication abnormality determination in progress.” Thereafter, the process returns to step S110 .

[0134] Figure 11 : is a flowchart showing an example of step S160. In step S160, the communication delay amount DL is considered.

[0135] In step S161 , the communication abnormality determination unit 160 obtains information on the communication delay amount DL from the reception unit 110 .

[0136] In step S162, the communication anomaly determination unit 160 determines whether the delay amount DL exceeds the first threshold DL_th1. The first threshold DL_th1 is the delay amount DL that can be considered to indicate a communication anomaly. For example, the first threshold DL_th1 is a delay amount DL that does not normally occur. If the delay amount DL exceeds the first threshold DL_th1 (step S162: "Yes"), the process proceeds to step S163. On the other hand, if the delay amount DL is less than the first threshold DL_th1 (step S162: "No"), the process proceeds to step S164.

[0137] In step S163 , the communication abnormality determination unit 160 determines (judgments) that a communication abnormality has occurred.

[0138] In step S164, the communication abnormality determination section 160 determines whether the delay amount DL exceeds a second threshold value DL_th2. The second threshold value DL_th2 is smaller than the above-mentioned first threshold value DL_thl. For example, the second threshold value DL_th2 is an upper limit value of the allowable range of the delay amount DL. In a case where the delay amount DL exceeds the second threshold value DL_th2 (step S164: YES), the process proceeds to step S165. On the other hand, in a case where the delay amount DL is the second threshold value DL_th2 or less (step S164: NO), the process proceeds to step S167.

[0139] In step S165, the communication abnormality determination section 160 determines whether the state in which the delay amount DL exceeds the second threshold value DL_th2 has continued for Tb seconds. In a case where such a state has continued for Tb seconds (step S165: YES), the process proceeds to step S163. On the other hand, in a case where such a state has not continued for Tb seconds (step S165: NO), the process proceeds to step S166.

[0140] In step S166, the communication abnormality determination section 160 does not determine the judgment, but sets the current state to "communication abnormality determination in progress". Thereafter, the process returns to step S110.

[0141] In step S167, the communication abnormality determination section 160 determines that no communication abnormality has occurred, and that the communication is normal. Thereafter, the process returns to step S110.

[0142] 4-4. Example of terminal abnormality determination process

[0143] Figure 12 Fig. 4 is a flowchart showing an example of a terminal abnormality determination process implemented by the terminal abnormality determination section 170.

[0144] In step S171, the terminal abnormality determination section 170 receives the remote operation information OPE. Also, the terminal abnormality determination section 170 acquires the operation amount change DX, which is the change amount of the first operation amount Xl from the previous value, performed by the remote operator O.

[0145] In step S172, the terminal abnormality determination section 170 determines whether the operation amount change DX exceeds a first threshold value DX_thl. The first threshold value DX_thl is an operation amount change DX at which it can be determined that a terminal abnormality has occurred. For example, the first threshold value DX_thl is an operation amount change DX that does not normally occur. In a case where the operation amount change DX exceeds the first threshold value DX_thl (step S172: YES), the process proceeds to step S173. On the other hand, in a case where the operation amount change DX is the first threshold value DX_thl or less (step S172: NO), the process proceeds to step S174.

[0146] In step S173, the terminal abnormality determination section 170 determines (concludes) that a terminal abnormality has occurred.

[0147] In step S174, the terminal abnormality determination section 170 determines whether the operation amount change DX exceeds a second threshold value DX th2. The second threshold value DX th2 is smaller than the above-mentioned first threshold value DX thl. For example, the second threshold value DX th2 is an upper limit value of an allowable range of the operation amount change DX. In a case where the operation amount change DX exceeds the second threshold value DX th2 (step S174: YES), the processing proceeds to step S175. On the other hand, in a case where the operation amount change DX is equal to or smaller than the second threshold value DX th2 (step S174: NO), the processing proceeds to step S177.

[0148] In step S175, the terminal abnormality determination section 170 determines whether the state in which the operation amount change DX exceeds the second threshold value DX th2 has continued for Tc seconds. In a case where such a state has continued for Tc seconds (step S175: YES), the processing proceeds to step S173. On the other hand, in a case where such a state has not continued for Tc seconds (step S175: NO), the processing proceeds to step S176.

[0149] In step S176, the terminal abnormality determination section 170 does not determine the judgment, but sets the current state as "terminal abnormality determination in progress". Thereafter, the processing returns to step S171.

[0150] In step S177, the terminal abnormality determination section 170 determines that no terminal abnormality has occurred, and that the remote operator terminal 200 is normal. Thereafter, the processing returns to step S171.

[0151] 4-5. Example of vehicle abnormality determination processing

[0152] Figure 13 Fig. 18 is a flowchart showing an example of vehicle abnormality determination processing performed by the vehicle abnormality determination section 180.

[0153] In step S181, the vehicle abnormality determination section 180 receives information of the vehicle travel control amount calculated by the control section 130. Also, the vehicle abnormality determination section 180 acquires the change amount of the vehicle travel control amount from the previous value, that is, "control amount change DC".

[0154] In step S182, the vehicle abnormality determination unit 180 determines whether the control amount change DC exceeds the first threshold DC_th1. The first threshold DC_th1 is a control amount change DC that can be used to determine that a vehicle abnormality has occurred. For example, the first threshold DC_th1 is a control amount change DC that does not normally occur. If the control amount change DC exceeds the first threshold DC_th1 (step S182: "Yes"), the process proceeds to step S183. On the other hand, if the control amount change DC is less than the first threshold DC_th1 (step S182: "No"), the process proceeds to step S184.

[0155] In step S183 , the vehicle abnormality determination unit 180 determines (assumes) that a vehicle abnormality has occurred.

[0156] In step S184, the vehicle abnormality determination unit 180 determines whether the control amount change DC exceeds the second threshold value DC_th2. The second threshold value DC_th2 is smaller than the first threshold value DC_th1. For example, the second threshold value DC_th2 is the upper limit of the permissible range of the control amount change DC. If the control amount change DC exceeds the second threshold value DC_th2 (step S184: "Yes"), the process proceeds to step S185. On the other hand, if the control amount change DC is less than the second threshold value DC_th2 (step S184: "No"), the process proceeds to step S187.

[0157] In step S185, the vehicle abnormality determination unit 180 determines whether the state in which the control amount change DC exceeds the second threshold value DC_th2 has continued for Td seconds. If this state has continued for Td seconds (step S185: "Yes"), the process proceeds to step S183. On the other hand, if this state has not continued for Td seconds (step S185: "No"), the process proceeds to step S186.

[0158] In step S186 , the vehicle abnormality determination unit 180 does not finalize the determination and sets the current state to “vehicle abnormality determination in progress.” Thereafter, the process returns to step S181 .

[0159] In step S187, the vehicle abnormality determination unit 180 determines that no vehicle abnormality has occurred and that the vehicle 100 is normal. Thereafter, the process returns to step S181.

[0160] 4-6. Example of operation amount judgment processing

[0161] Figure 14 1 is a diagram for explaining an example of an operation amount determination process performed by the operation amount determination unit 150. Figure 14In the example above, the circle mark means "normal (no abnormality)", the cross mark means "abnormality confirmed (abnormality occurred)", and the triangle mark means "under determination". Figures 10 to 13 In the example, the period under judgment is a temporary abnormal condition (e.g. Figure 12 In other words, the period of determination is the period from the detection of the possibility of an abnormality until the determination of whether an abnormality has occurred.

[0162] The first operation amount X1 is the operation amount performed by the remote operator O. The second operation amount X2 is the operation amount determined by the automatic driving function of the vehicle 100 under normal circumstances. The third operation amount X3 is a combination of the first operation amount X1 and the second operation amount X2, and is expressed by "X3 = a·X1+(1-a)X2". Here, the coefficient is a real number greater than 0 and smaller than 1. The fourth operation amount X4 is a type of the second operation amount X2 determined by the automatic driving function of the vehicle 100, and is an operation amount used in particular in an emergency. For example, the fourth operation amount X4 is an operation amount used to safely stop or retreat the vehicle 100.

[0163] The operation amount determination unit 150 determines any one of the first operation amount X1 , the second operation amount X2 , the third operation amount X3 , and the fourth operation amount X4 as the final operation amount X based on the results of various abnormality determination processes.

[0164] First, for simplicity, consider the combination of communication conditions and terminal conditions. If neither a communication anomaly nor a terminal anomaly occurs, the first operation variable X1 is selected (see Nos. 1 and 2). If a terminal anomaly is determined to have occurred, the second operation variable X2 is selected instead of the first operation variable X1 (see Nos. 7, 8, 16, 17, 25, and 26). This improves safety. If a communication anomaly is determined to have occurred, the second operation variable X2 is selected instead of the first operation variable X1 (see Nos. 19, 20, 22, 23, 25, and 26). This improves safety.

[0165] The "first determination period" is the period during which a terminal abnormality is being determined. During this first determination period, if it is determined that no communication abnormality has occurred, the third operation variable X3 (see Nos. 4 and 5) is selected. The second operation variable X2 partially contributes to the operation variable X, so safety is improved when a terminal abnormality actually occurs. In addition, the system does not completely switch to the second operation variable X2, but still retains the influence of the first operation variable X1, thereby suppressing the sense of disharmony of the remote operator O when no terminal abnormality actually occurs.

[0166] The "2nd determination period" is a determination period of a communication abnormality. In the 2nd determination period, the 1st operation amount X1 is selected in a case where it is determined that no terminal abnormality has occurred (refer to No. 10, 11). By continuing to use the 1st operation amount X1 also in the determination of a communication abnormality, the continuity of remote operation performed by the remote operator O is improved.

[0167] In a period in which the 1st determination period and the 2nd determination period are repeated, the 3rd operation amount X3 is selected (refer to No. 13, 14). The 2nd operation amount X2 partially contributes to the operation amount X, so the safety in a case where a terminal abnormality actually occurs is improved. In addition, instead of completely switching to the 2nd operation amount X2, the influence of the 1st operation amount X1 is left, so the sense of discomfort of the remote operator O in a case where no terminal abnormality actually occurs is suppressed.

[0168] Next, combinations of communication conditions, terminal conditions, and vehicle conditions are considered. In a case where no vehicle abnormality has occurred and in the determination of a vehicle abnormality, the same as the above is applied. On the other hand, in a case where it is determined that a vehicle abnormality has occurred, basically, the 4th operation amount X4 that is the 2nd operation amount X2 at the time of emergency is selected (refer to No. 3, 6, 12, 15, 18, 21, 24, 27). Thereby, it is possible to safely stop the vehicle 100. However, even if a vehicle abnormality has occurred, in a case where the camera mounted on the vehicle 100 is normal, the 1st operation amount X1 or the 3rd operation amount X3 can be selected (refer to No. 3, 6, 12, 15).

[0169] 4-7. Effects

[0170] As explained above, according to the present embodiment, the control device 105 of the vehicle 100 is configured to execute the abnormality determination process and the operation amount arbitration process. In the operation amount arbitration process, the control device 105 decides the "final operation amount X" from the 1st operation amount X1, the 2nd operation amount X2, the 3rd operation amount X3, and the 4th operation amount X4 according to the result of the abnormality determination process. Moreover, the vehicle 100 is controlled in accordance with the decided operation amount X. In this way, in a case where an abnormality has occurred in the remote operation system 1, it is possible to appropriately control the vehicle 100 in consideration of the conditions of the abnormality.

[0171] 5. Configuration Example of Combination

[0172] It is also possible to combine the delay compensation process explained in the above Section 3 and the abnormality determination process and the operation amount arbitration process explained in the above Section 4. Figure 15 A functional configuration example of a case where this combination is applied is shown. The vehicle 100 includes a reception section 110, a delay compensation section 120, a control section 130, a communication abnormality determination section 160, a terminal abnormality determination section 170, and a vehicle abnormality determination section 180.

[0173] The delay compensator 120 calculates a first corrected manipulated variable X1C based on the first manipulated variable X1 and the delay amount DL. The manipulated variable determination unit 150 receives the first corrected manipulated variable X1C instead of the first manipulated variable X1. Furthermore, the manipulated variable determination unit 150 performs manipulated variable determination processing based on the first corrected manipulated variable X1C and the second manipulated variable X2 to determine the final manipulated variable X.

[0174] This achieves both the effects of Section 3 and the effects of Section 4. That is, the safety of the remote operation of the vehicle 100 is further improved.

[0175] 6. Example of a remote operator terminal

[0176] Figure 16 2 is a block diagram showing a configuration example of the remote operator terminal 200 . The remote operator terminal 200 includes a communication device 210 , a display device 220 , an input device 230 , and a control device 250 .

[0177] The communication device 210 communicates with the vehicle 100 and the management device 300 .

[0178] The display device 220 presents various information to the remote operator O by displaying various information.

[0179] The input device 230 receives input from the remote operator O. For example, the input device 230 includes remote operating components operated when the remote operator O remotely operates the vehicle 100. The remote operating components include a steering wheel, an accelerator pedal, a brake pedal, and direction indicators.

[0180] The control device 250 controls the remote operator terminal 200. The control device 250 includes one or more processors 260 (hereinafter referred to as processors 260) and one or more storage devices 270 (hereinafter referred to as storage devices 270). The processors 260 execute various processes. For example, the processors 260 include a CPU. The storage devices 270 store various information required for the processes executed by the processors 260. Examples of the storage devices 270 include volatile memory, nonvolatile memory, HDD, SSD, and the like.

[0181] Remote operation program PROG2 is a computer program executed by processor 260. Processor 260 executes remote operation program PROG2 to implement the functions of control device 250. Remote operation program PROG2 is stored in storage device 270. Alternatively, remote operation program PROG2 may be recorded on a computer-readable recording medium. Remote operation program PROG2 may also be provided via a network.

[0182] The control device 250 communicates with the vehicle 100 via the communication device 210. The control device 250 receives the vehicle information VCL transmitted from the vehicle 100. The control device 250 displays the vehicle information VCL, including image information, on the display device 220, thereby presenting the vehicle information VCL to the remote operator O. The remote operator O can recognize the status of the vehicle 100 and the surrounding conditions based on the vehicle information VCL displayed on the display device 220.

[0183] Remote operator O operates a remote operating component of input device 230. A sensor provided on the remote operating component detects the amount of operation performed on the remote operating component. Control device 250 generates remote operation information OPE reflecting the amount of operation performed on the remote operating component by remote operator O. Control device 250 then transmits this remote operation information OPE to vehicle 100 via communication device 210.

Claims

1. A control device for controlling a mobile object, the mobile object being a target of remote operation by a remote operator, wherein: The control device includes one or more processors. The one or more processors are configured to perform: During the remote operation of the mobile object, communicating with a remote operator terminal on the remote operator side to receive remote operation information including a first operation amount performed by the remote operator; a process of acquiring an amount of delay in communication from the remote operator terminal to the mobile object based on a result of receiving the remote operation information; a delay compensation process for calculating a first correction operation amount by compensating for a delay of the first operation amount based on the delay amount; an upsampling process of increasing a sampling frequency of the first correction operation amount by estimating the first correction operation amount during a period without sampling; and A process of controlling the movable body based on the first correction operation amount.

2. The control device according to claim 1, wherein: The control frequency of the mobile body is higher than the communication frequency of the remote operation information. The one or more processors execute the up-sampling process so that the difference between the sampling frequency of the first correction operation amount and the control frequency becomes smaller than the difference between the communication frequency and the control frequency.

3. The control device according to claim 1 or 2, wherein: The one or more processors The first correction operation amount in a non-sampling period is estimated by feeding back the estimated value of the first correction operation amount obtained by the upsampling process and the first correction operation amount obtained by the delay compensation process. The first corrected operation amount is calculated by correcting the estimated value of the first corrected operation amount based on the first operation amount received from the remote operator terminal while performing the delay compensation process based on the delay amount.

4. The control device according to claim 3, wherein: The one or more processors execute the upsampling process and the delay compensation process by applying a delayed Kalman filter to the first operation variable.

5. The control device according to claim 1 or 2, wherein: In the upsampling process, the one or more processors further estimate the first correction operation amount based on supplementary information including at least one of a shape of a road on which the mobile body moves and a control amount of the mobile body.

6. The control device according to claim 1 or 2, wherein: The mobile object has an automatic driving function, The one or more processors are further configured to execute: a communication abnormality determination process for determining whether an abnormality has occurred in the communication with the remote operator terminal based on the state of the communication with the remote operator terminal; Terminal abnormality determination processing, determining whether an abnormality occurs in the remote operator terminal based on the first operation amount; an operation amount determination process for determining an operation amount based on at least one of the first correction operation amount and a second operation amount determined by the automatic driving function, based on results of the communication abnormality determination process and the terminal abnormality determination process; and The processing of the moving object is controlled according to the determined operation amount.

7. A control method for controlling a mobile object, the mobile object being a subject of a remote operation performed by a remote operator, wherein: The control method includes: During the remote operation of the mobile object, communicating with a remote operator terminal on the remote operator side to receive remote operation information including a first operation amount performed by the remote operator; a process of acquiring an amount of delay in communication from the remote operator terminal to the mobile object based on a result of receiving the remote operation information; a delay compensation process for calculating a first correction operation amount by compensating for a delay of the first operation amount based on the delay amount; an upsampling process of increasing a sampling frequency of the first correction operation amount by estimating the first correction operation amount during a period without sampling; and A process of controlling the movable body based on the first correction operation amount.

8. A computer-readable recording medium having a control program recorded thereon, wherein: The control program is a computer program for controlling a mobile object that is the subject of remote operation by a remote operator. The control program causes the computer to execute: During the remote operation of the mobile object, communicating with a remote operator terminal on the remote operator side to receive remote operation information including a first operation amount performed by the remote operator; a process of acquiring an amount of delay in communication from the remote operator terminal to the mobile object based on a result of receiving the remote operation information; a delay compensation process for calculating a first correction operation amount by compensating for a delay of the first operation amount based on the delay amount; an upsampling process of increasing a sampling frequency of the first correction operation amount by estimating the first correction operation amount during a period without sampling; and A process of controlling the movable body based on the first correction operation amount.

Images