Remote operating system and abnormality notification method

Abstract

The present disclosure provides a remote operation system and an abnormality notification method. When an abnormality occurs in the remote operation system, the remote operator is effectively notified of the abnormality. The remote operation system includes: a mobile body as an object to be remotely operated by the remote operator; and a remote operator terminal on the remote operator side. The abnormality notification method in the remote operation system includes an abnormality determination process of determining whether at least one of an abnormality in the communication between the mobile body and the remote operator terminal, an abnormality in the mobile body, and an abnormality in the remote operator terminal has occurred. The abnormality notification method also includes a gain setting process of setting a larger gain when an abnormality occurs than when no abnormality occurs. The abnormality notification method also includes a process of calculating a reaction force control amount by multiplying a reference reaction force control amount by a gain, and imparting an operating reaction force corresponding to the reaction force control amount to the remote operation component operated by the remote operator.

Description

Technical Field

[0001] The present disclosure relates to remote operation of a mobile object by a remote operator. Background Art

[0002] Patent Document 1 discloses a driving control system for remotely controlling a vehicle. A remote control device communicates wirelessly with a vehicle-side driving control device. The driving control device calculates the delay time of wireless communication with the remote control device. If the delay time exceeds a threshold, the driving control device changes the vehicle's driving control from normal control to safety control. Safety control involves, for example, reducing the vehicle speed compared to normal control.

[0003] Furthermore, Patent Document 2, Patent Document 3, and Patent Document 4 disclose technologies related to remote operation of a vehicle.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2016-071585

[0007] Patent Document 2: Japanese Patent Application Laid-Open No. 2004-295360

[0008] Patent Document 3: Japanese Patent Application Laid-Open No. 2013-020426

[0009] Patent Document 4: Japanese Patent Application Publication No. 2021-068132 Summary of the Invention

[0010] Consider remote operation of a mobile object (e.g., a vehicle or robot) by a remote operator. This remote operation system includes the mobile object being remotely operated and a remote operator terminal on the remote operator's side. During remote operation, the mobile object and the remote operator terminal communicate with each other. If an anomaly occurs in such a remote operation system, it is desirable to effectively notify the remote operator of the anomaly.

[0011] One object of the present disclosure is to provide a technology that can effectively notify a remote operator of the abnormality when an abnormality occurs in a remote operation system.

[0012] The first viewpoint relates to a remote operating system.

[0013] Remote operating system with:

[0014] A mobile object that is the subject of remote operation by a remote operator;

[0015] a remote operator terminal on the remote operator side; and

[0016] One or more processors included in at least one of the mobile object and the remote operator terminal.

[0017] One or more processors configured to perform:

[0018] an abnormality determination process for determining whether at least one of an abnormality in communication between the mobile object and the remote operator terminal, an abnormality in the mobile object, and an abnormality in the remote operator terminal has occurred;

[0019] a gain setting process of setting a larger gain when an abnormality occurs than when no abnormality occurs; and

[0020] The process of calculating the reaction force control amount by multiplying the reference reaction force control amount by a gain.

[0021] The remote operator terminal is configured to apply an operation reaction force corresponding to the reaction force control amount to the remote operation component operated by the remote operator.

[0022] The second aspect relates to an abnormality notification method in a remote operation system.

[0023] The remote operating system has:

[0024] A mobile object that is the subject of remote operation by a remote operator; and

[0025] Remote operator terminal on the remote operator side.

[0026] Exception notification methods include:

[0027] an abnormality determination process for determining whether at least one of an abnormality in communication between the mobile object and the remote operator terminal, an abnormality in the mobile object, and an abnormality in the remote operator terminal has occurred;

[0028] Gain setting processing, in the case of an abnormality, setting a larger gain than in the case of no abnormality;

[0029] a process of calculating a reaction force control amount by multiplying a reference reaction force control amount by a gain; and

[0030] A process of applying an operation reaction force corresponding to the reaction force control amount to a remote operation component operated by a remote operator.

[0031] According to the present disclosure, it is possible to determine whether an abnormality has occurred in a remote operating system. When an abnormality occurs, the reaction force felt by the remote operator is greater than normal. This allows the remote operator to easily identify the abnormality. In other words, the occurrence of an abnormality in the remote operating system can be effectively notified to the remote operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a schematic diagram showing a configuration example of a remote operation system according to an embodiment of the present disclosure.

[0033] Figure 2 This is a conceptual diagram for explaining the outline of an abnormality determination process and an abnormality notification process according to an embodiment of the present disclosure.

[0034] Figure 3 This is a flowchart showing an overview of an abnormality determination process and an abnormality notification process according to an embodiment of the present disclosure.

[0035] Figure 4 This is a block diagram illustrating a configuration example of a vehicle according to an embodiment of the present disclosure.

[0036] Figure 5 This is a block diagram for explaining abnormality determination processing and abnormality notification processing in a vehicle according to an embodiment of the present disclosure.

[0037] Figure 6 This is a flowchart illustrating an example of communication abnormality determination processing according to an embodiment of the present disclosure.

[0038] Figure 7 This is a flowchart illustrating an example of communication abnormality determination processing according to an embodiment of the present disclosure.

[0039] Figure 8 This is a flowchart illustrating an example of vehicle abnormality determination processing according to an embodiment of the present disclosure.

[0040] Figure 9 This is a block diagram illustrating an example of the functional configuration of a reaction force controlled amount calculation unit in the vehicle according to the present embodiment.

[0041] Figure 10 This is a block diagram illustrating an example of the functional configuration of a reference reaction force controlled amount calculation unit in the vehicle according to the present embodiment.

[0042] Figure 11 This is a block diagram showing a configuration example of a remote operator terminal according to an embodiment of the present disclosure.

[0043] Figure 12 This is a block diagram for explaining an abnormality determination process and an abnormality notification process in the remote operator terminal according to the embodiment of the present disclosure.

[0044] Figure 13 This is a flowchart illustrating an example of terminal abnormality determination processing according to an embodiment of the present disclosure.

[0045] Figure 14This is a block diagram showing a functional configuration example of a reaction force controlled variable calculation unit in the remote operator terminal according to the present embodiment.

[0046] (Explanation of Symbols)

[0047] 1: Remote operation system; 100: Vehicle; 150: Control device; 152: Communication anomaly determination unit; 154: Vehicle anomaly determination unit; 160: Processor; 170: Storage device; 180: Reference reaction force control quantity calculation unit; 190: Reaction force control quantity calculation unit; 200: Remote operator terminal; 230: Remote operation component; 240: Operation reaction force actuator; 250: Control device; 252: Communication anomaly determination unit; 254: Terminal anomaly determination unit; 260: Processor; 270: Storage device; 290: Reaction force control quantity calculation unit; 300: Management device; FB: Reaction force control quantity; FB0: Reference reaction force control quantity; FB1: First reaction force control quantity; FL1-C: Communication anomaly flag; FL1-V: Vehicle anomaly flag; FL2-C: Communication anomaly flag; FL2-T: Terminal anomaly flag; OPE: Remote operation information; VCL: Vehicle information. DETAILED DESCRIPTION

[0048] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0049] 1. Overview of Remote Operation System

[0050] Consider remote operation (remote driving) of mobile objects. Examples of mobile objects that can be remotely operated include vehicles, robots, and flying objects. Vehicles can be either autonomous or driverless. Examples of robots include logistics robots and work robots. Examples of flying objects include aircraft and drones.

[0051] As an example, in the following description, a case where the mobile object to be remotely operated is a vehicle is considered. In general, the term "vehicle" in the following description is replaced with "mobile object".

[0052] Figure 1: is a schematic diagram showing a structural example of the remote operation system 1 involved in this embodiment. The remote operation system 1 includes a vehicle 100, a remote operator terminal 200, and a management device 300. The vehicle 100 is the object of remote operation. The remote operator terminal 200 is a terminal device used when the remote operator O remotely operates the vehicle 100. The remote operator terminal 200 can also be called a remote operation HMI (Human Machine Interface). The management device 300 manages the remote operation system 1. The management of the remote operation system 1 includes, for example, assigning a remote operator O to a vehicle 100 that requires remote operation. The management device 300 can communicate with the vehicle 100 and the remote operator terminal 200 via a communication network. Typically, the management device 300 is a management server on the cloud. The management server can also be composed of multiple servers that perform distributed processing.

[0053] Vehicle 100 is equipped with various sensors, including a camera. The camera captures the surrounding conditions of vehicle 100, acquiring image information representing the surrounding conditions. Vehicle information VCL is information acquired by various sensors, including image information acquired by the camera. Vehicle 100 transmits vehicle information VCL to remote operator terminal 200 via management device 300. Specifically, vehicle 100 transmits vehicle information VCL to management device 300, and management device 300 transmits the received vehicle information VCL to remote operator terminal 200.

[0054] The remote operator terminal 200 receives the vehicle information VCL transmitted from the vehicle 100. The remote operator terminal 200 presents the vehicle information VCL to the remote operator O. Specifically, the remote operator terminal 200 includes a display device, and displays image information and the like on the display device. The remote operator O observes the displayed information, identifies the conditions around the vehicle 100, and performs remote operations on the vehicle 100. The remote operation information OPE is information related to the remote operation performed by the remote operator O. For example, the remote operation information OPE includes the amount of operation performed by the remote operator O. The remote operator terminal 200 transmits the remote operation information OPE to the vehicle 100 via the management device 300. That is, the remote operator terminal 200 transmits the remote operation information OPE to the management device 300, and the management device 300 transmits the received remote operation information OPE to the vehicle 100.

[0055] The vehicle 100 receives the remote operation information OPE transmitted from the remote operator terminal 200. The vehicle 100 performs vehicle driving control according to the received remote operation information OPE. In this way, remote operation of the vehicle 100 is achieved.

[0056] 2. Abnormal determination and notification processing

[0057] Next, consider "abnormalities" that may occur in the remote operation system 1. Abnormality is a concept that includes failures, malfunctions, inconsistencies, and malfunctions. Examples of abnormalities in the remote operation system 1 include "communication abnormalities" (abnormalities in the communication between the vehicle 100 and the remote operator terminal 200), "vehicle abnormalities" (abnormalities in the vehicle 100), and "terminal abnormalities" (abnormalities in the remote operator terminal 200).

[0058] The remote operation system 1 performs an "abnormality determination process" to determine whether at least one of a communication abnormality, a vehicle abnormality, and a terminal abnormality has occurred within the remote operation system 1. If an abnormality occurs during remote operation of the vehicle 100, a quick response is desirable to ensure safety. To this end, the remote operation system 1 performs an "abnormality notification process" to notify the remote operator O of the abnormality.

[0059] Figure 2 This is a conceptual diagram for explaining the outline of the abnormality determination process and the abnormality notification process according to the present embodiment. At least one of the vehicle 100 and the remote operator terminal 200 performs the abnormality determination process.

[0060] For example, the vehicle 100 may also include a "communication anomaly determination function" that determines whether a communication anomaly has occurred. Specifically, the vehicle 100 communicates with the remote operator terminal 200 and receives information such as remote operation information OPE from the remote operator terminal 200. This received information allows the vehicle 100 to understand the status of communication with the remote operator terminal 200. Based on this communication status, the vehicle 100 can determine whether a communication anomaly has occurred from the remote operator terminal 200 to the vehicle 100 itself.

[0061] As another example, the vehicle 100 may include a “vehicle abnormality determination function” for determining whether a vehicle abnormality has occurred.

[0062] As another example, the remote operator terminal 200 may also include a "communication anomaly determination function" for determining whether a communication anomaly has occurred. Specifically, the remote operator terminal 200 communicates with the vehicle 100 and receives information such as vehicle information VCL from the vehicle 100. Based on this received information, the remote operator terminal 200 can determine whether a communication anomaly has occurred between the vehicle 100 and the remote operator terminal 200.

[0063] As still another example, the remote operator terminal 200 may include a “terminal abnormality determination function” for determining whether a terminal abnormality has occurred.

[0064] The abnormality notification process for notifying the remote operator O of the occurrence of an abnormality is performed via the "operation reaction force". Specifically, the remote operator terminal 200 includes a remote operation component 230 and an operation reaction force actuator 240. The remote operation component 230 is a component operated when the remote operator O remotely operates the vehicle 100. The remote operation component 230 includes a steering wheel (steering wheel), an accelerator pedal, a brake pedal, a direction indicator, etc. The operation reaction force actuator 240 applies an operation reaction force to the remote operation component 230. For example, the operation reaction force is a steering reaction force applied to the steering wheel. The remote operator O who operates the remote operation component 230 feels the operation reaction force applied to the remote operation component 230.

[0065] The remote operator terminal 200 controls (adjusts) the magnitude of the operational reaction force applied to the remote operating component 230 based on the results of the abnormality determination process. Specifically, the operational reaction force applied during an abnormality is controlled (adjusted) to be greater than the operational reaction force during normal operation, when compared under identical vehicle conditions, such as the steering angle and speed of the vehicle 100. In other words, when an abnormality occurs, the operational reaction force felt by the remote operator O is greater than during normal operation. This allows the remote operator O to easily identify the occurrence of an abnormality. In other words, the occurrence of an abnormality in the remote operation system 1 can be effectively notified to the remote operator O.

[0066] The abnormality determination process also considers situations where, while an abnormality has not yet been confirmed, the possibility (signs) of an abnormality being detected. The "determination" period is the period from the detection of the possibility of an abnormality until the determination of whether an abnormality has occurred. During this determination period, the magnitude of the operational reaction force can also be controlled (adjusted) to be between the abnormality and normal conditions. This allows for more detailed notification to the remote operator O.

[0067] Figure 3 This is a flowchart showing an overview of the abnormality determination process and the abnormality notification process according to the present embodiment.

[0068] In step S10 , the remote operation system 1 performs “abnormality determination processing.” Specifically, the remote operation system 1 determines whether at least one of a communication abnormality, a vehicle abnormality, and a terminal abnormality has occurred in the remote operation system 1 .

[0069] In step S20, the remote operation system 1 performs a "gain setting process" to set the reaction force gain Gr based on the results of the abnormality determination process. More specifically, if it is determined that no abnormality has occurred, the remote operation system 1 sets the reaction force gain Gr to the first gain. On the other hand, if it is determined that an abnormality has occurred, the remote operation system 1 sets the reaction force gain Gr to a second gain that is greater than the first gain. Alternatively, after the possibility of an abnormality is detected and until it is determined whether an abnormality has occurred, the remote operation system 1 may set the reaction force gain Gr to a third gain between the first and second gains.

[0070] In step S30, the remote operation system 1 performs a "reaction force control amount calculation process" to calculate the reaction force control amount FB used to generate the operational reaction force. Specifically, the remote operation system 1 calculates the reaction force control amount FB by multiplying the baseline reaction force control amount FB0 by the reaction force gain Gr. The baseline reaction force control amount FB0 is calculated based on vehicle conditions such as the steering angle and vehicle speed of the vehicle 100. When compared under the same baseline reaction force control amount FB0, the reaction force control amount FB during an abnormality is greater than the reaction force control amount FB during a normal state.

[0071] In step S40 , the remote operator terminal 200 controls the operational reaction force actuator 240 according to the reaction force control amount FB, and applies the operational reaction force corresponding to the reaction force control amount FB to the remote operating component 230 .

[0072] <Effect>

[0073] As described above, according to this embodiment, whether an abnormality has occurred in the remote operation system 1 is determined. When an abnormality occurs, the operational reaction force felt by the remote operator O is greater than during normal operation. This allows the remote operator O to easily identify the abnormality. In other words, the occurrence of an abnormality in the remote operation system 1 can be effectively notified to the remote operator O.

[0074] The remote operator O who recognizes the occurrence of an abnormality can take measures to ensure safety. For example, the remote operator O can retreat the vehicle 100 to a safe place and stop it.

[0075] During the period until the determination of whether an abnormality has occurred is finalized, the magnitude of the operational reaction force may be controlled to be between the abnormality and normal times. This allows for more detailed notification to the remote operator O.

[0076] The operational reaction force can also be a steering reaction force applied to the steering wheel. In this case, when an abnormality occurs, the steering reaction force increases. This increased steering reaction force suppresses excessive steering. This, for example, suppresses swerving of the vehicle 100 caused by remote operation. In other words, the stability of the vehicle 100 during an abnormality is improved, allowing safer remote operation of the vehicle 100 during an abnormality.

[0077] Hereinafter, the remote operation system 1 according to the present embodiment will be described in more detail.

[0078] 3. Vehicle Example

[0079] 3-1. Structural example

[0080] Figure 4 1 is a block diagram showing a configuration example of a vehicle 100 . The vehicle 100 includes a communication device 110 , a sensor group 120 , a travel device 130 , and a control device 150 .

[0081] The communication device 110 communicates with the outside of the vehicle 100 . For example, the communication device 110 communicates with the remote operator terminal 200 and the management device 300 .

[0082] The sensor group 120 includes an identification sensor, a vehicle state sensor, a position sensor, and the like. The identification sensor identifies (detects) the surrounding conditions of the vehicle 100. Examples of the identification sensor include a camera, LIDAR (Laser Imaging Detection and Ranging), and a radar. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, and the like. The position sensor detects the position and orientation of the vehicle 100. For example, the position sensor includes a GNSS (Global Navigation Satellite System).

[0083] The travel device 130 includes a steering device, a drive device, and a brake device. The steering device steers the wheels. For example, the steering device includes an Electric Power Steering (EPS) device. The drive device is a power source that generates driving force. Examples of drive devices include an engine, an electric motor, and an in-wheel motor. The brake device generates braking force.

[0084] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter referred to as the processor 160) and one or more storage devices 170 (hereinafter referred to as the storage device 170). The processor 160 performs various processes. For example, the processor 160 includes a CPU (Central Processing Unit). The storage device 170 stores various information required for the processes implemented by the processor 160. Examples of the storage device 170 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), and the like. The control device 150 may also include one or more ECUs (Electronic Control Units).

[0085] The vehicle control program PROG1 is a computer program executed by the processor 160. The processor 160 executes the vehicle control program PROG1 to realize the functions of the control device 150. The vehicle control program PROG1 is stored in the storage device 170. Alternatively, the vehicle control program PROG1 may be recorded in a computer-readable recording medium.

[0086] 3-2. Driving environment information

[0087] The control device 150 uses the sensor group 120 to obtain driving environment information ENV indicating the driving environment of the vehicle 100 . The driving environment information ENV is stored in the storage device 170 .

[0088] Driving environment information ENV includes surrounding condition information representing the results of recognition by the recognition sensor. For example, surrounding condition information includes image information captured by a camera. Surrounding condition information may also include object information regarding objects surrounding vehicle 100. Examples of objects surrounding vehicle 100 include pedestrians, other vehicles (such as preceding vehicles and parked vehicles), white lines, signals, signs, and roadside structures. Object information indicates the relative position and speed of the object relative to vehicle 100.

[0089] In addition, the driving environment information ENV includes vehicle state information indicating the vehicle state detected by the vehicle state sensor.

[0090] Furthermore, the driving environment information ENV includes vehicle position information indicating the position and orientation of the vehicle 100. The vehicle position information is obtained by a position sensor. High-precision vehicle position information can also be obtained through localization processing using map information and surrounding situation information (object information).

[0091] 3-3. Vehicle driving control

[0092] The control device 150 performs vehicle travel control to control the travel of the vehicle 100. Vehicle travel control includes steering control, drive control, and brake control. The control device 150 performs vehicle travel control by controlling the travel device 130 (steering device, drive device, and brake device).

[0093] The control device 150 can also perform autonomous driving control based on the driving environment information ENV. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the driving environment information ENV. Furthermore, the control device 150 generates a target trajectory required for the vehicle 100 to follow the driving plan based on the driving environment information ENV. The target trajectory includes a target position and a target speed. Furthermore, the control device 150 controls vehicle driving so that the vehicle 100 follows the target trajectory.

[0094] 3-4. Processing related to remote operation

[0095] Hereinafter, a case where the vehicle 100 is remotely controlled will be described. The control device 150 communicates with the remote operator terminal 200 via the communication device 110 .

[0096] The control device 150 transmits vehicle information VCL to the remote operator terminal 200. Vehicle information VCL is information required for remote operation by the remote operator O and includes at least a portion of the aforementioned driving environment information ENV. For example, vehicle information VCL includes surrounding condition information (particularly image information). Vehicle information VCL may also include vehicle status information and vehicle location information.

[0097] Furthermore, the control device 150 receives remote operation information OPE from the remote operator terminal 200. The remote operation information OPE is information regarding remote operations performed by the remote operator O. For example, the remote operation information OPE includes the amount of operation performed by the remote operator O. The control device 150 controls vehicle travel according to the received remote operation information OPE.

[0098] 3-4-1. Abnormal determination and notification processing

[0099] Figure 5 This is a block diagram for explaining the abnormality determination process and abnormality notification process in vehicle 100. Vehicle 100 includes a receiving unit 151, a communication abnormality determination unit 152, a control unit 153, a vehicle abnormality determination unit 154, and a transmitting unit 155 as functional blocks. These functional blocks are implemented by communication device 110 and control device 150.

[0100] During remote operation of the vehicle 100, the receiving unit 151 receives information transmitted from the remote operator terminal 200. The information transmitted from the remote operator terminal 200 includes the aforementioned remote operation information OPE. Based on the received information, the receiving unit 151 determines the communication status with the remote operator terminal 200. Examples of the communication status include the presence or absence of received data, delay, communication speed, and radio wave reception strength.

[0101] The communication anomaly determination unit 152 performs a "communication anomaly determination process." More specifically, the communication anomaly determination unit 152 obtains information regarding the communication status from the receiving unit 151. Furthermore, based on the communication status, the communication anomaly determination unit 152 determines whether an anomaly has occurred in the communication from the remote operator terminal 200 to the vehicle 100. A specific example of the communication anomaly determination process will be described later (see section 3-4-2).

[0102] The communication anomaly flag FL1-C is information indicating the result of the communication anomaly determination process. If it is determined that no communication anomaly has occurred, i.e., if no communication anomaly has been detected, the communication anomaly flag FL1-C is set to, for example, "0." On the other hand, if it is determined that a communication anomaly has occurred, i.e., if a communication anomaly has been detected, the communication anomaly flag FL1-C is set to, for example, "1." The communication anomaly determination unit 152 outputs the communication anomaly flag FL1-C.

[0103] The control unit 153 receives remote operation information OPE and abnormality flags (FL1-C, FL1-V). The control unit 153 controls vehicle driving according to the remote operation information OPE. Furthermore, the control unit 153 outputs vehicle information VCL. If the abnormality flag indicates an abnormality has been detected, the control unit 153 may also perform a predetermined abnormality response process (e.g., evacuation control).

[0104] The vehicle abnormality determination unit 154 performs a "vehicle abnormality determination process." More specifically, the vehicle abnormality determination unit 154 receives information about the vehicle's driving control variable calculated by the control unit 153. Furthermore, the vehicle abnormality determination unit 154 determines whether an abnormality has occurred in the vehicle's driving control based on the vehicle's driving control variable. A specific example of the vehicle abnormality determination process will be described later (see section 3-4-3).

[0105] The vehicle abnormality flag FL1-V is information indicating the result of the vehicle abnormality determination process. If it is determined that no vehicle abnormality has occurred, that is, if no vehicle abnormality has been detected, the vehicle abnormality flag FL1-V is set to, for example, "0." On the other hand, if it is determined that a vehicle abnormality has occurred, that is, if a vehicle abnormality has been detected, the vehicle abnormality flag FL1-V is set to, for example, "1." The vehicle abnormality determination unit 154 outputs the vehicle abnormality flag FL1-V.

[0106] The transmitting unit 155 transmits the vehicle information VCL output from the control unit 153 to the remote operator terminal 200 .

[0107] 3-4-2. Example of communication abnormality determination processing

[0108] Figure 6 1 is a flowchart showing an example of the communication abnormality determination process performed by the communication abnormality determination unit 152 .

[0109] In step S110, the communication anomaly determination unit 152 determines whether the receiving unit 151 has received data. If the receiving unit 151 has received data (step S110: Yes), the process proceeds to step S120. Otherwise (step S110: No), the process proceeds to step S130.

[0110] In step S120, the communication anomaly determination unit 152 determines whether the reception status is good. The reception status is indicated by parameters such as the communication speed and the radio wave reception strength. If the parameter is above a predetermined threshold (step S120: "Yes"), the reception status is determined to be good, and the process proceeds to step S160. Otherwise (step S120: "No"), the process proceeds to step S130.

[0111] In step S130, the communication anomaly determination unit 152 determines whether the state of no data being received or the poor reception state has continued for Ta seconds. If such a poor state has continued for Ta seconds (step S130: "Yes"), the process proceeds to step S140. On the other hand, if such a poor state has not continued for Ta seconds (step S130: "No"), the process proceeds to step S150.

[0112] In step S140 , the communication abnormality determination unit 152 determines (judgments) that a communication abnormality has occurred.

[0113] In step S150 , the communication abnormality determination unit 152 does not finalize the determination and sets the current state to “communication abnormality determination in progress.” Thereafter, the process returns to step S110 .

[0114] Figure 7 : is a flowchart showing an example of step S160. In step S160, the communication delay amount DL is considered.

[0115] In step S161 , the communication abnormality determination unit 152 obtains information on the communication delay amount DL from the reception unit 151 .

[0116] In step S162, the communication anomaly determination unit 152 determines whether the delay amount DL exceeds the first threshold DL_th1. The first threshold DL_th1 is the delay amount DL that can be considered to indicate a communication anomaly. For example, the first threshold DL_th1 is the delay amount DL that would not normally occur. If the delay amount DL exceeds the first threshold DL_th1 (step S162: "Yes"), the process proceeds to step S163. On the other hand, if the delay amount DL is below the first threshold DL_th1 (step S162: "No"), the process proceeds to step S164.

[0117] In step S163 , the communication abnormality determination unit 152 determines (judgments) that a communication abnormality has occurred.

[0118] In step S164, the communication anomaly determination unit 152 determines whether the delay amount DL exceeds the second threshold DL_th2. The second threshold DL_th2 is smaller than the first threshold DL_th1. For example, the second threshold DL_th2 is the upper limit of the permissible range of the delay amount DL. If the delay amount DL exceeds the second threshold DL_th2 (step S164: "Yes"), the process proceeds to step S165. On the other hand, if the delay amount DL is less than the second threshold DL_th2 (step S164: "No"), the process proceeds to step S167.

[0119] In step S165, the communication anomaly determination unit 152 determines whether the delay amount DL has exceeded the second threshold value DL_th2 for Tb seconds. If this state has continued for Tb seconds (step S165: "Yes"), the process proceeds to step S163. On the other hand, if this state has not continued for Tb seconds (step S165: "No"), the process proceeds to step S166.

[0120] In step S166 , the communication abnormality determination unit 152 does not finalize the determination and sets the current state to “communication abnormality determination in progress.” Thereafter, the process returns to step S110 .

[0121] In step S167, the communication anomaly determination unit 152 determines that no communication anomaly has occurred and that communication is normal. Thereafter, the process returns to step S110.

[0122] 3-4-3. Example of vehicle abnormality determination processing

[0123] Figure 8 1 is a flowchart showing an example of vehicle abnormality determination processing performed by the vehicle abnormality determination unit 154 .

[0124] In step S181, the vehicle abnormality determination unit 154 receives information on the vehicle travel control amount calculated by the control unit 153. The vehicle abnormality determination unit 154 also obtains a "control amount change DC" which is a change in the vehicle travel control amount from the previous value.

[0125] In step S182, the vehicle abnormality determination unit 154 determines whether the control amount change DC exceeds the first threshold value DC_th1. The first threshold value DC_th1 is a control amount change DC that can also be determined to indicate a vehicle abnormality. For example, the first threshold value DC_th1 is a control amount change DC that would not normally occur. If the control amount change DC exceeds the first threshold value DC_th1 (step S182: "Yes"), the process proceeds to step S183. On the other hand, if the control amount change DC is less than the first threshold value DC_th1 (step S182: "No"), the process proceeds to step S184.

[0126] In step S183 , the vehicle abnormality determination unit 154 determines (assumes) that a vehicle abnormality has occurred.

[0127] In step S184, the vehicle abnormality determination unit 154 determines whether the control amount change DC exceeds the second threshold value DC_th2. The second threshold value DC_th2 is smaller than the first threshold value DC_th1. For example, the second threshold value DC_th2 is the upper limit of the permissible range of the control amount change DC. If the control amount change DC exceeds the second threshold value DC_th2 (step S184: "Yes"), the process proceeds to step S185. On the other hand, if the control amount change DC is less than the second threshold value DC_th2 (step S184: "No"), the process proceeds to step S187.

[0128] In step S185, the vehicle abnormality determination unit 154 determines whether the state in which the control amount change DC exceeds the second threshold value DC_th2 has continued for Tc seconds. If this state has continued for Tc seconds (step S185: "Yes"), the process proceeds to step S183. On the other hand, if this state has not continued for Tc seconds (step S185: "No"), the process proceeds to step S186.

[0129] In step S186 , the vehicle abnormality determination unit 154 does not finalize the determination and sets the current state to “vehicle abnormality determination in progress.” Thereafter, the process returns to step S181 .

[0130] In step S187, the vehicle abnormality determination unit 154 determines that no vehicle abnormality has occurred and that the vehicle 100 is normal. Thereafter, the process returns to step S181.

[0131] 3-4-4. Example of reaction force control amount calculation processing

[0132] As mentioned above Figure 5 As shown, the control unit 153 includes a reaction force control amount calculation unit 190. The reaction force control amount calculation unit 190 calculates a "first reaction force control amount FB1" for generating an operational reaction force in the remote operator terminal 200. The first reaction force control amount FB1 is fed back to the remote operator terminal 200 as part of the vehicle information VCL.

[0133] Figure 9 1 is a block diagram showing a functional configuration example of the reaction force controlled variable calculation unit 190. The reaction force controlled variable calculation unit 190 includes a gain setting unit 191, a gain setting unit 192, a multiplication unit 193, a filter unit 194, a multiplication unit 195, and a reference reaction force controlled variable calculation unit 180.

[0134] The gain setting unit 191 performs a gain setting process for setting the reaction force gain Gr1-C. More specifically, the gain setting unit 191 sets the reaction force gain Gr1-C based on the communication abnormality flag FL1-C indicating the result of the communication abnormality determination process performed by the communication abnormality determination unit 152. For example, when it is determined that no communication abnormality has occurred, the gain setting unit 191 sets the reaction force gain Gr1-C to the first gain Gr1-C1. When it is determined that a communication abnormality has occurred, the gain setting unit 191 sets the reaction force gain Gr1-C to the second gain Gr1-C2 that is larger than the first gain Gr1-C1. During the communication abnormality determination, the gain setting unit 191 may also set the reaction force gain Gr1-C to the third gain Gr1-C3 between the first gain Gr1-C1 and the second gain Gr1-C2.

[0135] Gain setting unit 192 performs a gain setting process for setting the reaction force gain Gr1-V. More specifically, gain setting unit 192 sets the reaction force gain Gr1-V based on a vehicle abnormality flag FL1-V indicating the result of the vehicle abnormality determination process performed by vehicle abnormality determination unit 154. For example, if it is determined that no vehicle abnormality has occurred, gain setting unit 192 sets the reaction force gain Gr1-V to the first gain Gr1-V1. If it is determined that a vehicle abnormality has occurred, gain setting unit 192 sets the reaction force gain Gr1-V to the second gain Gr1-V2, which is greater than the first gain Gr1-V1. While the vehicle abnormality determination is in progress, gain setting unit 192 may also set the reaction force gain Gr1-V to the third gain Gr1-V3, which is between the first gain Gr1-V1 and the second gain Gr1-V2.

[0136] The multiplication unit 193 calculates the reaction force gain Gr1 by multiplying the reaction force gain Gr1 -C by the reaction force gain Gr1 -V.

[0137] Filter unit 194 is provided to suppress abrupt changes in reaction force gain Gr1. Specifically, when switching reaction force gain Gr1 based on the results of the abnormality determination process, filter unit 194 gradually changes reaction force gain Gr1. For example, filter unit 194 includes a low-pass filter, and applies the low-pass filter to reaction force gain Gr1.

[0138] The multiplication unit 195 calculates the first reaction force control amount FB1 by multiplying the reference reaction force control amount FB0 by the reaction force gain Gr1 (FB1=Gr1×FB0).

[0139] The reference reaction force controlled variable calculation unit 180 calculates the reference reaction force controlled variable FB0. Figure 10 18 is a block diagram showing a functional configuration example of the reference reaction force controlled variable calculation unit 180. The reference reaction force controlled variable calculation unit 180 includes a target steering angle calculation unit 181, a switch 182, and a calculation unit 183.

[0140] The target steering angle calculation unit 181 calculates the target steering angle of the vehicle 100. Here, the steering angle refers to the steering angle of the wheels of the vehicle 100 , and corresponds to a pinion angle, a motor angle, a steering angle of a steering wheel, and the like.

[0141] For example, the target steering angle calculation unit 181 calculates the target steering angle according to the driving mode of the vehicle 100. During the remote operation of the vehicle 100, the target steering angle calculation unit 181 calculates the target steering angle based on the operation amount represented by the remote operation information OPE. During the automatic driving of the vehicle 100, the target steering angle calculation unit 181 calculates the target steering angle required for the automatic driving control. When the automatic driving and remote operation are performed simultaneously, the target steering angle calculation unit 181 may combine the target steering angle corresponding to the operation amount operated by the remote operator O with the target steering angle. and the target steering angle required for autopilot control During emergency control, the target steering angle calculation unit 181 can also calculate the target steering angle for emergency control.

[0142] Actual steering angle is the actual steering angle of the wheels of the vehicle 100. Detected by the vehicle status sensor and obtained from the vehicle status information.

[0143] Switch 182 will turn the actual steering angle and target steering angle Either of the two options is to select the steering angle

[0144] The calculation unit 183 selects the steering angle and vehicle speed, etc., calculate the reference reaction force control amount FB0. Typically, as the steering angle is selected As , the reference reaction force control amount FB0 becomes larger.

[0145] For example, considering interference, select the steering angle For example, if a crosswind blows while driving on a highway, the control of the steering wheel may be affected. In order to prevent such interference from being transmitted to the remote operator O, the actual steering angle is not selected. Select the target steering angle To this end, the switch 182 detects the presence of interference based on the vehicle state information. More specifically, the vehicle state information includes the vehicle speed, steering angle, and actual lateral acceleration of the vehicle 100. The target lateral acceleration is calculated based on the vehicle speed and steering angle. If the difference between the actual lateral acceleration and the target lateral acceleration exceeds a threshold, it is determined that there is interference such as crosswind. In this case, the switch 182 does not select the actual steering angle. The target steering angle Select to select the rudder angle

[0146] As described above, the reference reaction force control variable calculation unit 180 calculates the reference reaction force control variable FB0. The reaction force control variable calculation unit 190 sets the reaction force gain Gr1 based on the results of the abnormality determination process. Furthermore, the reaction force control variable calculation unit 190 multiplies the reference reaction force control variable FB0 by the reaction force gain Gr1 to calculate the first reaction force control variable FB1 (FB1 = Gr1 × FB0). The thus calculated first reaction force control variable FB1 is fed back to the remote operator terminal 200 as part of the vehicle information VCL.

[0147] 4. Example of a remote operator terminal

[0148] 4-1. Structural example

[0149] Figure 11 2 is a block diagram showing a configuration example of the remote operator terminal 200 . The remote operator terminal 200 includes a communication device 210 , a display device 220 , a remote operating member 230 , an operating reaction force actuator 240 , and a control device 250 .

[0150] The communication device 210 communicates with the vehicle 100 and the management device 300 .

[0151] The display device 220 presents various information to the remote operator O by displaying various information.

[0152] The remote operation components 230 are components operated when the remote operator O remotely operates the vehicle 100. The remote operation components 230 include a steering wheel (steering wheel), an accelerator pedal, a brake pedal, a direction indicator, and the like.

[0153] The operational reaction force actuator 240 applies an operational reaction force to the remote control component 230. For example, the operational reaction force actuator 240 includes a reaction force motor that applies a steering reaction force (steering reaction torque) to the steering wheel. The rotor of the reaction force motor is connected to the steering wheel via a speed reducer. The rotation of the reaction force motor can apply a steering reaction force to the steering wheel. The operation of the operational reaction force actuator 240 is controlled by the control device 250.

[0154] The control device 250 controls the remote operator terminal 200. The control device 250 includes one or more processors 260 (hereinafter referred to as processors 260) and one or more storage devices 270 (hereinafter referred to as storage devices 270). The processors 260 execute various processes. For example, the processor 260 includes a CPU. The storage device 270 stores various information required for the processes performed by the processor 260. Examples of the storage device 270 include volatile memory, nonvolatile memory, HDD, SSD, and the like.

[0155] Remote operation program PROG2 is a computer program executed by processor 260. Processor 260 executes remote operation program PROG2 to implement the functions of control device 250. Remote operation program PROG2 is stored in storage device 270. Alternatively, remote operation program PROG2 may be recorded on a computer-readable recording medium. Remote operation program PROG2 may also be provided via a network.

[0156] The control device 250 communicates with the vehicle 100 via the communication device 210. The control device 250 receives vehicle information VCL transmitted from the vehicle 100. The control device 250 displays the vehicle information VCL, including image information, on the display device 220, thereby presenting the vehicle information VCL to the remote operator O. The remote operator O can recognize the status of the vehicle 100 and the surrounding conditions based on the vehicle information VCL displayed on the display device 220.

[0157] Remote operator O operates remote operating component 230. A sensor provided on remote operating component 230 detects the amount of operation performed on remote operating component 230. Control device 250 generates remote operation information OPE reflecting the amount of operation performed on remote operating component 230 by remote operator O. Control device 250 then transmits this remote operation information OPE to vehicle 100 via communication device 210.

[0158] 4-2. Abnormality determination and notification processing

[0159] Figure 12 This is a block diagram for explaining the abnormality determination process and abnormality notification process in the remote operator terminal 200. The remote operator terminal 200 includes a receiving unit 251, a communication abnormality determination unit 252, a control unit 253, a terminal abnormality determination unit 254, and a transmitting unit 255 as functional blocks. These functional blocks are implemented by the communication device 210 and the control device 250.

[0160] During remote operation of vehicle 100, receiving unit 251 receives information transmitted from vehicle 100. The information transmitted from vehicle 100 includes the aforementioned vehicle information VCL. Based on the received information, receiving unit 251 determines the communication status with vehicle 100. Examples of the communication status include the presence or absence of received data, delay, communication speed, and radio wave reception strength.

[0161] The communication abnormality determination unit 252 performs a "communication abnormality determination process". More specifically, the communication abnormality determination unit 252 obtains information related to the communication status from the receiving unit 251. Furthermore, the communication abnormality determination unit 252 determines whether an abnormality has occurred in the communication from the vehicle 100 to the remote operator terminal 200 based on the communication status. A specific example of the communication abnormality determination process is the same as that described above. Figure 6 、 Figure 7 The examples shown are the same.

[0162] Communication anomaly flag FL2-C is information indicating the result of the communication anomaly determination process. If it is determined that no communication anomaly has occurred, i.e., if no communication anomaly has been detected, communication anomaly flag FL2-C is set to, for example, "0." On the other hand, if it is determined that a communication anomaly has occurred, i.e., if a communication anomaly has been detected, communication anomaly flag FL2-C is set to, for example, "1." Communication anomaly determination unit 252 outputs communication anomaly flag FL2-C.

[0163] The control unit 253 receives the vehicle information VCL and the abnormality flags (FL2-C, FL2-T). The control unit 253 presents the vehicle information VCL to the remote operator O. The control unit 253 also outputs the remote operation information OPE. If the abnormality flag indicates abnormality detection, the control unit 253 notifies the remote operator O that an abnormality has occurred.

[0164] The terminal abnormality determination unit 254 performs "terminal abnormality determination processing." More specifically, the terminal abnormality determination unit 254 receives remote operation information OPE output from the control unit 253. Furthermore, the terminal abnormality determination unit 254 determines whether an abnormality has occurred in the remote operator terminal 200 based on the remote operation information OPE. A specific example of the terminal abnormality determination processing will be described later (see paragraph 4-3).

[0165] Terminal abnormality flag FL2-T is information indicating the result of the terminal abnormality determination process. If it is determined that no terminal abnormality has occurred, that is, if no terminal abnormality has been detected, terminal abnormality flag FL2-T is set to, for example, "0." On the other hand, if it is determined that a terminal abnormality has occurred, that is, if a terminal abnormality has been detected, terminal abnormality flag FL2-T is set to, for example, "1." Terminal abnormality determination unit 254 outputs terminal abnormality flag FL2-T.

[0166] The transmitting unit 255 transmits the remote operation information OPE output from the control unit 253 to the vehicle 100 .

[0167] 4-3. Example of terminal abnormality determination processing

[0168] Figure 13 This is a flowchart showing an example of the terminal abnormality determination process performed by the terminal abnormality determination unit 254 .

[0169] In step S281, the terminal abnormality determination unit 254 receives the remote operation information OPE and obtains "operation amount change DO" which is the amount of change in the operation amount performed by the remote operator O from the previous value.

[0170] In step S282, the terminal abnormality determination unit 254 determines whether the operation amount change DO exceeds the first threshold value DO_th1. The first threshold value DO_th1 is an operation amount change DO that can also be determined to indicate that a terminal abnormality has occurred. For example, the first threshold value DO_th1 is an operation amount change DO that does not normally occur. If the operation amount change DO exceeds the first threshold value DO_th1 (step S282; "Yes"), the process proceeds to step S283. On the other hand, if the operation amount change DO is less than the first threshold value DO_th1 (step S282; "No"), the process proceeds to step S284.

[0171] In step S283 , the terminal abnormality determination unit 254 determines (judgments) that a terminal abnormality has occurred.

[0172] In step S284, the terminal abnormality determination unit 254 determines whether the operation amount change DO exceeds the second threshold value DO_th2. The second threshold value DO_th2 is smaller than the first threshold value DO_th1. For example, the second threshold value DO_th2 is the upper limit of the permissible range of the operation amount change DO. If the operation amount change DO exceeds the second threshold value DO_th2 (step S284: "Yes"), the process proceeds to step S285. On the other hand, if the operation amount change DO is less than the second threshold value DO_th2 (step S284: "No"), the process proceeds to step S287.

[0173] In step S285, the terminal abnormality determination unit 254 determines whether the state in which the operation amount change DO exceeds the second threshold value DO_th2 has continued for Td seconds. If this state has continued for Td seconds (step S285: "Yes"), the process proceeds to step S283. On the other hand, if this state has not continued for Td seconds (step S285: "No"), the process proceeds to step S286.

[0174] In step S286, the terminal abnormality determination unit 254 does not finalize the determination and sets the current state to “terminal abnormality determination in progress.” Thereafter, the process returns to step S281.

[0175] In step S287, the terminal abnormality determination unit 254 determines that no terminal abnormality has occurred and that the remote operator terminal 200 is normal. Thereafter, the process returns to step S281.

[0176] 4-4. Example of reaction force control amount calculation processing

[0177] As mentioned above Figure 12 As shown, the control unit 253 includes a reaction force control amount calculation unit 290. The reaction force control amount calculation unit 290 calculates a "reaction force control amount FB" for generating an operational reaction force.

[0178] Figure 14 2 is a block diagram showing a functional configuration example of the reaction force controlled variable calculation unit 290. The reaction force controlled variable calculation unit 290 includes a gain setting unit 291, a gain setting unit 292, a multiplication unit 293, a filter unit 294, and a multiplication unit 295.

[0179] The gain setting unit 291 performs a gain setting process for setting the reaction force gain Gr2-C. More specifically, the gain setting unit 291 sets the reaction force gain Gr2-C based on the communication abnormality flag FL2-C indicating the result of the communication abnormality determination process performed by the communication abnormality determination unit 252. For example, when it is determined that no communication abnormality has occurred, the gain setting unit 291 sets the reaction force gain Gr2-C to the first gain Gr2-C1. When it is determined that a communication abnormality has occurred, the gain setting unit 291 sets the reaction force gain Gr2-C to the second gain Gr2-C2 that is larger than the first gain Gr2-C1. During the communication abnormality determination, the gain setting unit 291 may also set the reaction force gain Gr2-C to the third gain Gr2-C3 between the first gain Gr2-C1 and the second gain Gr2-C2.

[0180] The gain setting unit 292 performs a gain setting process for setting the reaction force gain Gr2-T. More specifically, the gain setting unit 292 sets the reaction force gain Gr2-T based on the terminal abnormality flag FL2-T indicating the result of the terminal abnormality determination process performed by the terminal abnormality determination unit 254. For example, when it is determined that no terminal abnormality has occurred, the gain setting unit 292 sets the reaction force gain Gr2-T to the first gain Gr2-T1. When it is determined that a terminal abnormality has occurred, the gain setting unit 292 sets the reaction force gain Gr2-T to the second gain Gr2-T2 that is greater than the first gain Gr2-T1. During the period of terminal abnormality determination, the gain setting unit 292 may also set the reaction force gain Gr2-T to the third gain Gr2-T3 between the first gain Gr2-T1 and the second gain Gr2-T2.

[0181] The multiplication unit 293 calculates the reaction force gain Gr2 by multiplying the reaction force gain Gr2-C by the reaction force gain Gr2-T.

[0182] Filter unit 294 is provided to suppress abrupt changes in reaction force gain Gr2. Specifically, when switching reaction force gain Gr2 based on the results of the abnormality determination process, filter unit 294 gradually changes reaction force gain Gr2. For example, filter unit 294 includes a low-pass filter, and applies the low-pass filter to reaction force gain Gr2.

[0183] Multiplication unit 295 receives first reaction force control variable FB1 fed back from vehicle 100. Multiplication unit 295 calculates reaction force control variable FB (FB=Gr2×FB1=Gr1×Gr2×FB0) by multiplying first reaction force control variable FB1 by reaction force gain Gr2.

[0184] The control unit 253 controls the operational reaction force actuator 240 according to the reaction force control amount FB calculated by the reaction force control amount calculation unit 290. This applies an operational reaction force corresponding to the reaction force control amount FB to the remote operating component 230. The remote operator O operating the remote operating component 230 feels the operational reaction force applied to the remote operating component 230.

Claims

1. A remote operating system comprising: A mobile object that is the subject of remote operation by a remote operator; A remote operator terminal on the remote operator side; and one or more processors included in at least one of the mobile object and the remote operator terminal, The one or more processors are configured to perform: an abnormality determination process for determining whether at least one of an abnormality in communication between the mobile object and the remote operator terminal, an abnormality in the mobile object, and an abnormality in the remote operator terminal has occurred; a gain setting process of setting a larger gain when the abnormality occurs than when the abnormality does not occur; and a process of calculating the reaction force control amount by multiplying the reference reaction force control amount by the gain, The remote operator terminal is configured to apply an operation reaction force corresponding to the reaction force control amount to the remote operation component operated by the remote operator. The gain setting process includes: If it is determined that the abnormality has not occurred, the gain is set to the first gain; When it is determined that the abnormality has occurred, the gain is set to a second gain that is larger than the first gain; as well as The process of setting the gain to a third gain between the first gain and the second gain after the possibility of the abnormality occurring is detected and until whether the abnormality has occurred is determined.

2. The remote operation system according to claim 1, wherein: The abnormality determination process includes a communication abnormality determination process for determining whether the abnormality in the communication between the mobile object and the remote operator terminal has occurred.

3. The remote operation system according to claim 1 or 2, wherein: The abnormality determination process includes a moving body abnormality determination process of determining whether the abnormality of the moving body has occurred.

4. The remote operation system according to claim 1 or 2, wherein: The abnormality determination process includes a terminal abnormality determination process for determining whether the abnormality has occurred in the remote operator terminal.

5. An exception notification method is an exception notification method in a remote operating system, the remote operating system having: A mobile object that is the subject of remote operation by a remote operator; and The remote operator terminal on the remote operator side, in, The abnormal notification method includes: an abnormality determination process for determining whether at least one of an abnormality in communication between the mobile object and the remote operator terminal, an abnormality in the mobile object, and an abnormality in the remote operator terminal has occurred; a gain setting process of setting a larger gain when the abnormality occurs than when the abnormality does not occur; a process of calculating a reaction force control amount by multiplying a reference reaction force control amount by the gain; and a process of applying an operation reaction force corresponding to the reaction force control amount to the remote operation component operated by the remote operator; The gain setting process includes: If it is determined that the abnormality has not occurred, the gain is set to the first gain; When it is determined that the abnormality has occurred, setting the gain to a second gain that is larger than the first gain; and The process of setting the gain to a third gain between the first gain and the second gain after the possibility of the abnormality occurring is detected and until whether the abnormality has occurred is determined.

Images