Management device, remote operation system, and management method

Abstract

The present disclosure relates to a management device, a remote operation system, and a management method, and provides a technology capable of quickly responding to a communication abnormality occurring in the course of remote operation of a mobile body. The management device communicates with a mobile body that is an object of remote operation by a remote operator and a remote operator terminal on the remote operator side. In more detail, in the course of remote operation of the mobile body, the management device receives first information associated with remote operation from a first device that is one of the mobile body and the remote operator terminal, and transmits the first information to a second device that is the other of the mobile body and the remote operator terminal. The management device performs a communication abnormality determination process of determining whether a first communication from the first device to the management device is abnormal based on a communication state of the first information. In a case where it is determined that the first communication is abnormal, the management device notifies the first device of the abnormality of the first communication.

Description

Technical Field

[0001] This disclosure relates to remote operation of a mobile body by a remote operator. Background Technology

[0002] Patent Document 1 discloses a vehicle remote operation device for remotely operating a vehicle. The vehicle remote operation device includes: an operator's operating unit that outputs operating signals for remotely operating the vehicle; and multiple control units that receive the operating signals output from the operator's operating unit. Each control unit transmits the received operating signal to an operation information determination unit. If the operating content of the operating signals received from the multiple control units is consistent, the operation information determination unit transmits the operating signal to the vehicle. Conversely, if the operating content of the operating signals received from the multiple control units is inconsistent, the operation information determination unit restricts the transmission of operating signals to the vehicle.

[0003] In addition, Patent Documents 2, 3 and 4 disclose a technology associated with remote operation of a vehicle.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2021-061516

[0007] Patent Document 2: Japanese Patent Application Publication No. 2018-506800

[0008] Patent Document 3: International Publication No. 2020 / 157942

[0009] Patent Document 4: Japanese Patent Application Publication No. 2018-077649 Summary of the Invention

[0010] Consider remote operation of a mobile object (e.g., a vehicle, a robot) by a remote operator. During remote operation, communication occurs between the mobile object and the remote operator's terminal. In the event of a communication failure, a rapid response is desired to ensure safety.

[0011] One object of this disclosure is to provide a technique for quickly responding to communication anomalies that occur during the remote operation of a mobile body.

[0012] The first viewpoint relates to a management device that communicates with a mobile body that is the object of remote operation by a remote operator and a remote operator terminal on the remote operator's side.

[0013] The management device has one or more processors.

[0014] One or more processors are configured to execute:

[0015] During the remote operation of the mobile body, the first device, which is both the mobile body and the remote operator's terminal, receives first information associated with the remote operation and sends the first information to the second device, which is both the mobile body and the remote operator's terminal, for processing.

[0016] The communication anomaly detection and processing method determines, based on the communication status of the first information, whether an anomaly has occurred in the first communication from the first device to the management device; and

[0017] If it is determined that the first communication has malfunctioned, the first device is notified of the processing of the first communication malfunction.

[0018] The second point concerns remote operating systems.

[0019] Remote operating systems have the following features:

[0020] A moving object that is remotely operated by a remote operator;

[0021] The remote operator terminal on the remote operator side; and

[0022] A management device that communicates with mobile bodies and remote operator terminals.

[0023] The management device is configured to execute:

[0024] During the remote operation of the mobile body, the first device, which is both the mobile body and the remote operator's terminal, receives first information associated with the remote operation and sends the first information to the second device, which is both the mobile body and the remote operator's terminal, for processing.

[0025] The first communication anomaly determination process determines, based on the communication status of the first information, whether an anomaly has occurred in the first communication from the first device to the management device; and

[0026] If it is determined that the first communication has malfunctioned, the first device is notified of the processing of the first communication malfunction.

[0027] The third viewpoint involves management methods for remote operating systems, including mobile bodies that are objects of remote operation by remote operators and remote operator terminals on the remote operator side.

[0028] Management methods include:

[0029] During the remote operation of the mobile body, a first device, which is both the mobile body and the remote operator's terminal, sends first information associated with the remote operation to a management device, and then sends the first information from the management device to a second device, which is both the mobile body and the remote operator's terminal, for processing.

[0030] The first communication anomaly detection process involves, within the management device, determining whether an anomaly has occurred in the first communication from the first device to the management device, based on the communication status of the first information; and

[0031] If the first communication is determined to be abnormal, the management device notifies the first device of the processing of the first communication abnormality.

[0032] According to this disclosure, the management device between the mobile unit and the remote operator's terminal has the function of determining whether a communication anomaly has occurred. Furthermore, the management device notifies the other party of the communication anomaly. Therefore, the communication anomaly is notified to the other party as quickly as possible. Thus, a rapid response to communication anomalies is possible. Attached Figure Description

[0033] Figure 1 This is a schematic diagram illustrating an example of the structure of a remote operating system according to an embodiment of the present disclosure.

[0034] Figure 2 It is a conceptual diagram used to illustrate the communication between vehicles, management devices, and remote operator terminals.

[0035] Figure 3 This is a concept map used to illustrate comparative examples.

[0036] Figure 4 This is a conceptual diagram illustrating a first example of communication anomaly determination processing and anomaly notification processing involved in the embodiments of this disclosure.

[0037] Figure 5 This is a conceptual diagram illustrating a first example of communication anomaly determination processing and anomaly notification processing involved in the embodiments of this disclosure.

[0038] Figure 6 This is a conceptual diagram illustrating a second example of communication anomaly determination processing and anomaly notification processing involved in the embodiments of this disclosure.

[0039] Figure 7 This is a conceptual diagram illustrating a second example of communication anomaly determination processing and anomaly notification processing involved in the embodiments of this disclosure.

[0040] Figure 8 This is a conceptual diagram illustrating a third example of communication anomaly determination processing and anomaly notification processing involved in the embodiments of this disclosure.

[0041] Figure 9 This is a flowchart illustrating an example of a communication anomaly determination process according to an embodiment of this disclosure.

[0042] Figure 10 This is a flowchart illustrating an example of a communication anomaly determination process according to an embodiment of this disclosure.

[0043] Figure 11 This is a block diagram illustrating a structural example of a vehicle according to an embodiment of the present disclosure.

[0044] Figure 12 This is a block diagram illustrating a structural example of a remote operator terminal according to an embodiment of the present disclosure.

[0045] Figure 13 This is a block diagram illustrating a structural example of a management device according to an embodiment of the present disclosure.

[0046] (Symbol Explanation)

[0047] 1: Remote operating system; 10: Communication anomaly detection unit; 100: Vehicle; 150: Control device; 160: Processor; 170: Storage device; 200: Remote operator terminal; 250: Control device; 260: Processor; 270: Storage device; 300: Management device; 350: Control device; 360: Processor; 370: Storage device; FL: Communication anomaly flag; OPE: Remote operation information; VCL: Vehicle information. Detailed Implementation

[0048] Embodiments of this disclosure are described with reference to the accompanying drawings.

[0049] 1. Overview of Remote Operating Systems

[0050] Consider remote operation (remote driving) of mobile bodies. Examples of mobile bodies that can be remotely operated include vehicles, robots, and flying objects. Vehicles can be either autonomous vehicles or vehicles driven by a human. Examples of robots include logistics robots and operational robots. Examples of flying objects include airplanes and drones.

[0051] As an example, in the following description, we consider the case where the moving body, as the object of remote operation, is a vehicle. In general, we will replace "vehicle" in the following description with "moving body".

[0052] Figure 1This is a schematic diagram illustrating an example structure of the remote operating system 1 according to this embodiment. The remote operating system 1 includes a vehicle 100, a remote operator terminal 200, and a management device 300. The vehicle 100 is the object of remote operation. The remote operator terminal 200 is a terminal device used by a remote operator O to remotely operate the vehicle 100. The remote operator terminal 200 can also be referred to as a remote operation HMI (Human Machine Interface). The management device 300 manages the remote operating system 1. Management of the remote operating system 1 includes, for example, assigning a remote operator O to the vehicle 100 requiring remote operation. The management device 300 can communicate with the vehicle 100 and the remote operator terminal 200 via a communication network. Typically, the management device 300 is a cloud-based management server. The management server may also consist of multiple servers performing distributed processing.

[0053] Vehicle 100 is equipped with various sensors, including cameras. The cameras capture images of the environment surrounding vehicle 100, obtaining image information representing the surrounding conditions. Vehicle Information Class (VCL) is information obtained from various sensors, including the image information obtained from the cameras. Vehicle 100 transmits the VCL to remote operator terminal 200 via management device 300. That is, vehicle 100 sends the VCL to management device 300, and management device 300 transmits the received VCL to remote operator terminal 200.

[0054] The remote operator terminal 200 receives vehicle information (VCL) sent from vehicle 100. The remote operator terminal 200 then displays the vehicle information VCL to the remote operator O. Specifically, the remote operator terminal 200 includes a display device that displays image information, etc. The remote operator O observes the displayed information, identifies the surrounding conditions of vehicle 100, and performs remote operations on vehicle 100. Remote operation information (OPE) is information related to the remote operation performed by the remote operator O. For example, remote operation information OPE includes the operation quantity performed by the remote operator O. The remote operator terminal 200 sends the remote operation information OPE to vehicle 100 via management device 300. That is, the remote operator terminal 200 sends the remote operation information OPE to management device 300, and management device 300 transmits the received remote operation information OPE to vehicle 100.

[0055] Vehicle 100 receives remote operation information (OPE) sent from remote operator terminal 200. Vehicle 100 then performs vehicle driving control according to the received OPE. This enables remote operation of vehicle 100.

[0056] Figure 2 This is a conceptual diagram used to explain the terminology used in the following description. For convenience, communication from vehicle 100 to vehicle information VCL from management device 300 is referred to as "uplink communication U100", and communication from management device 300 to remote operation information OPE from vehicle 100 is referred to as "downlink communication D100". Similarly, communication from remote operator terminal 200 to remote operation information OPE from management device 300 is referred to as "uplink communication U200", and communication from management device 300 to vehicle information VCL from remote operator terminal 200 is referred to as "downlink communication D200".

[0057] 2. Handling of communication anomalies and notifications

[0058] The remote operating system 1 described in this embodiment has an "anomaly detection function" to determine whether an anomaly has occurred. Examples of anomalies in the remote operating system 1 include anomalies in the vehicle 100, anomalies in the remote operator terminal 200, and communication anomalies. Hereinafter, we will particularly consider "communication anomalies." In the event of a communication anomaly during the remote operation of the vehicle 100, a rapid response is desirable to ensure safety.

[0059] Figure 3 A comparative example is shown. The remote operator terminal 200 determines whether an anomaly has occurred in the communication from vehicle 100 to the remote operator terminal 200 based on the communication status of the vehicle information VCL received from vehicle 100 (communication anomaly determination processing). For example, if an uplink communication U100 anomaly occurs, the remote operator terminal 200 detects where the anomaly occurred in the communication from vehicle 100 to the remote operator terminal 200. Upon detecting a communication anomaly, the remote operator terminal 200 notifies (feedback) the management device 300 and vehicle 100 of the communication anomaly (anomaly notification processing). Upon receiving this notification, the management device 300 and vehicle 100 take measures to ensure safety. For example, vehicle 100 performs a backoff control, reversing to a safe position and stopping.

[0060] However, in the comparative example, a relatively long time is taken until the communication partner reports a communication anomaly. Furthermore, it is impossible to pinpoint the exact location of the communication anomaly. Therefore, this embodiment proposes a technique that enables faster feedback of communication anomalies to the communication partner.

[0061] 2-1.Example 1

[0062] Figure 4 as well as Figure 5 This is a conceptual diagram used to illustrate the first example. According to the first example, the management device 300 includes a "communication anomaly determination unit 10".

[0063] exist Figure 4 In the example shown, the communication anomaly determination unit 10 determines whether an anomaly has occurred in the uplink communication U100 from vehicle 100 to management device 300 based on the communication status of the vehicle information VCL received from vehicle 100 (communication anomaly determination processing). A specific example of this communication anomaly determination processing will be described later (see Parts 2-4).

[0064] The communication anomaly flag FL indicates the result of the communication anomaly determination process. If no communication anomaly is detected, the communication anomaly flag FL is set to, for example, "0". Conversely, if a communication anomaly is detected, the communication anomaly flag FL is set to, for example, "1".

[0065] If an anomaly is detected in at least the uplink communication U100, the communication anomaly determination unit 10 notifies (feedback) the vehicle 100, which is the communication counterpart, of the communication anomaly flag FL (anomaly notification processing) indicating that an anomaly has occurred in the uplink communication U100.

[0066] This allows for early feedback to vehicle 100 regarding uplink communication U100 anomalies. Consequently, it enables rapid response to uplink communication U100 anomalies. For example, vehicle 100 can initiate backoff control at a more timely time. This improves safety.

[0067] exist Figure 5 In the example shown, the communication anomaly determination unit 10 determines whether an anomaly has occurred in the uplink communication U200 from the remote operator terminal 200 to the management device 300 based on the communication status of the remote operation information OPE received from the remote operator terminal 200 (communication anomaly determination processing). If an anomaly is detected in at least the uplink communication U200, the communication anomaly determination unit 10 notifies (feedbacks) the remote operator terminal 200, which is the communication counterpart, of a communication anomaly flag FL indicating that the uplink communication U200 has an anomaly (anomaly notification processing).

[0068] This allows for early feedback to the remote operator terminal 200 regarding uplink communication U200 anomalies. Consequently, it enables rapid response to uplink communication U200 anomalies. For example, the remote operator can instruct the vehicle 100 to execute retreat control at an earlier timing. This improves safety.

[0069] In a generalized manner, it is described below. "First device" refers to one party, the vehicle 100, and the remote operator terminal 200; "second device" refers to the other party, the vehicle 100, and the remote operator terminal 200. During the remote operation of the vehicle 100, the management device 300 receives first information associated with the remote operation (vehicle information VCL or remote operation information OPE) from the first device and sends this first information to the second device. The communication anomaly determination unit 10 of the management device 300 determines, based on the communication status of the first information received from the first device, whether an anomaly has occurred in the first communication (U100 or U200) from the first device to the management device 300 (communication anomaly determination processing). If an anomaly is determined in the first communication, the communication anomaly determination unit 10 of the management device 300 notifies the first device of the anomaly (anomaly notification processing).

[0070] Thus, according to Example 1, the management device 300 between vehicle 100 and remote operator terminal 200 has the function of determining whether a communication anomaly has occurred. Furthermore, the management device 300 notifies the other party of the communication anomaly. Therefore, the communication anomaly is notified to the other party as quickly as possible. Consequently, a rapid response to communication anomalies is possible. As a result, a more secure remote operating system 1 is achieved.

[0071] Furthermore, based on Example 1, the location of the anomaly can be pinpointed to either uplink communication U100 or uplink communication U200. That is, the location of the anomaly can be determined more precisely.

[0072] 2-2.Example 2

[0073] Figure 6 as well as Figure 7 This is a concept diagram used to illustrate the second example. Explanations that are repeated in the first example above are appropriately omitted.

[0074] Figure 6 The example shown is the one mentioned above. Figure 4 A variation of the example shown. In the event that an anomaly is detected in at least the uplink communication U100, the communication anomaly determination unit 10 notifies not only the vehicle 100 but also the remote operator terminal 200 of a communication anomaly flag FL indicating that an anomaly has occurred in the uplink communication U100.

[0075] Figure 7 The example shown is the one mentioned above. Figure 5 A variation of the example shown. In the event that an anomaly is detected in at least the uplink communication U200, the communication anomaly determination unit 10 notifies not only the remote operator terminal 200 but also the vehicle 100 of a communication anomaly flag FL indicating that an anomaly has occurred in the uplink communication U200.

[0076] Thus, according to Example 2, the communication anomaly is promptly notified not only to the first device on the sending side but also to the second device on the receiving side. Therefore, a rapid response to communication anomalies is possible. This results in a more secure remote operating system 1.

[0077] 2-3.Example 3

[0078] Figure 8 This is a conceptual diagram used to illustrate Example 3. According to Example 3, the vehicle 100, the remote operator terminal 200, and the management device 300 each possess a communication anomaly determination unit 10. For convenience, the communication anomaly determination unit 10 of the vehicle 100 is referred to as Communication Anomaly Determination Unit 10-1. The communication anomaly determination unit 10 of the remote operator terminal 200 is referred to as Communication Anomaly Determination Unit 10-2. The communication anomaly determination unit 10 of the management device 300 is referred to as Communication Anomaly Determination Unit 10-3.

[0079] The communication anomaly determination unit 10-3 of the management device 300 is the same as the communication anomaly determination unit 10 described in the first or second example above. That is, the communication anomaly determination unit 10-3 determines whether an anomaly has occurred in the first communication (U100 or U200) from the first device to the management device 300 based on the communication status of the first information received from the first device (first communication anomaly determination processing). If the first communication is determined to be an anomaly, the communication anomaly determination unit 10-3 notifies the first device or both the first device and the second device of the first communication anomaly (first anomaly notification processing).

[0080] The communication anomaly determination unit 10-1 of vehicle 100 determines whether an anomaly has occurred in the downlink communication D100 from management device 300 to vehicle 100 based on the communication status of the remote operation information OPE received from management device 300 (second communication anomaly determination process). If an anomaly is detected in at least the downlink communication D100, the communication anomaly determination unit 10-1 notifies management device 300 of a communication anomaly flag FL indicating that the downlink communication D100 has an anomaly (second anomaly notification process). Furthermore, management device 300 may also notify the remote operator terminal 200 of the communication anomaly flag FL.

[0081] The communication anomaly determination unit 10-2 of the remote operator terminal 200 determines whether an anomaly has occurred in the downlink communication D200 from the management device 300 to the remote operator terminal 200 based on the communication status of the vehicle information VCL received from the management device 300 (second communication anomaly determination process). If an anomaly is detected in at least the downlink communication D200, the communication anomaly determination unit 10-2 notifies the management device 300 of a communication anomaly flag FL indicating that the downlink communication D200 has an anomaly (second anomaly notification process). Furthermore, the management device 300 may also notify the vehicle 100 of the communication anomaly flag FL.

[0082] In a generalized manner, it is described below. During the remote operation of vehicle 100, management device 300 receives first information (vehicle information VCL or remote operation information OPE) associated with the remote operation from first device and sends the first information to second device. Communication anomaly determination unit 10 of the second device determines, based on the communication status of the first information received from management device 300, whether a second communication (D100 or D200) from management device 300 to the second device has experienced an anomaly (second communication anomaly determination processing). If it is determined that a second communication anomaly has occurred, communication anomaly determination unit 10 of the second device notifies at least management device 300 of the second communication anomaly (second anomaly notification processing).

[0083] According to the third example, the same effect as the first or second example above is obtained. Furthermore, it is possible to determine exactly which of the following communication types—uplink U100, uplink U200, downlink D100, and downlink D200—the location of the anomaly can be determined more precisely.

[0084] 2-4. Examples of communication anomaly detection and handling

[0085] Figure 9 This is a flowchart illustrating an example of communication anomaly determination processing performed by the communication anomaly determination unit 10. As an example, consider the communication anomaly determination unit 10-3 of the management device 300. The same applies to the communication anomaly determination unit 10-1 of the vehicle 100 and the communication anomaly determination unit 10-2 of the remote operator terminal 200.

[0086] In step S110, the communication error determination unit 10 determines whether the management device 300 has received data. If data has been received (step S110; "Yes"), the process proceeds to step S120. Otherwise (step S110; "No"), the process proceeds to step S130.

[0087] In step S120, the communication anomaly determination unit 10 determines whether the reception status is good. The reception status is represented by parameters such as communication speed and radio wave reception strength. If the parameter is above a predetermined threshold (step S120; "Yes"), the reception status is determined to be good, and the process proceeds to step S160. Otherwise (step S120; "No"), the process proceeds to step S130.

[0088] In step S130, the communication error determination unit 10 determines whether the state of no data received or the state of poor reception has lasted for Ta seconds. If such a poor state has lasted for Ta seconds (step S130; "Yes"), the process proceeds to step S140. On the other hand, if such a poor state has not lasted for Ta seconds (step S130; "No"), the process proceeds to step S150.

[0089] In step S140, the communication anomaly determination unit 10 determines that a communication anomaly has occurred.

[0090] In step S150, the communication error determination unit 10 makes an uncertain determination, setting the current state to "communication error determination in progress". After that, the process returns to step S110.

[0091] Figure 10 This is a flowchart illustrating an example of step S160. In step S160, the communication delay DL is taken into account.

[0092] In step S161, the communication anomaly determination unit 10 obtains the communication delay amount DL based on the received information received from the communication counterpart.

[0093] In step S162, the communication anomaly determination unit 10 determines whether the delay amount DL exceeds a first threshold DL_th1. The first threshold DL_th1 is the delay amount DL that can be determined to be a communication anomaly. For example, the first threshold DL_th1 is a delay amount DL that would not normally cause a communication anomaly. If the delay amount DL exceeds the first threshold DL_th1 (step S162; "Yes"), the process proceeds to step S163. On the other hand, if the delay amount DL is below the first threshold DL_th1 (step S162; "No"), the process proceeds to step S164.

[0094] In step S163, the communication anomaly determination unit 10 determines that a communication anomaly has occurred.

[0095] In step S164, the communication anomaly determination unit 10 determines whether the delay amount DL exceeds the second threshold DL_th2. The second threshold DL_th2 is less than the first threshold DL_th1. For example, the second threshold DL_th2 is the upper limit of the allowable range of the delay amount DL. If the delay amount DL exceeds the second threshold DL_th2 (step S164; "Yes"), the process proceeds to step S165. On the other hand, if the delay amount DL is below the second threshold DL_th2 (step S164; "No"), the process proceeds to step S167.

[0096] In step S165, the communication anomaly determination unit 10 determines whether the state in which the delay amount DL exceeds the second threshold DL_th2 has lasted for Tb seconds. If such a state has lasted for Tb seconds (step S165; "Yes"), the process proceeds to step S163. On the other hand, if such a state has not lasted for Tb seconds (step S165; "No"), the process proceeds to step S166.

[0097] In step S166, the communication error determination unit 10 makes an uncertain determination, setting the current state to "communication error determination in progress". Afterwards, the process returns to step S110.

[0098] In step S167, the communication anomaly determination unit 10 determines that no communication anomaly has occurred and communication is normal. Afterwards, the process returns to step S110.

[0099] 3. Examples of vehicles

[0100] 3-1. Structural Example

[0101] Figure 11 This is a block diagram showing a structural example of a vehicle 100. The vehicle 100 includes a communication device 110, a sensor group 120, a driving device 130, and a control device 150.

[0102] The communication device 110 communicates with the outside of the vehicle 100. For example, the communication device 110 communicates with the remote operator terminal 200 and the management device 300.

[0103] Sensor group 120 includes identification sensors, vehicle status sensors, and position sensors. Identification sensors identify (detect) the surrounding conditions of vehicle 100. Examples of identification sensors include cameras, LIDAR (Laser Imaging Detection and Ranging), and radar. Vehicle status sensors detect the state of vehicle 100. Vehicle status sensors include speed sensors, acceleration sensors, yaw rate sensors, and steering angle sensors. Position sensors detect the position and orientation of vehicle 100. For example, position sensors include GNSS (Global Navigation Satellite System).

[0104] The driving device 130 includes a steering device, a drive device, and a braking device. The steering device steers the wheels. For example, the steering device includes an electric power steering (EPS) device. The drive device is the power source that generates driving force. Examples of drive devices include engines, electric motors, and wheel hub motors. The braking device generates braking force.

[0105] The control unit 150 is a computer that controls the vehicle 100. The control unit 150 includes one or more processors 160 (hereinafter referred to as processors 160) and one or more storage devices 170 (hereinafter referred to as storage devices 170). The processors 160 perform various processes. For example, the processor 160 includes a CPU (Central Processing Unit). The storage devices 170 store various information required for the processes performed by the processors 160. Examples of storage devices 170 include volatile memory, non-volatile memory, HDD (Hard Disk Drive), SSD (Solid State Drive), etc. The control unit 150 may also include one or more ECUs (Electronic Control Units).

[0106] The vehicle control program PROG1 is a computer program executed by the processor 160. By executing the vehicle control program PROG1, the processor 160 enables the functions of the control device 150. The vehicle control program PROG1 is stored in the storage device 170. Alternatively, the vehicle control program PROG1 can also be recorded to a computer-readable recording medium.

[0107] 3-2. Driving Environment Information

[0108] The control device 150 uses the sensor group 120 to acquire driving environment information ENV, which represents the driving environment of the vehicle 100. The driving environment information ENV is stored in the storage device 170.

[0109] The driving environment information (ENV) includes surrounding condition information representing the results identified by the identification sensors. For example, the surrounding condition information includes image information captured by a camera. The surrounding condition information may also include object information relating to objects surrounding the vehicle 100. Examples of objects surrounding the vehicle 100 include pedestrians, other vehicles (vehicles in the lead, parked vehicles, etc.), white lines, signals, signs, roadside structures, etc. The object information indicates the relative position and relative speed of the object relative to the vehicle 100.

[0110] In addition, the driving environment information (ENV) includes vehicle status information that indicates the vehicle's status, detected by vehicle status sensors.

[0111] Furthermore, the driving environment information (ENV) includes vehicle position information representing the location and orientation of vehicle 100. Vehicle position information is obtained through position sensors. Alternatively, high-precision vehicle position information can be obtained through localization processing using map information and surrounding environment information (object information).

[0112] 3-3. Vehicle Driving Control

[0113] The control device 150 performs vehicle driving control to control the movement of the vehicle 100. Vehicle driving control includes steering control, drive control, and braking control. The control device 150 performs vehicle driving control by controlling the driving devices 130 (steering device, drive device, and braking device).

[0114] The control device 150 can also perform autonomous driving control based on the driving environment information ENV. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the driving environment information ENV. Furthermore, the control device 150 generates a target trajectory required for the vehicle 100 to drive according to the driving plan based on the driving environment information ENV. The target trajectory includes a target position and a target speed. Moreover, the control device 150 performs vehicle driving control by causing the vehicle 100 to track the target trajectory.

[0115] 3-4. Processing associated with remote operation

[0116] The following describes the remote operation of vehicle 100. Control device 150 communicates with remote operator terminal 200 via communication device 110.

[0117] The control device 150 sends the vehicle information VCL to the remote operator terminal 200. The vehicle information VCL is the information required for remote operation performed by the remote operator O, including at least a portion of the aforementioned driving environment information ENV. For example, the vehicle information VCL includes surrounding condition information (especially image information). The vehicle information VCL may also include vehicle status information and vehicle location information.

[0118] Additionally, the control device 150 receives remote operation information (OPE) from the remote operator terminal 200. The remote operation information (OPE) is information related to a remote operation performed by the remote operator O. For example, the remote operation information (OPE) includes the operation quantity performed by the remote operator O. The control device 150 performs vehicle driving control according to the received remote operation information (OPE).

[0119] Furthermore, the control device 150 may also include the aforementioned communication anomaly determination unit 10-1. The communication anomaly determination unit 10-1 performs the aforementioned communication anomaly determination processing (see Part 2). Moreover, when a communication anomaly is detected, the communication anomaly determination unit 10-1 performs an anomaly notification processing to notify the management device 300 of the communication anomaly flag FL indicating that a communication anomaly has occurred.

[0120] 4. Examples of remote operator terminals

[0121] Figure 12This is a block diagram illustrating an example of the structure of a remote operator terminal 200. The remote operator terminal 200 includes a communication device 210, a display device 220, an input device 230, and a control device 250.

[0122] The communication device 210 communicates with the vehicle 100 and the management device 300.

[0123] The display device 220 displays various information to provide prompts to the remote operator O.

[0124] Input device 230 accepts input from remote operator O. For example, input device 230 includes remote operating components operated by remote operator O when remotely operating vehicle 100. Remote operating components include a steering wheel, accelerator pedal, brake pedal, direction indicator, etc.

[0125] Control device 250 controls remote operator terminal 200. Control device 250 includes one or more processors 260 (hereinafter referred to as processor 260) and one or more storage devices 270 (hereinafter referred to as storage devices 270). Processor 260 performs various processes. For example, processor 260 includes a CPU. Storage device 270 stores various information required for the processes performed by processor 260. Examples of storage devices 270 include volatile memory, non-volatile memory, HDD, SSD, etc.

[0126] Remote operation program PROG2 is a computer program executed by processor 260. By executing remote operation program PROG2 by processor 260, the functions of control device 250 are realized. Remote operation program PROG2 is stored in storage device 270. Alternatively, remote operation program PROG2 can also be recorded to a computer-readable recording medium. Remote operation program PROG2 can also be provided via a network.

[0127] The control device 250 communicates with the vehicle 100 via the communication device 210. The control device 250 receives vehicle information VCL sent from the vehicle 100. The control device 250 displays the vehicle information VCL, including image information, on the display device 220 to prompt the remote operator O. The remote operator O can identify the status of the vehicle 100 and the surrounding conditions based on the vehicle information VCL displayed on the display device 220.

[0128] The remote operator O operates the remote operation component of the input device 230. The amount of operation performed by the remote operation component is detected by a sensor installed on it. The control device 250 generates remote operation information OPE reflecting the amount of operation performed by the remote operator O on the remote operation component. Furthermore, the control device 250 transmits this remote operation information OPE to the vehicle 100 via the communication device 210.

[0129] Furthermore, the control device 250 may also include the aforementioned communication anomaly determination unit 10-2. The communication anomaly determination unit 10-2 performs the aforementioned communication anomaly determination processing (see Part 2). Moreover, when a communication anomaly is detected, the communication anomaly determination unit 10-2 performs an anomaly notification processing to notify the management device 300 of the communication anomaly flag FL indicating that a communication anomaly has occurred.

[0130] 5. Examples of management devices

[0131] Figure 13 This is a block diagram illustrating a structural example of the management device 300. The management device 300 includes a communication device 310 and a control device 350.

[0132] The communication device 310 communicates with the vehicle 100 and the remote operator terminal 200.

[0133] Control device 350 controls management device 300. Control device 350 includes one or more processors 360 (hereinafter referred to as processors 360) and one or more storage devices 370 (hereinafter referred to as storage devices 370). Processor 360 performs various processes. For example, processor 360 includes a CPU. Storage device 370 stores various information required for the processes performed by processor 360. Examples of storage devices 370 include volatile memory, non-volatile memory, HDD, SSD, etc.

[0134] The management program PROG3 is a computer program executed by the processor 360. The processor 360 executes the management program PROG3 to implement the functions of the control device 350. The management program PROG3 is stored in the storage device 370. Alternatively, the management program PROG3 can also be recorded to a computer-readable recording medium. The management program PROG3 can also be provided via a network.

[0135] The control device 350 communicates with the vehicle 100 and the remote operator terminal 200 via the communication device 310. The control device 350 receives vehicle information (VCL) sent from the vehicle 100. Furthermore, the control device 350 sends the received vehicle information (VCL) to the remote operator terminal 200. Additionally, the control device 350 receives remote operation information (OPE) sent from the remote operator terminal 200. Furthermore, the control device 350 sends the received remote operation information (OPE) to the vehicle 100.

[0136] Furthermore, the control device 350 includes the aforementioned communication anomaly determination unit 10 (10-3). The communication anomaly determination unit 10 performs the aforementioned communication anomaly determination processing (see Part 2). Moreover, when a communication anomaly is detected, the communication anomaly determination unit 10 performs anomaly notification processing, notifying the vehicle 100 and the remote operator terminal 200 of a communication anomaly flag FL indicating that a communication anomaly has occurred.

Claims

1. A management device that communicates with a mobile body that is an object remotely operated by a remote operator and a remote operator terminal used by the remote operator during the remote operation, wherein, The management device has one or more processors. The one or more processors are configured to execute: The remote operator is assigned to the mobile body that requires remote operation; During the remote operation of the mobile body, a first piece of information associated with the remote operation is received from a first device that is one of the mobile body and the remote operator terminal, and the first piece of information is sent to a second device that is the other of the mobile body and the remote operator terminal for processing. Communication anomaly detection and processing: Based on the communication status of the first information, determine whether an anomaly has occurred in the first communication from the first device to the management device; as well as The process of notifying the first device of the occurrence of the abnormality in the first communication if it is determined that the first communication has encountered the abnormality.

2. The management device according to claim 1, wherein, The one or more processors then notify the second device that the first communication has encountered the anomaly.

3. The management device according to claim 1 or 2, wherein, The communication anomaly detection and processing includes: Processing for obtaining the delay amount of the first communication based on the first information received from the first device; and The process of determining that the first communication has encountered an anomaly when the delay exceeds a threshold.

4. A remote operating system, comprising: A moving object that is remotely operated by a remote operator; The remote operator terminal used by the remote operator during the remote operation; and A management device that communicates with the mobile body and the remote operator terminal. The management device is configured to perform: The remote operator is assigned to the mobile body that requires remote operation; During the remote operation of the mobile body, a first piece of information associated with the remote operation is received from a first device that is one of the mobile body and the remote operator terminal, and the first piece of information is sent to a second device that is the other of the mobile body and the remote operator terminal for processing. The first communication anomaly detection process determines, based on the communication status of the first information, whether an anomaly has occurred in the first communication from the first device to the management device; and The process of notifying the first device of the occurrence of the abnormality in the first communication if it is determined that the first communication has encountered the abnormality.

5. The remote operating system according to claim 4, wherein, The second device performs: The second communication anomaly determination process determines whether an anomaly has occurred in the second communication from the management device to the second device, based on the communication status of the first information. as well as The management device is notified of the abnormality in the second communication if the abnormality is determined to have occurred.

6. A management method for managing a remote operating system comprising a mobile body as an object to be remotely operated by a remote operator and a remote operator terminal used by the remote operator during the remote operation, wherein, The management method includes: In the management device, the remote operator is assigned processing for the mobile body that requires remote operation; During the remote operation of the mobile body, a first device, which is one of the mobile body and the remote operator terminal, sends first information associated with the remote operation to the management device, and then sends the first information from the management device to a second device, which is the other of the mobile body and the remote operator terminal, for processing. The first communication anomaly determination process involves, within the management device, determining whether an anomaly has occurred in the first communication from the first device to the management device, based on the communication status of the first information; and In the event that the first communication is determined to have an abnormality, the management device notifies the first device of the processing of the abnormality.

7. The management method according to claim 6, wherein, Also includes: The second communication anomaly determination process involves determining, based on the communication status of the first information, whether an anomaly has occurred in the second communication from the management device to the second device. as well as In the event that the second communication is determined to have an anomaly, the second device notifies the management device of the processing of the anomaly occurring in the second communication.

Images