Management device, remote operation system, and management method

Abstract

To provide technique that enables a quick response to a communication abnormality that occurs during remote operation of a moving body.SOLUTION: A management device communicates with a moving body which is a target of remote operation by a remote operator and a remote operator terminal on the remote operator side. More specifically, during remote operation of the moving body, the management device receives first information related to the remote operation from a first device, which is one of the moving body and the remote operator terminal, and transmits the first information to a second device, which is the other of the moving body and the remote operator terminal. The management device performs a communication abnormality determination process of determining whether an abnormality has occurred in first communication from the first device to the management device, based on the communication state of the first information. When it is determined that an abnormality in the first communication has occurred, the management device notifies the first device of the occurrence of the abnormality in the first communication.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to remote operation of a mobile object by a remote operator. [Background technology]

[0002] Patent Document 1 discloses a vehicle remote control device for remotely controlling a vehicle. The vehicle remote control device includes an operator operation unit that outputs an operation signal for remotely controlling the vehicle, and a plurality of control units that receive the operation signals output from the operator operation unit. Each control unit transmits the received operation signal to an operation information determination unit. If the operation contents of the operation signals received from the plurality of control units match, the operation information determination unit transmits the operation signal to the vehicle. On the other hand, if the operation contents of the operation signals received from the plurality of control units do not match, the operation information determination unit restricts the transmission of the operation signal to the vehicle.

[0003] Other technologies relating to remote control of vehicles are disclosed in Patent Documents 2, 3, and 4. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-061516 [Patent Document 2] JP 2018-506800 A [Patent Document 3] International Publication No. 2020 / 157942 [Patent Document 4] JP 2018-077649 A Summary of the Invention [Problem to be solved by the invention]

[0005] Consider the remote operation of a moving object (e.g., a vehicle or a robot) by a remote operator. During remote operation of the moving object, communication takes place between the moving object and the remote operator's terminal. If a communication abnormality occurs, it is desirable to take a quick response to ensure safety.

[0006] One object of the present disclosure is to provide a technology that enables a quick response to a communication abnormality that occurs during remote operation of a moving object. [Means for solving the problem]

[0007] The first aspect relates to a management device that communicates with a mobile object that is a target of remote operation by a remote operator and a remote operator terminal on the remote operator side. The management device includes one or more processors. The one or more processors During remote operation of the mobile object, receiving first information related to the remote operation from a first device, which is one of the mobile object and the remote operator terminal, and transmitting the first information to a second device, which is the other of the mobile object and the remote operator terminal; a communication abnormality determination process for determining whether or not an abnormality has occurred in the first communication from the first device to the management device based on a communication state of the first information; When it is determined that an abnormality has occurred in the first communication, a process of notifying the first device of the occurrence of the abnormality in the first communication. The apparatus is configured to execute the following steps:

[0008] The second aspect relates to a remote control system. The remote control system is A moving object that is a target of remote operation by a remote operator; a remote operator terminal on the remote operator side; A management device that communicates with the mobile unit and the remote operator terminal; Equipped with. The management device During remote operation of the mobile object, receiving first information related to the remote operation from a first device, which is one of the mobile object and the remote operator terminal, and transmitting the first information to a second device, which is the other of the mobile object and the remote operator terminal; a first communication anomaly determination process for determining whether or not an anomaly has occurred in the first communication from the first device to the management device based on a communication state of the first information; When it is determined that an abnormality has occurred in the first communication, a process of notifying the first device of the occurrence of the abnormality in the first communication. The apparatus is configured to execute the following steps:

[0009] The third aspect relates to a management method for managing a remote operation system including a mobile object that is a target of remote operation by a remote operator and a remote operator terminal on the remote operator side. The management method is During remote operation of the mobile object, a first device, which is one of the mobile object and the remote operator terminal, transmits first information related to the remote operation to a management device, and further transmits the first information from the management device to a second device, which is the other of the mobile object and the remote operator terminal; a first communication anomaly determination process in the management device for determining whether or not an anomaly has occurred in the first communication from the first device to the management device based on a communication state of the first information; when it is determined that an abnormality has occurred in the first communication, a process of notifying the first device of the occurrence of the abnormality in the first communication from the management device; Includes. Effect of the Invention

[0010] According to the present disclosure, a management device between a mobile object and a remote operator terminal has a function of determining whether a communication abnormality has occurred. Then, the management device notifies the communication partner of the occurrence of the communication abnormality. This allows the communication partner to be quickly notified of the occurrence of the communication abnormality. Therefore, it becomes possible to quickly respond to the occurrence of the communication abnormality. [Brief description of the drawings]

[0011] [Figure 1]1 is a schematic diagram illustrating a configuration example of a remote control system according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a conceptual diagram for explaining communication between a vehicle, a management device, and a remote operator terminal. [Diagram 3] FIG. 13 is a conceptual diagram for explaining a comparative example. [Figure 4] 4A to 4C are conceptual diagrams for explaining a first example of a communication abnormality determination process and an abnormality notification process according to an embodiment of the present disclosure. [Diagram 5] 4A to 4C are conceptual diagrams for explaining a first example of a communication abnormality determination process and an abnormality notification process according to an embodiment of the present disclosure. [Figure 6] 11 is a conceptual diagram for explaining a second example of a communication abnormality determination process and an abnormality notification process according to an embodiment of the present disclosure. FIG. [Figure 7] 11 is a conceptual diagram for explaining a second example of a communication abnormality determination process and an abnormality notification process according to an embodiment of the present disclosure. FIG. [Figure 8] 11A and 11B are conceptual diagrams for explaining a third example of a communication abnormality determination process and an abnormality notification process according to an embodiment of the present disclosure. [Figure 9] 10 is a flowchart illustrating an example of a communication abnormality determination process according to an embodiment of the present disclosure. [Figure 10] 10 is a flowchart illustrating an example of a communication abnormality determination process according to an embodiment of the present disclosure. [Figure 11] 1 is a block diagram illustrating an example configuration of a vehicle according to an embodiment of the present disclosure. [Figure 12] 1 is a block diagram illustrating a configuration example of a remote operator terminal according to an embodiment of the present disclosure. [Figure 13] 1 is a block diagram illustrating a configuration example of a management device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0013] 1. Overview of the remote control system Consider remote control (remote driving) of a moving object. Examples of moving objects that can be remotely controlled include vehicles, robots, flying objects, etc. A vehicle may be an autonomous vehicle or a vehicle driven by a driver. Examples of robots include logistics robots and work robots, etc. Examples of flying objects include airplanes and drones, etc.

[0014] As an example, in the following description, a case will be considered in which the moving object to be remotely controlled is a vehicle. In general, the term "vehicle" in the following description will be replaced with "moving object."

[0015] FIG. 1 is a schematic diagram showing a configuration example of a remote operation system 1 according to the present embodiment. The remote operation system 1 includes a vehicle 100, a remote operator terminal 200, and a management device 300. The vehicle 100 is a target of remote operation. The remote operator terminal 200 is a terminal device used when a remote operator O remotely operates the vehicle 100. The remote operator terminal 200 can also be called a remote operation HMI (Human Machine Interface). The management device 300 manages the remote operation system 1. The management of the remote operation system 1 includes, for example, assigning a remote operator O to the vehicle 100 that needs to be remotely operated. The management device 300 can communicate with the vehicle 100 and the remote operator terminal 200 via a communication network. Typically, the management device 300 is a management server on a cloud. The management server may be composed of multiple servers that perform distributed processing.

[0016] The vehicle 100 is equipped with various sensors including a camera. The camera captures an image of the surroundings of the vehicle 100 and acquires image information showing the surroundings of the vehicle 100. The vehicle information VCL is information obtained by various sensors and includes image information obtained by the camera. The vehicle 100 transmits the vehicle information VCL to the remote operator terminal 200 via the management device 300. That is, the vehicle 100 transmits the vehicle information VCL to the management device 300, and the management device 300 transfers the received vehicle information VCL to the remote operator terminal 200.

[0017] The remote operator terminal 200 receives the vehicle information VCL transmitted from the vehicle 100. The remote operator terminal 200 presents the vehicle information VCL to the remote operator O. Specifically, the remote operator terminal 200 is equipped with a display device and displays image information, etc. on the display device. The remote operator O recognizes the surrounding situation of the vehicle 100 by looking at the displayed information, and remotely operates the vehicle 100. The remote operation information OPE is information related to remote operation by the remote operator O. For example, the remote operation information OPE includes an operation amount by the remote operator O. The remote operator terminal 200 transmits the remote operation information OPE to the vehicle 100 via the management device 300. That is, the remote operator terminal 200 transmits the remote operation information OPE to the management device 300, and the management device 300 transfers the received remote operation information OPE to the vehicle 100.

[0018] The vehicle 100 receives the remote operation information OPE transmitted from the remote operator terminal 200. The vehicle 100 performs vehicle travel control in accordance with the received remote operation information OPE. In this manner, the remote operation of the vehicle 100 is realized.

[0019] 2 is a conceptual diagram for explaining terms used in the following description. For convenience, the communication of vehicle information VCL from the vehicle 100 to the management device 300 is called "upstream communication U100," and the communication of remote operation information OPE from the management device 300 to the vehicle 100 is called "downstream communication D100." Similarly, the communication of remote operation information OPE from the remote operator terminal 200 to the management device 300 is called "upstream communication U200," and the communication of vehicle information VCL from the management device 300 to the remote operator terminal 200 is called "downstream communication D200."

[0020] 2. Communication anomaly detection and notification The remote control system 1 according to this embodiment is equipped with an "abnormality determination function (abnormality detection function)" that determines whether an abnormality has occurred. Abnormalities in the remote control system 1 include an abnormality in the vehicle 100, an abnormality in the remote operator terminal 200, and an abnormality in communication. In the following, we will particularly consider "communication abnormality." If a communication abnormality occurs during remote operation of the vehicle 100, it is desirable to take a quick response to ensure safety.

[0021] 3 shows a comparative example. The remote operator terminal 200 judges whether or not an abnormality has occurred in the communication from the vehicle 100 to the remote operator terminal 200 based on the communication state of the vehicle information VCL received from the vehicle 100 (communication abnormality judgment process). For example, if an abnormality occurs in the uplink communication U100, the remote operator terminal 200 detects that an abnormality has occurred somewhere in the communication from the vehicle 100 to the remote operator terminal 200. If a communication abnormality is detected, the remote operator terminal 200 notifies (feeds back) the occurrence of the communication abnormality to the management device 300 and the vehicle 100 (abnormality notification process). Upon receiving the notification, the management device 300 and the vehicle 100 take measures to ensure safety. For example, the vehicle 100 executes evacuation control to evacuate to a safe position and stop.

[0022] However, in the comparative example, it takes a relatively long time for the occurrence of a communication abnormality to be fed back to the communication partner. Also, it is not possible to precisely identify the location where the communication abnormality occurred. Therefore, this embodiment proposes a technique that can quickly feed back the occurrence of a communication abnormality to the communication partner.

[0023] 2-1. First Example 4 and 5 are conceptual diagrams for explaining the first example. According to the first example, the management device 300 includes a "communication abnormality determination unit 10."

[0024] 4, the communication abnormality determination unit 10 determines whether or not an abnormality has occurred in the uplink communication U100 from the vehicle 100 to the management device 300 based on the communication state of the vehicle information VCL received from the vehicle 100 (communication abnormality determination process). A specific example of this communication abnormality determination process will be described later (see Section 2-4).

[0025] The communication abnormality flag FL is information indicating the result of the communication abnormality determination process. If it is determined that no communication abnormality has occurred, that is, if no communication abnormality has been detected, the communication abnormality flag FL is set to, for example, "0." On the other hand, if it is determined that a communication abnormality has occurred, that is, if a communication abnormality has been detected, the communication abnormality flag FL is set to, for example, "1."

[0026] When an abnormality in at least the upstream communication U100 is detected, the communication abnormality determination unit 10 notifies (feeds back) a communication abnormality flag FL indicating the occurrence of an abnormality in the upstream communication U100 to the vehicle 100, which is the communication partner (abnormality notification process).

[0027] In this way, the occurrence of an abnormality in the upstream communication U100 is fed back to the vehicle 100 as soon as possible. Therefore, it is possible to quickly respond to the occurrence of an abnormality in the upstream communication U100. For example, the vehicle 100 can perform evacuation control at an earlier timing. This improves safety.

[0028] 5, the communication anomaly determination unit 10 determines whether an anomaly has occurred in the upstream communication U200 from the remote operator terminal 200 to the management device 300 based on the communication state of the remote operation information OPE received from the remote operator terminal 200 (communication anomaly determination process). When an anomaly in at least the upstream communication U200 is detected, the communication anomaly determination unit 10 notifies (feeds back) a communication anomaly flag FL indicating the occurrence of an anomaly in the upstream communication U200 to the remote operator terminal 200, which is the communication partner (anomaly notification process).

[0029] In this way, the occurrence of an abnormality in the upstream communication U200 is fed back to the remote operator terminal 200 as soon as possible. This allows a quick response to the occurrence of an abnormality in the upstream communication U200. For example, the remote operator O can instruct the vehicle 100 to execute evacuation control at an earlier timing. This improves safety.

[0030] It can be generalized as follows. The "first device" is either the vehicle 100 or the remote operator terminal 200, and the "second device" is the other of the vehicle 100 and the remote operator terminal 200. During remote operation of the vehicle 100, the management device 300 receives first information related to the remote operation (vehicle information VCL or remote operation information OPE) from the first device and transmits the first information to the second device. The communication abnormality determination unit 10 of the management device 300 determines whether an abnormality has occurred in the first communication (U100 or U200) from the first device to the management device 300 based on the communication state of the first information received from the first device (communication abnormality determination process). If it is determined that an abnormality has occurred in the first communication, the communication abnormality determination unit 10 of the management device 300 notifies the first device of the occurrence of the abnormality in the first communication (abnormality notification process).

[0031] Thus, according to the first example, the management device 300 between the vehicle 100 and the remote operator terminal 200 has a function of determining whether or not a communication abnormality has occurred. Then, the management device 300 notifies the communication partner of the occurrence of the communication abnormality. This allows the communication partner to be promptly notified of the occurrence of the communication abnormality. Therefore, it becomes possible to take a prompt response to the occurrence of the communication abnormality. As a result, a safer remote operation system 1 is realized.

[0032] Furthermore, according to the first example, it is possible to specify whether the abnormality has occurred in the upstream communication U100 or the upstream communication U200. In other words, it is possible to specify the abnormality location in more detail.

[0033] 2-2. Second Example 6 and 7 are conceptual diagrams for explaining the second example. Descriptions that overlap with the above-mentioned first example will be omitted as appropriate.

[0034] The example shown in Fig. 6 is a modified example of the example shown in Fig. 4. When an abnormality in at least the upstream communication U100 is detected, the communication abnormality determination unit 10 notifies not only the vehicle 100 but also the remote operator terminal 200 of a communication abnormality flag FL indicating the occurrence of an abnormality in the upstream communication U100.

[0035] The example shown in Fig. 7 is a modified example of the example shown in Fig. 5. When an abnormality is detected in at least the upstream communication U200, the communication abnormality determination unit 10 notifies not only the remote operator terminal 200 but also the vehicle 100 of a communication abnormality flag FL indicating the occurrence of an abnormality in the upstream communication U200.

[0036] In this way, according to the second example, the occurrence of a communication abnormality is promptly notified not only to the first device on the transmitting side but also to the second device on the receiving side. Therefore, it is possible to take a prompt action against the occurrence of the communication abnormality. This realizes a safer remote control system 1.

[0037] 2-3.Third Example 8 is a conceptual diagram for explaining the third example. According to the third example, the vehicle 100, the remote operator terminal 200, and the management device 300 each include a communication anomaly judgment unit 10. For convenience, the communication anomaly judgment unit 10 of the vehicle 100 is referred to as a communication anomaly judgment unit 10-1. The communication anomaly judgment unit 10 of the remote operator terminal 200 is referred to as a communication anomaly judgment unit 10-2. The communication anomaly judgment unit 10 of the management device 300 is referred to as a communication anomaly judgment unit 10-3.

[0038] The communication anomaly determination unit 10-3 of the management device 300 is similar to the communication anomaly determination unit 10 described in the first or second example above. That is, the communication anomaly determination unit 10-3 determines whether or not an anomaly has occurred in the first communication (U100 or U200) from the first device to the management device 300 based on the communication state of the first information received from the first device (first communication anomaly determination process). If it is determined that an anomaly has occurred in the first communication, the communication anomaly determination unit 10-3 notifies the first device or both the first device and the second device of the occurrence of the anomaly in the first communication (first anomaly notification process).

[0039] The communication abnormality determination unit 10-1 of the vehicle 100 determines whether or not an abnormality has occurred in the downstream communication D100 from the management device 300 to the vehicle 100 based on the communication state of the remote operation information OPE received from the management device 300 (second communication abnormality determination process). When an abnormality in at least the downstream communication D100 is detected, the communication abnormality determination unit 10-1 notifies the management device 300 of a communication abnormality flag FL indicating the occurrence of an abnormality in the downstream communication D100 (second abnormality notification process). Furthermore, the management device 300 may notify the remote operator terminal 200 of the communication abnormality flag FL.

[0040] The communication abnormality determination unit 10-2 of the remote operator terminal 200 determines whether an abnormality has occurred in the downstream communication D200 from the management device 300 to the remote operator terminal 200 based on the communication state of the vehicle information VCL received from the management device 300 (second communication abnormality determination process). When an abnormality in at least the downstream communication D200 is detected, the communication abnormality determination unit 10-2 notifies the management device 300 of a communication abnormality flag FL indicating the occurrence of an abnormality in the downstream communication D200 (second abnormality notification process). Furthermore, the management device 300 may notify the vehicle 100 of the communication abnormality flag FL.

[0041] Generally speaking, it is as follows. During remote operation of the vehicle 100, the management device 300 receives first information related to the remote operation (vehicle information VCL or remote operation information OPE) from the first device, and transmits the first information to the second device. The communication abnormality determination unit 10 of the second device determines whether or not an abnormality has occurred in the second communication (D100 or D200) from the management device 300 to the second device based on the communication state of the first information received from the management device 300 (second communication abnormality determination process). If it is determined that an abnormality has occurred in the second communication, the communication abnormality determination unit 10 of the second device notifies at least the management device 300 of the occurrence of the abnormality in the second communication (second abnormality notification process).

[0042] According to the third example, the same effects as those of the first or second example can be obtained. Furthermore, it is possible to specify whether the abnormality occurs in the upstream communication U100, the upstream communication U200, the downstream communication D100, or the downstream communication D200. In other words, it is possible to specify the abnormality occurrence location in more detail.

[0043] 2-4. Example of communication anomaly detection process 9 is a flowchart showing an example of communication anomaly determination processing by the communication anomaly determination unit 10. As an example, consider the communication anomaly determination unit 10-3 of the management device 300. The same applies to the communication anomaly determination unit 10-1 of the vehicle 100 and the communication anomaly determination unit 10-2 of the remote operator terminal 200.

[0044] In step S110, the communication abnormality determination unit 10 determines whether or not the management device 300 is receiving data. If data is being received (step S110; Yes), the process proceeds to step S120. Otherwise (step S110; No), the process proceeds to step S130.

[0045] In step S120, the communication abnormality determination unit 10 determines whether the reception state is good or not. The reception state is represented by parameters such as communication speed and radio wave reception strength. If the parameter is equal to or greater than a predetermined threshold (step S120; Yes), the reception state is determined to be good, and the process proceeds to step S160. Otherwise (step S120; No), the process proceeds to step S130.

[0046] In step S130, the communication abnormality determination unit 10 determines whether a state in which no data is received or a state in which the reception state is not good has continued for Ta seconds. If such a poor state has continued for Ta seconds (step S130; Yes), the process proceeds to step S140. On the other hand, if such a poor state has not continued for Ta seconds (step S130; No), the process proceeds to step S150.

[0047] In step S140, the communication abnormality determination unit 10 determines (concludes) that a communication abnormality has occurred.

[0048] In step S150, the communication abnormality determination unit 10 does not confirm the determination, and sets the current state to “communication abnormality determination in progress.” After that, the process returns to step S110.

[0049] 10 is a flowchart showing an example of step S 160. In step S160, the amount of communication delay DL is taken into consideration.

[0050] In step S161, the communication abnormality determination unit 10 acquires the amount of communication delay DL based on reception information received from the communication partner.

[0051] In step S162, the communication abnormality determination unit 10 determines whether the delay amount DL exceeds a first threshold DL_th1. The first threshold DL_th1 is an amount of delay DL at which it may be concluded that a communication abnormality has occurred. For example, the first threshold DL_th1 is an amount of delay DL that does not normally occur. If the amount of delay DL exceeds the first threshold DL_th1 (step S162; Yes), the process proceeds to step S163. On the other hand, if the amount of delay DL is equal to or less than the first threshold DL_th1 (step S162; No), the process proceeds to step S164.

[0052] In step S163, the communication abnormality determination unit 10 determines (concludes) that a communication abnormality has occurred.

[0053] In step S164, the communication abnormality determination unit 10 determines whether the delay amount DL exceeds a second threshold DL_th2. The second threshold DL_th2 is smaller than the above-mentioned first threshold DL_th1. For example, the second threshold DL_th2 is the upper limit of the allowable range of the delay amount DL. If the delay amount DL exceeds the second threshold DL_th2 (step S164; Yes), the process proceeds to step S165. On the other hand, if the delay amount DL is equal to or smaller than the second threshold DL_th2 (step S164; No), the process proceeds to step S167.

[0054] In step S165, the communication abnormality determination unit 10 determines whether the state in which the delay amount DL exceeds the second threshold DL_th2 has continued for Tb seconds. If such a state has continued for Tb seconds (step S165; Yes), the process proceeds to step S163. On the other hand, if such a state has not continued for Tb seconds (step S165; No), the process proceeds to step S166.

[0055] In step S166, the communication abnormality determination unit 10 does not confirm the determination, and sets the current state to “communication abnormality determination in progress.” After that, the process returns to step S110.

[0056] In step S167, the communication abnormality determination unit 10 determines that no communication abnormality has occurred and that the communication is normal, and then the process returns to step S110.

[0057] 3. Examples of vehicles 3-1. Configuration example 11 is a block diagram showing an example of the configuration of the vehicle 100. The vehicle 100 includes a communication device 110, a sensor group 120, a traveling device 130, and a control device 150.

[0058] The communication device 110 communicates with the outside of the vehicle 100. For example, the communication device 110 communicates with a remote operator terminal 200 and a management device 300.

[0059] The sensor group 120 includes a recognition sensor, a vehicle state sensor, a position sensor, etc. The recognition sensor recognizes (detects) the situation around the vehicle 100. Examples of the recognition sensor include a camera, a LIDAR (Laser Imaging Detection and Ranging), a radar, etc. The vehicle state sensor detects the state of the vehicle 100. The vehicle state sensor includes a speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, etc. The position sensor detects the position and orientation of the vehicle 100. For example, the position sensor includes a GNSS (Global Navigation Satellite System).

[0060] The traveling device 130 includes a steering device, a drive device, and a braking device. The steering device steers the wheels. For example, the steering device includes an electric power steering (EPS) device. The drive device is a power source that generates a driving force. Examples of the drive device include an engine, an electric motor, and an in-wheel motor. The braking device generates a braking force.

[0061] The control device 150 is a computer that controls the vehicle 100. The control device 150 includes one or more processors 160 (hereinafter simply referred to as processor 160) and one or more storage devices 170 (hereinafter simply referred to as storage devices 170). The processor 160 executes various processes. For example, the processor 160 includes a CPU (Central Processing Unit). The storage device 170 stores various information required for processing by the processor 160. Examples of the storage device 170 include a volatile memory, a non-volatile memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), and the like. The control device 150 may include one or more ECUs (Electronic Control Units).

[0062] The vehicle control program PROG1 is a computer program executed by the processor 160. The functions of the control device 150 are realized as a result of the processor 160 executing the vehicle control program PROG1. The vehicle control program PROG1 is stored in the storage device 170. Alternatively, the vehicle control program PROG1 may be recorded in a computer-readable recording medium.

[0063] 3-2. Driving environment information The control device 150 uses the sensor group 120 to acquire driving environment information ENV indicating the driving environment of the vehicle 100. The driving environment information ENV is stored in the storage device 170.

[0064] The driving environment information ENV includes surrounding situation information indicating the recognition result by the recognition sensor. For example, the surrounding situation information includes image information captured by a camera. The surrounding situation information may include object information related to objects around the vehicle 100. Examples of objects around the vehicle 100 include pedestrians, other vehicles (preceding vehicles, parked vehicles, etc.), white lines, traffic signals, signs, roadside structures, etc. The object information indicates the relative position and relative speed of the object with respect to the vehicle 100.

[0065] The driving environment information ENV also includes vehicle state information indicating the vehicle state detected by the vehicle state sensor.

[0066] Furthermore, the driving environment information ENV includes vehicle position information indicating the position and direction of the vehicle 100. The vehicle position information is obtained by a position sensor. Highly accurate vehicle position information may be obtained by a self-position estimation process (localization) using map information and surrounding situation information (object information).

[0067] 3-3.Vehicle driving control The control device 150 executes vehicle driving control for controlling the driving of the vehicle 100. The vehicle driving control includes steering control, drive control, and braking control. The control device 150 executes the vehicle driving control by controlling the driving device 130 (the steering device, the drive device, and the braking device).

[0068] The control device 150 may perform automatic driving control based on the driving environment information ENV. More specifically, the control device 150 generates a driving plan for the vehicle 100 based on the driving environment information ENV. Furthermore, the control device 150 generates a target trajectory required for the vehicle 100 to drive according to the driving plan based on the driving environment information ENV. The target trajectory includes a target position and a target speed. Then, the control device 150 performs vehicle driving control so that the vehicle 100 follows the target trajectory.

[0069] 3-4. Processing related to remote control The following describes a case where remote operation of the vehicle 100 is performed. The control device 150 communicates with the remote operator terminal 200 via the communication device 110.

[0070] The control device 150 transmits the vehicle information VCL to the remote operator terminal 200. The vehicle information VCL is information necessary for remote operation by the remote operator O, and includes at least a part of the driving environment information ENV described above. For example, the vehicle information VCL includes surrounding situation information (particularly image information). The vehicle information VCL may further include vehicle state information and vehicle position information.

[0071] Furthermore, the control device 150 receives remote operation information OPE from the remote operator terminal 200. The remote operation information OPE is information related to remote operation by a remote operator O. For example, the remote operation information OPE includes an operation amount by the remote operator O. The control device 150 performs vehicle travel control according to the received remote operation information OPE.

[0072] Furthermore, the control device 150 may include the above-mentioned communication abnormality determination unit 10-1. The communication abnormality determination unit 10-1 performs the above-mentioned communication abnormality determination process (see Section 2). When the occurrence of a communication abnormality is detected, the communication abnormality determination unit 10-1 performs an abnormality notification process to notify the management device 300 of a communication abnormality flag FL indicating the occurrence of a communication abnormality.

[0073] 4. Example of a remote operator terminal 12 is a block diagram showing an example of the configuration of the remote operator terminal 200. The remote operator terminal 200 includes a communication device 210, a display device 220, an input device 230, and a control device 250.

[0074] The communication device 210 communicates with the vehicle 100 and the management device 300 .

[0075] The display device 220 presents various types of information to the remote operator O by displaying the various types of information.

[0076] The input device 230 accepts input from a remote operator O. For example, the input device 230 includes remote operation members that the remote operator O operates when remotely operating the vehicle 100. The remote operation members include a steering wheel, an accelerator pedal, a brake pedal, a turn signal, and the like.

[0077] The control device 250 controls the remote operator terminal 200. The control device 250 includes one or more processors 260 (hereinafter simply referred to as processor 260) and one or more storage devices 270 (hereinafter simply referred to as storage device 270). The processor 260 executes various processes. For example, the processor 260 includes a CPU. The storage device 270 stores various information required for processing by the processor 260. Examples of the storage device 270 include a volatile memory, a non-volatile memory, a HDD, and an SSD.

[0078] The remote operation program PROG2 is a computer program executed by the processor 260. The functions of the control device 250 are realized as a result of the processor 260 executing the remote operation program PROG2. The remote operation program PROG2 is stored in the storage device 270. Alternatively, the remote operation program PROG2 may be recorded in a computer-readable recording medium. The remote operation program PROG2 may be provided via a network.

[0079] The control device 250 communicates with the vehicle 100 via the communication device 210. The control device 250 receives vehicle information VCL transmitted from the vehicle 100. The control device 250 presents the vehicle information VCL, including image information, to the remote operator O by displaying the vehicle information VCL on the display device 220. The remote operator O can recognize the state of the vehicle 100 and the surrounding situation based on the vehicle information VCL displayed on the display device 220.

[0080] The remote operator O operates a remote control member of the input device 230. The amount of operation of the remote control member is detected by a sensor installed on the remote control member. The control device 250 generates remote operation information OPE that reflects the amount of operation of the remote control member by the remote operator O. Then, the control device 250 transmits the remote operation information OPE to the vehicle 100 via the communication device 210.

[0081] Furthermore, the control device 150 may include the above-mentioned communication abnormality determination unit 10-2. The communication abnormality determination unit 10-2 performs the above-mentioned communication abnormality determination process (see Section 2). When the occurrence of a communication abnormality is detected, the communication abnormality determination unit 10-2 performs an abnormality notification process of notifying the management device 300 of a communication abnormality flag FL indicating the occurrence of a communication abnormality.

[0082] 5.Example of management device 13 is a block diagram showing an example of the configuration of the management device 300. The management device 300 includes a communication device 310 and a control device 350.

[0083] The communication device 310 communicates with the vehicle 100 and the remote operator terminal 200 .

[0084] The control device 350 controls the management device 300. The control device 350 includes one or more processors 360 (hereinafter simply referred to as processor 360) and one or more storage devices 370 (hereinafter simply referred to as storage devices 370). The processor 360 executes various processes. For example, the processor 360 includes a CPU. The storage device 370 stores various information required for processing by the processor 360. Examples of the storage device 370 include a volatile memory, a non-volatile memory, an HDD, and an SSD.

[0085] The management program PROG3 is a computer program executed by the processor 360. The functions of the control device 350 are realized as a result of the processor 360 executing the management program PROG3. The management program PROG3 is stored in the storage device 370. Alternatively, the management program PROG3 may be recorded in a computer-readable recording medium. The management program PROG3 may be provided via a network.

[0086] The control device 350 communicates with the vehicle 100 and the remote operator terminal 200 via the communication device 310. The control device 350 receives vehicle information VCL transmitted from the vehicle 100. Then, the control device 350 transmits the received vehicle information VCL to the remote operator terminal 200. The control device 350 also receives remote operation information OPE transmitted from the remote operator terminal 200. Then, the control device 350 transmits the received remote operation information OPE to the vehicle 100.

[0087] Furthermore, the control device 350 includes the above-mentioned communication abnormality determination unit 10 (10-3). The communication abnormality determination unit 10 performs the above-mentioned communication abnormality determination process (see Section 2). When the occurrence of a communication abnormality is detected, the communication abnormality determination unit 10 performs an abnormality notification process to notify the vehicle 100 and the remote operator terminal 200 of a communication abnormality flag FL indicating the occurrence of a communication abnormality. [Explanation of symbols]

[0088] 1 Remote Control System 10 Communication abnormality determination section 100 vehicles 150 Control device 160 processors 170 Storage device 200 Remote operator terminal 250 Control device 260 processor 270 Storage device 300 Management device 350 Control device 360 Processor 370 Storage device FL Communication error flag OPE Remote Operation Information VCL vehicle information

Claims

1. A management device that communicates with a mobile object that is a target of remote operation by a remote operator and a remote operator terminal used by the remote operator during the remote operation, One or more processors; The one or more processors: assigning the remote operator to the mobile object requiring the remote operation; receiving, during the remote operation of the mobile object, first information related to the remote operation from a first device, which is one of the mobile object and the remote operator terminal, and transmitting the first information to a second device, which is the other of the mobile object and the remote operator terminal; a communication abnormality determination process for determining whether or not an abnormality has occurred in a first communication from the first device to the management device based on a communication state of the first information; when it is determined that the abnormality has occurred in the first communication, a process of notifying the first device of the occurrence of the abnormality in the first communication; configured to run Management device.

2. The management device according to claim 1 , The one or more processors further notify the second device of the occurrence of the anomaly in the first communication. Management device.

3. 3. The management device according to claim 1, The communication abnormality determination process includes: acquiring a delay amount of the first communication based on the first information received from the first device; a process of determining that the abnormality has occurred in the first communication when the amount of delay exceeds a threshold value; Includes Management device.

4. A moving object that is a target of remote operation by a remote operator; a remote operator terminal used by the remote operator during the remote operation; a management device that communicates with the mobile unit and the remote operator terminal; Equipped with The management device includes: assigning the remote operator to the mobile object requiring the remote operation; receiving, during the remote operation of the mobile object, first information related to the remote operation from a first device, which is one of the mobile object and the remote operator terminal, and transmitting the first information to a second device, which is the other of the mobile object and the remote operator terminal; a first communication anomaly determination process for determining whether or not an anomaly has occurred in a first communication from the first device to the management device based on a communication state of the first information; when it is determined that the abnormality has occurred in the first communication, a process of notifying the first device of the occurrence of the abnormality in the first communication; configured to run Remote control system.

5. A moving object that is a target of remote operation by a remote operator; a remote operator terminal on the remote operator side; a management device that communicates with the mobile unit and the remote operator terminal; Equipped with The management device includes: receiving, during the remote operation of the mobile object, first information related to the remote operation from a first device, which is one of the mobile object and the remote operator terminal, and transmitting the first information to a second device, which is the other of the mobile object and the remote operator terminal; a first communication anomaly determination process for determining whether or not an anomaly has occurred in a first communication from the first device to the management device based on a communication state of the first information; when it is determined that the abnormality has occurred in the first communication, a process of notifying the first device of the occurrence of the abnormality in the first communication; configured to run The second device is a second communication anomaly determination process for determining whether or not an anomaly has occurred in a second communication from the management device to the second device based on a communication state of the first information; when it is determined that the abnormality has occurred in the second communication, a process of notifying the management device of the occurrence of the abnormality in the second communication; configured to run Remote control system.

6. A management method for managing a remote operation system including a moving object to be remotely operated by a remote operator and a remote operator terminal used by the remote operator during the remote operation, comprising: A process of assigning the remote operator to the mobile object requiring the remote operation in a management device; a process of transmitting, during the remote operation of the mobile object, first information related to the remote operation from a first device, which is one of the mobile object and the remote operator terminal, to the management device, and further transmitting the first information from the management device to a second device, which is the other of the mobile object and the remote operator terminal; a first communication anomaly determination process in the management device, for determining whether or not an anomaly has occurred in a first communication from the first device to the management device based on a communication state of the first information; when it is determined that the abnormality has occurred in the first communication, a process of notifying the first device of the occurrence of the abnormality in the first communication from the management device; Includes Management method.

7. A management method for managing a remote operation system including a mobile object that is a target of remote operation by a remote operator and a remote operator terminal on the remote operator side, comprising: a process of transmitting, during the remote operation of the mobile object, first information related to the remote operation from a first device, which is one of the mobile object and the remote operator terminal, to a management device, and further transmitting the first information from the management device to a second device, which is the other of the mobile object and the remote operator terminal; a first communication anomaly determination process in the management device, for determining whether or not an anomaly has occurred in a first communication from the first device to the management device based on a communication state of the first information; when it is determined that the abnormality has occurred in the first communication, a process of notifying the first device of the occurrence of the abnormality in the first communication from the management device; a second communication anomaly determination process in the second device, for determining whether or not an anomaly has occurred in a second communication from the management device to the second device based on a communication state of the first information; when it is determined that the abnormality has occurred in the second communication, a process of notifying the management device of the occurrence of the abnormality in the second communication from the second device; Includes Management method.

Images