Remote Operation System
The remote driving system addresses the challenge of detecting abnormalities on the remote side by integrating processors for communication cutoff and autonomous switching, enhancing safety during remote travel.
Patent Information
- Application Number
- JP2021161393
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-09-30
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-09-30
AI Technical Summary
Existing remote operation systems fail to reliably detect abnormalities on the remote side that could affect the operation of a mobile object, potentially leading to unsafe conditions during remote travel.
A remote driving system with integrated processors on both the remote and mobile sides to detect abnormalities, including communication cutoff control and switching to autonomous operation when necessary, ensuring both sides can independently monitor and respond to potential operational issues.
Enhances the safety of remote travel by enabling real-time detection and response to abnormalities on the remote side, ensuring the mobile object can switch to autonomous operation when needed, thereby maintaining safe operation.
Smart Images

Figure 0007586041000001 
Figure 0007586041000002 
Figure 0007586041000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a technology for remotely controlling the operation of a mobile object. [Background technology]
[0002] Patent Document 1 discloses a technology related to a remote control system. In this technology, a camera image captured by a camera mounted on a moving object is transmitted to a remote side via a communication line, and the camera image transmitted from the moving object is displayed on a display device on the remote side, thereby realizing remote control. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Japanese Patent Application Publication No. 10-145777 Summary of the Invention [Problem to be solved by the invention]
[0004] In remote driving of a mobile body, operation amount data acquired by an operation of an operator who drives the mobile body remotely is transmitted to the mobile body via a communication line. The mobile body controls driving, braking, and steering based on the operation amount data transmitted from the remote side, thereby performing remote driving based on remote control.
[0005] When an abnormality occurs on the remote side that may affect the operation of the mobile body while the mobile body is traveling, the mobile body is expected to switch from remote traveling based on the operation amount data transmitted from the remote side to evacuation traveling based on autonomous operation. However, in reality, the mobile body may not be able to grasp the abnormality that occurs on the remote side. Therefore, an improvement is desired so that the mobile body can reliably grasp the abnormality that occurs on the remote side.
[0006] An object of the present invention is to provide a technique capable of detecting, both on the remote side and on the mobile body side, an abnormality that occurs on the remote side while the mobile body is traveling. [Means for solving the problem]
[0007] The first aspect relates to a remote driving system including a moving object to be remotely driven and a remote driving device that communicates with the moving object. The remote driving device includes a remote communication device and a remote information processing device. The mobile object includes a mobile object communication device and a mobile object information processing device. The remote information processing device includes a remote storage device in which remote operation data including operation amount data is recorded, and a remote processor. The mobile-side information processing device includes a mobile-side processor. The remote processor of the remote information processing device executes an abnormality determination process to determine whether or not the state of the remote operation data is an abnormal state that may affect the operation of performing the motion of the moving object. The remote processor of the remote information processing device executes a communication cutoff control process for cutting off communication of the operation amount data to be transmitted to the mobile body based on the result of the abnormality determination process. The mobile-side processor of the mobile-side information processing device executes a communication interruption determination process for determining whether the state of the operation amount data is a communication interruption state. The mobile body processor of the mobile body information processing device executes anomaly handling processing to switch from a driving mode based on remote control to a driving mode based on autonomous operation by the mobile body based on the result of the communication outage determination processing. Effect of the Invention
[0008] According to the first aspect, when an abnormality occurs on the remote side that may affect the operation of the moving body while the moving body is running, the abnormality can be detected by both the remote side information processing device and the moving body side information processing device, which leads to ensuring the running safety of the moving body. [Brief description of the drawings]
[0009] [Figure 1] 1 is a block diagram showing a configuration example of a remote operation system according to an embodiment of the present invention; [Diagram 2] 1 is a block diagram showing an example of functions of a remote driving system according to an embodiment of the present invention; [Diagram 3] 5 is a flowchart showing a process performed by an abnormality determination processing unit of the remote operation system according to the embodiment of the present invention. [Figure 4] 5 is a flowchart showing a process performed by a communication cutoff control processing unit of the remote operation system according to the embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram showing an example of communication interruption in the remote operation system according to the embodiment of the present invention. [Figure 6] 5 is a flowchart showing a process performed by a communication interruption determination processing unit of the remote driving system according to the embodiment of the present invention. [Figure 7] 5 is a flowchart showing a process performed by an abnormality handling processing unit of the remote operation system according to the embodiment of the present invention. [Figure 8] 1 is a diagram showing an example of an abnormality that may occur in a remote driving system according to an embodiment of the present invention. [Figure 9] 11 is a diagram showing a reference example of a communication cutoff method in a communication cutoff control processing unit of a remote operation system. FIG. [Figure 10] 11 is a diagram showing a reference example of an abnormality detection method in an abnormality determination processing unit of a remote operation system. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A remote operation system according to an embodiment of the present invention will be described with reference to the accompanying drawings.
[0011] 1. Embodiment 1-1. Configuration example The remote driving system according to this embodiment includes a moving body to be remotely driven, and a remote driving device that communicates with the moving body. Fig. 1 is a block diagram showing a configuration example of a remote driving system 10 according to this embodiment. The remote driving system 10 includes a remote driving device 20 and a moving body 30. The remote driving device 20 includes a remote side information processing device 40 and a remote side communication device 50. The moving body 30 includes a moving body side communication device 60, a moving body side information processing device 70, and a control device 80.
[0012] The remote side information processing device 40 performs various information processing. The remote side information processing device 40 includes one or more remote side processors 100 (hereinafter simply referred to as the remote side processor 100), one or more remote side storage devices 110 (hereinafter simply referred to as the remote side storage device 110), a remote side encoder, and a remote side decoder. The remote side processor 100 executes various processing. For example, the remote side processor 100 is exemplified by an ECU, a CPU, etc. The remote side storage device 110 stores information of remote operation data 120 including at least operation amount data 130 to be transmitted to the moving body 30. The remote side storage device 110 is exemplified by a volatile memory, a non-volatile memory, a HDD, an SSD, etc. The remote side processor 100 executes a remote driving program, which is a computer program, to realize the functions of the remote side information processing device 40. The remote driving program is stored in the remote side storage device 110. The remote driving program may be recorded in a computer-readable recording medium. The remote driving program may be provided via a network.
[0013] The remote side communication device 50 is a communication interface that communicates with the mobile body 30, and in order to transmit the operation amount data 130 contained in the remote control data 120 to the mobile body 30, the remote side communication device 50 performs a process of allocating the operation amount data 130 to a communication line.
[0014] The mobile body side information processing device 70 performs various information processing. The mobile body side information processing device 70 includes one or more mobile body side processors 200 (hereinafter simply referred to as mobile body side processors 200), one or more mobile body side storage devices 210 (hereinafter simply referred to as mobile body side storage devices 210), and a mobile body side encoder. The mobile body side processor 200 executes various processes. For example, the mobile body side processor 200 is exemplified by an ECU, a CPU, etc. The mobile body side storage device 210 stores information of camera image data 220 captured by a camera mounted on the mobile body 30. Examples of the mobile body side storage device 210 include a volatile memory, a non-volatile memory, a HDD, an SSD, etc. The mobile body side processor 200 executes a mobile body driving program, which is a computer program, to realize the functions of the mobile body side information processing device 70. The mobile body driving program is stored in the mobile body side storage device 210. The mobile body driving program may be recorded in a computer-readable recording medium. The vehicle driving program may be provided via a network.
[0015] The control device 80 includes one or more ECUs, and the one or more ECUs perform steering control, drive control, and braking control of the moving body.
[0016] 1-2. Details of information processing When the remote side information processing device 40 judges that the state of the remote operation data 120 is an abnormal state that may affect the operation of the moving object, it issues a command to cut off communication of the operation amount data 130 to be transmitted to the remote side communication device 50. On the other hand, when the moving object side information processing device 70 judges that the state of the operation amount data 130 is a communication interruption state, it selects to switch from a driving mode based on remote operation to a driving mode based on autonomous operation. The information processing according to this embodiment includes characteristic processing as described below.
[0017] 2 is a block diagram showing an example of functions related to information processing according to this embodiment. The remote side information processing device 40 includes, as functional blocks, a remote side information input unit 300, an abnormality determination processing unit 310, and a communication cutoff control processing unit 320. These functional blocks are realized by the remote side processor 100 executing a remote driving program. Meanwhile, the mobile body side information processing device 70 of the mobile body 30 includes a mobile body side information input unit 330, a communication interruption determination processing unit 340, and an abnormality handling processing unit 350. These functional blocks are realized by the mobile body side processor 200 executing a mobile body driving program.
[0018] The remote side information input unit 300 performs a process of inputting the remote operation data 120 recorded in the remote side storage device 110. The remote operation data 120 recorded in the remote side storage device 110 includes the camera image data 220 received from the mobile object 30 in addition to the operation amount data 130 to be transmitted to the mobile object 30. After that, the input remote operation data 120 is output to the abnormality determination processing unit 310.
[0019] The abnormality determination processing unit 310 monitors the state of the input remote control data 120, determines whether or not there is an abnormal state in the remote control data 120, and generates an abnormality determination result. The process of the abnormality determination processing unit 310 will be described in detail later.
[0020] The communication cutoff control processing unit 320 generates a communication cutoff control command for performing communication cutoff control to the remote communication device 50 based on the abnormality determination result generated by the abnormality determination processing unit 310. The processing of the communication cutoff control processing unit 320 will be described in detail later.
[0021] The mobile-side information input unit 330 performs a process of inputting the operation amount data 130 transmitted from the remote driving device 20 via the mobile-side communication device 60. After that, the input operation amount data 130 is output to the communication interruption determination processing unit 340.
[0022] The communication interruption determination processing unit 340 monitors the state of the input operation amount data 130, determines whether or not a communication interruption has occurred in the communication with the remote driving device 20, and generates a communication interruption determination result. The processing of the communication interruption determination processing unit 340 will be described in detail later.
[0023] The abnormality handling processing unit 350 executes a process for selecting a driving mode based on the communication disruption determination result generated by the communication disruption determination processing unit 340. The process of the abnormality handling processing unit 350 will be described in detail later.
[0024] The control device 80 performs driving based on remote control or autonomous driving according to the selected driving mode. When the communication interruption determination processing unit 340 determines that the state of the operation amount data is a communication interruption state and the abnormality handling processing unit 350 selects a driving mode based on autonomous operation, the mobile body 30 performs an evacuation driving operation based on autonomous operation by the control device 80. The evacuation driving based on autonomous operation is not particularly limited, but examples thereof include deceleration driving based on braking control and stopping on the roadside based on braking control and steering control.
[0025] FIG. 3 is a flowchart showing the process of the abnormality determination processing unit 310.
[0026] In step S100, the abnormality determination processing unit 310 monitors the input remote control data 120 and determines whether the remote control data 120 indicates an abnormal state.
[0027] If it is determined that the state is abnormal (step S100; Yes), the process proceeds to step S110. Otherwise (step S100; No), the process proceeds to step S120.
[0028] In step S110, the abnormality determination processing unit 310 generates an abnormality determination result indicating that the remote control data 120 is in an abnormal state.
[0029] In step S120, the abnormality determination processing unit 310 generates an abnormality determination result indicating that the remote control data 120 is in a normal state.
[0030] In step S130, abnormality determination processing unit 310 outputs the abnormality determination result to communication cutoff control processing unit 320.
[0031] FIG. 4 is a flowchart showing the process of the communication cutoff control processing unit 320.
[0032] In step S200, communication cutoff control processor 320 determines whether the input abnormality determination result indicates an abnormal state.
[0033] If it is determined that an abnormal state exists (step S200; Yes), the process proceeds to step S210. Otherwise (step S200; No), the process proceeds to step S220.
[0034] In step S210, the communication cutoff control processing unit 320 generates a communication cutoff control command for instructing the remote communication device 50 to cut off communication.
[0035] In step S220, the communication cutoff control processing unit 320 generates a communication cutoff control command not to instruct the remote communication device 50 to cut off communication.
[0036] In step S230, the communication cutoff control processing unit 320 outputs a communication cutoff control command to the remote communication device 50.
[0037] 5 is a diagram showing an example of communication cutoff based on a communication cutoff control command. For example, when the communication cutoff control command output from the remote side information processing device 40 is a command to cut off communication with the remote side communication device 50, the power supply of the remote side communication device 50 is turned off based on the communication cutoff control command, thereby cutting off communication with the mobile object 30. Alternatively, a cutoff device including a switch is mounted on the remote side communication device 50, and the connection state of the cutoff device is switched to a cutoff state based on the communication cutoff control command, thereby cutting off communication with the mobile object 30.
[0038] FIG. 6 is a flowchart showing the process of the communication interruption determination processing unit 340.
[0039] In step S300, the communication interruption determination processing unit 340 monitors the state of the input operation amount data 130, and determines whether the operation amount data 130 is in a communication interruption state.
[0040] If it is determined that the communication is interrupted (step S300; Yes), the process proceeds to step S310. Otherwise (step S300; No), the process proceeds to step S320.
[0041] In step S310, the communication disruption determination processing unit 340 generates a communication disruption determination result indicating that the operation amount data 130 is in a communication disruption state.
[0042] In step S320, the communication disruption determination processing unit 340 generates a communication disruption determination result indicating that the operation amount data 130 is in a communication connected state.
[0043] In step S330, the communication disruption determination processing unit 340 outputs the communication disruption determination result to the abnormality handling processing unit 350.
[0044] FIG. 7 is a flowchart showing the processing of the abnormality handling processing unit 350.
[0045] In step S400, the abnormality handling processing unit 350 determines whether the input communication disruption determination result indicates a communication disruption state.
[0046] If it is determined that the communication is interrupted (step S400; Yes), the process proceeds to step S410. Otherwise (step S400; No), the process proceeds to step S420.
[0047] In step S410, the abnormality handling processing unit 350 selects a driving mode based on autonomous operation.
[0048] In step S420, the abnormality handling processor 350 selects a driving mode based on remote control.
[0049] In step S430, the driving mode is switched to the selected mode.
[0050] 1-3. Examples of abnormalities occurring in the remote control device FIG. 8 is a diagram showing an example of an abnormality that may occur in the remote driving device 20. Here, an example is given in which the camera image data 220 transmitted from the moving object 30 is displayed on the display device included in the remote driving device 20, and the camera image data 220 displayed in time series becomes abnormal. Specifically, in the frames of the camera image data 220 displayed in time series, if the difference in time between the time of a reference frame and the time of the previous frame does not satisfy a predetermined time, that is, if the time interval between frames is not constant, it is considered that the operation by the operator who viewed the camera image data 220 will be affected. Therefore, in this case, the abnormality determination processing unit 310 determines that the state of the remote operation data 120 is in an abnormal state that may affect the operation of performing the operation of the moving object.
[0051] 1-4.Effects When an abnormality occurs in the remote driving device that may affect the operation of the moving body while the moving body is running, the abnormality that occurred on the remote side can be detected by both the remote side information processing device and the moving body side information processing device, which leads to ensuring the running safety of the moving body.
[0052] 2.Reference example A reference example of a communication cutoff method in a communication cutoff control processing unit that can be executed in the remote operation system according to the embodiment and a reference example of an abnormality detection method in an abnormality determination processing unit will be described.
[0053] 2-1. Reference example of communication cutoff control method in communication cutoff control processing unit 9 shows a reference example of a communication cutoff control method in the communication cutoff control processing unit 320 of the remote operation system. Specifically, when the remote side information processing device 40 detects an abnormality in the abnormality determination processing unit 310, the remote side information processing device 40 transmits an abnormality notification to the mobile side information processing device 70. The mobile side information processing device 70 performs an abnormality handling process in the abnormality handling processing unit 350 based on the abnormality notification transmitted from the remote side information processing device 40. Then, in order to inform the remote side information processing device 40 that an abnormality handling process has been performed, the remote side information processing device 40 transmits a handling response to the remote side information processing device 40. When a handling response is received within a predetermined time in the handling response transmitted from the mobile side information processing device 70, the remote side information processing device 40 does not issue a command to cut off communication to the remote side communication device 50. On the other hand, when a handling response is not received within the predetermined time, the remote side information processing device 40 issues a command to cut off communication to the remote side communication device 50.
[0054] From the above, when an abnormality occurs in the remote driving device 20 that may affect the operation of the moving body while the moving body is running, the abnormality that occurred on the remote side can be detected by both the remote side information processing device 40 and the moving body side information processing device 70. This leads to ensuring the running safety of the moving body.
[0055] 2-2. Reference example of anomaly detection method in the anomaly judgment processing unit 10 shows a reference example of an abnormality detection method in the abnormality determination processing unit 310 of the remote driving system. Here, a reference example of an abnormality detection method is given in the case where the camera image data 220 transmitted from the moving body 30 is displayed on the display device included in the remote driving device 20, and the camera image data 220 displayed in time series is abnormal. Specifically, the camera image data 220 displayed on the display device is an image in which a marker is superimposed at a predetermined position on a camera image captured by a camera mounted on the moving body. Then, luminance value information of the marker acquired by an optical sensor attached to the part on the screen of the display device where the marker is displayed is recorded in the remote side storage device 110 as information included in the remote operation data 120.
[0056] Anomaly determination processing unit 310 compares the brightness value information of the marker with predetermined brightness value information to determine whether the marker is detected or not. Then, it is determined whether the time interval of the frames determined to have detected the marker is within a predetermined time interval range. If it is determined that the time interval of the frames is within the predetermined time interval range, an abnormality determination result is generated indicating that the state of the camera image data 220 is normal. On the other hand, if it is determined that the time interval of the frames determined to have detected the marker is outside the predetermined time interval range, an abnormality determination result is generated indicating that the state of the camera image data 220 is abnormal.
[0057] From the above, even if an abnormality occurs in the camera image data 220 while the moving body is traveling and may affect the operation of the moving body, the abnormality can be detected on the remote side. Furthermore, an abnormality that occurs on the remote side can be detected by both the remote side information processing device 40 and the moving body side information processing device 70. This leads to ensuring the traveling safety of the moving body. [Explanation of symbols]
[0058] 10 Remote Operation System 20 Remote control device 30 Mobile 40 Remote information processing device 50 Remote communication device 60 Mobile communication device 70 Mobile side information processing device 80 Control device 100 Remote Processor 110 Remote storage device 120 Remote Control Data 130 Operational volume data 200 Mobile side processor 210 Mobile side storage device 220 Camera image data 300 Remote side information input unit 310 Abnormality determination processing unit 320 Communication cutoff control processing unit 330 Mobile side information input unit 340 Communication interruption determination processing unit 350 Abnormality handling processing unit
Claims
[Claim 1] A remote driving system including a moving object to be remotely driven and a remote driving device that communicates with the moving object, The remote operation device includes a remote communication device and a remote information processing device, The mobile unit includes a mobile unit communication device and a mobile unit information processing device, The remote information processing device includes a remote storage device in which remote operation data including operation amount data is recorded, and a remote processor; The mobile-side information processing device includes a mobile-side processor, The remote processor of the remote information processing device execute an abnormality determination process for determining whether or not the state of the remote operation data is an abnormal state that may affect the operation of the moving object; executes a communication cutoff control process for cutting off communication of the operation amount data to be transmitted to the moving body based on a result of the abnormality determination process; The mobile-side processor of the mobile-side information processing device execute a communication interruption determination process for determining whether or not the state of the operation amount data is a communication interruption state; Based on the result of the communication interruption determination process, an abnormality handling process is executed to switch from a driving mode based on remote control to a driving mode based on autonomous operation by the moving body. Remote driving system.
Citation Information
Patent Citations
Camera image display device for mobile object
JP1998145777A
Electrical apparatus
JP2012186717A
Communication method, communication system, transmitter and receiver
JP2020142906A
Control apparatus, control method and program
JP2020164056A
Vehicle remote indication system
JP2020177292A
