Remote operation support device

Abstract

The present disclosure relates to a remote operation support device comprising an operation interface that accepts input from a remote operator, a state detection unit that detects surrounding environment information of the autonomous operating system and information about the autonomous operating system, a fault occurrence determination unit that acquires the surrounding environment information and information of the autonomous operating system from the state detection unit, determines whether a fault has occurred in the autonomous operating system, and outputs this as information regarding the fault occurrence, a post-fault state recording unit that acquires the surrounding environment information and information of the autonomous operating system from the state detection unit and acquires information regarding the fault occurrence from the fault occurrence determination unit and records it as information at least from before the fault occurred, and a post-fault situation presentation unit that presents the information recorded in the post-fault state recording unit to a remote operator.

Description

[Technical field]

[0001] The present disclosure relates to a remote operation support device, and more particularly to a remote operation support device that supports recovery work of an autonomous operating system when an event occurs in which the autonomous operating system cannot continue its autonomous operation. [Background technology]

[0002] Patent Document 1 discloses a technology that uses remote control to avoid accidents involving autonomous vehicles, and describes how the cause of the accident is identified from vehicle information and surrounding information before and after the accident using sensors, and notifies the remote operator. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-166541 A Summary of the Invention [Problem to be solved by the invention]

[0004] With the technology disclosed in Patent Document 1, when a fault occurs in an autonomous vehicle, the remote operator can grasp the cause of the fault, but has the problem of being unable to grasp whether the cause of the fault is still affecting the vehicle to this day.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a remote operation support device that, when some kind of failure occurs in an autonomous operation system and autonomous operation stops, allows an operator to determine from a remote location whether the cause of the failure is still having an effect to the present when performing recovery work to restore autonomous operation. [Means for solving the problem]

[0006] The present disclosure relates to a remote operation support device that temporarily operates an autonomous operation system from a remote location, and includes an operation interface that accepts input from a remote operator, and a peripheral environment information and a remote control information for the autonomous operation system. system a state detection unit that detects surrounding environment information and the surrounding environment information of the autonomous operating system from the state detection unit; system Acquire information, determine whether a fault has occurred in the autonomous operating system, and provide a report on the occurrence of the fault. Error Occurrence a fault occurrence determination unit that outputs the surrounding environment information and the system and acquiring information on the occurrence of a fault from the fault occurrence determination unit. Life situation At least, the failure information is obtained. straight From the front up to now a post-failure status recording unit for recording the information on the post-failure status as information on the post-failure status; and a post-failure status presentation unit for presenting the information recorded by the post-failure status recording unit to the remote operator. a playback speed calculation unit that calculates a playback speed of the information recorded in the post-fault state recording unit in accordance with the operation status of the remote operator received by the operation interface; Equipped with The post-failure state presentation unit presents the information recorded by the post-failure state recording unit to the remote operator using the playback speed calculated by the playback speed calculation unit. do. Effect of the Invention

[0007] According to the remote operation support device of the present disclosure, by providing not only the current state but also information from at least before the occurrence of the failure, it is possible to identify the cause of the failure and determine whether the cause of the failure is still having an effect. [Brief description of the drawings]

[0008] [Figure 1] 10 is a flowchart showing a flow of an action of a remote operator operating a remote operation assistance device according to the present disclosure. [Diagram 2] 1 is a block diagram showing a configuration of a remote operation assistance device according to a first embodiment of the present disclosure. [Diagram 3] 5 is a flowchart showing an operation of the remote operation assistance device according to the first embodiment of the present disclosure. [Figure 4]13 is a diagram showing an example in which information for assisting in grasping information about a failure is added to data recorded in a post-failure information recording unit; [Diagram 5] 11 is a block diagram showing a configuration of a remote operation assistance device according to a second embodiment of the present disclosure. FIG. [Figure 6] 10 is a flowchart showing an operation of a remote operation assistance device according to a second embodiment of the present disclosure. [Figure 7] FIG. 11 is a block diagram showing a configuration of a remote operation assistance device according to a third embodiment of the present disclosure. [Figure 8] 11 is a diagram showing an example in which information that assists in identifying a location related to the cause of a failure is added to data recorded in a post-failure information recording unit. FIG. [Figure 9] 11 is a diagram showing an example in which information that assists in identifying a location related to the cause of a failure is added to data recorded in a post-failure information recording unit. FIG. [Figure 10] 13 is a flowchart showing an operation of the remote operation assistance device according to the third embodiment of the present disclosure. [Figure 11] FIG. 11 is a block diagram showing a configuration of a remote operation assistance device according to a fourth embodiment of the present disclosure. [Figure 12] 13 is a flowchart showing an operation of the remote operation assistance device according to the fourth embodiment of the present disclosure. [Figure 13] FIG. 13 is a block diagram showing a configuration of a remote operation assistance device according to a fifth embodiment of the present disclosure. [Figure 14] 13 is a flowchart showing an operation of the remote operation assistance device according to the fifth embodiment of the present disclosure. [Figure 15] FIG. 1 is a diagram illustrating a hardware configuration for implementing a remote operation support device according to first to fifth embodiments of the present disclosure. [Figure 16] FIG. 1 is a diagram illustrating a hardware configuration for implementing a remote operation support device according to first to fifth embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] <Introduction> 1 is a flowchart showing a flow of actions of a remote operator who operates a remote operation support device according to the present disclosure. Hereinafter, the role of the remote operation support device according to the present disclosure will be described with reference to FIG.

[0010] 1, the autonomous operating system notifies the occurrence of a fault from the autonomous operating system. This notification is made, for example, via a monitor device for the autonomous operating system provided separately from the remote operation support device.

[0011] The remote operator who receives the notification checks the current state of the corresponding autonomous operating system (step S2). This is also checked via a monitor device of the autonomous operating system.

[0012] Next, the remote operator grasps the details of the current fault in the autonomous operation system via the remote operation support device (step S3).

[0013] Next, the remote operator uses the remote operation support device to consider a means for recovering from the current failure of the autonomous operation system (step S4).

[0014] Finally, the remote operator restores the failure by remotely operating the autonomous operation system based on the restoration means (step S5), thereby completing the series of operations.

[0015] In the following explanation, a self-driving vehicle will be used as an example of an autonomous operating system; however, this is not limited to this and examples include automatically moving man-made objects such as flying objects, drones, probes, and mobility robots.

[0016] <Embodiment 1> <Device configuration> 2 is a block diagram showing a configuration of a remote operation support device 100 according to a first embodiment of the present disclosure. The remote operation support device 100 is used when temporarily controlling an autonomous operation system 32, such as an autonomous driving vehicle, from a remote location, and includes a control unit 1, a state detection unit 2, an operation interface 3, a fault occurrence determination unit 4, a current state presentation unit 5, a post-fault state recording unit 6, and a post-fault situation presentation unit 7.

[0017] The control unit 1 receives, via the operation interface 3, operations for remotely controlling the autonomous operating system 32 input by a remote operator via the input device 31, and controls the autonomous operating system 32 based on the operations input by the remote operator.

[0018] Here, a remote operator is a person who, when some kind of failure occurs in the autonomous operating system 32 and autonomous operation stops, remotely operates the autonomous operating system 32 from a remote location in accordance with requests from the autonomous operating system 32 and the remote operation support device 100, thereby restoring the system to autonomous operation again.

[0019] The state detection unit 2 detects surrounding environment information 33 of the autonomous operation system 32 and information about the autonomous operation system 32. Examples of the surrounding environment information 33 of the autonomous operation system 32 include the state of the road surface, the state of the road such as the presence or absence of obstacles on the road, the state of other traffic participants such as the positional relationship with pedestrians and the state of pedestrians, the weather state such as sunny, rainy, cloudy, snowy, and surrounding sounds. In addition, the information may be acquired by a plurality of sensors equipped in the autonomous operation system 32, or may be acquired from an infrastructure system installed on the road.

[0020] The information of the autonomous operation system 32 includes information determined by the autonomous operation system 32, information of the devices and sensors installed in the autonomous operation system 32, and status information inside the vehicle.

[0021] More specifically, the information determined by the autonomous operation system 32 includes the determination result of the autonomous operation system 32. The information of the devices and sensors mounted on the autonomous operation system 32 includes speed, acceleration, engine RPM, horn information, steering information, accelerator information, brake information, turn signal information, position information, etc. The status information inside the vehicle includes status information of objects present inside the vehicle such as passengers and luggage, and environmental information of the vehicle interior space such as sound and temperature.

[0022] The surrounding environment information 33 and the information of the autonomous operation system 32 are transmitted to a fault occurrence determination unit 4 , a current state presentation unit 5 and a post-fault state recording unit 6 .

[0023] The operation interface 3 accepts operations input by a remote operator. The operations input by the remote operator include an operation to identify the cause of the occurrence of a failure, an operation to change the display content of the display device DP of the remote operation support device 100 in order to grasp the status of the autonomous operation system 32, an operation to remotely control the autonomous operation system 32, and the like. The operations input by the remote operator are transmitted to the control unit 1, the current state presentation unit 5, and the post-failure status presentation unit 7.

[0024] The fault occurrence determination unit 4 determines whether or not a fault has occurred in the autonomous operating system 32 based on the information of the autonomous operating system 32 and the surrounding environment information 33 of the autonomous operating system 32 , and transmits the result to the post-fault state recording unit 6 .

[0025] For example, the fault occurrence determination unit 4 determines whether or not a fault has occurred based on the result of the determination of whether or not the autonomous operating system 32 is capable of self-propelling, which is transmitted from the autonomous operating system 32. As a specific example, when the autonomous operating system 32 is running outside the ODD (Operational Design Domain) setting, for example when it is unable to recognize surrounding vehicles and roads due to bad weather caused by heavy rain, the autonomous operating system 32 determines that it cannot run on its own, and the determination result is transmitted to the fault occurrence determination unit 4 via the state detection unit 2, and the fault occurrence determination unit 4 determines that a fault has occurred in the autonomous operating system 32.

[0026] The current state presenting unit 5 provides the latest surrounding environment information 33 of the autonomous operation system 32 transmitted from the state detecting unit 2 and the latest information of the autonomous operation system 32 to the remote operator via the display device DP.

[0027] The post-fault state recording unit 6 records the surrounding environment information 33 of the autonomous operating system 32 and the information of the autonomous operating system 32 acquired by the state detection unit 2 before and after the occurrence of a fault. Here, before and after the occurrence of a fault refers to, for example, a period of about 10 seconds before and after the time of the fault occurrence, and information before the fault occurrence is also recorded in order to grasp the precursor of the fault occurrence. Note that this period before and after the fault occurrence can be varied depending on the cause of the fault occurrence, and the time before the fault occurrence can be about 10 seconds, or the time after the fault occurrence can be longer, and can also be from before the fault occurrence to the present.

[0028] The post-failure status presenting unit 7 presents the information recorded in the post-failure status recording unit 6 to the remote operator via the display device DP in response to a post-failure status information request from the remote operator transmitted from the operation interface 3.

[0029] <Operation> Next, the operation of the remote operation support device 100 will be described with reference to the flowchart shown in Fig. 3. When the remote operation support device 100 is operated, first, the state detection unit 2 acquires the surrounding environment information 33 of the autonomous operation system 32 and information about the autonomous operation system 32 (step S11).

[0030] The state detection unit 2 transmits the acquired surrounding environment information 33 of the autonomous operation system 32 and information about the autonomous operation system 32 to the post-fault state recording unit 6 (step S12).

[0031] The state detection unit 2 transmits the acquired information on the autonomous operation system 32 to the failure occurrence determination unit 4 (step S13).

[0032] The fault occurrence determination unit 4 determines whether a fault has occurred in the autonomous operating system 32 based on the information of the autonomous operating system 32 (step S14). If it is determined that no fault has occurred (No), the operations from step S11 onwards are repeated.

[0033] On the other hand, if it is determined that a fault has occurred in the autonomous operation system 32 (Yes), the process proceeds to step S15.

[0034] In step S15, the failure occurrence determination unit 4 transmits a notification of the occurrence of a failure to the post-failure state recording unit 6.

[0035] The post-failure state recording unit 6 records information obtained from the state detection unit 2 before and after the occurrence of the failure, based on the notification of the failure occurrence sent from the failure occurrence determination unit 4 (step S16).

[0036] As described using step S1 of the flowchart shown in FIG. 1, when the remote operator notices the occurrence of a fault and requests the remote operation support device 100 to present information for understanding the current fault status of the autonomous operating system 32 as described using step S3 of the flowchart shown in FIG. 1, the requested operation is accepted by the operation interface 3 (step S17).

[0037] The operation interface 3 transmits information on the received requested operation to the post-failure situation presentation unit 7 (step S18).

[0038] Based on the information of the requested operation, the post-failure status presentation unit 7 displays the information recorded in the post-failure status recording unit 6 on the display device DP (step S19). At this time, information for assisting in grasping the information of the failure can be added to the data recorded in the post-failure status recording unit 6 and provided.

[0039] For example, in order to grasp surrounding information, points where there are many changes in the surrounding environment information 32 of the autonomous operating system 32 are colored, surrounded by a frame, displaying points of caution in text, alerting points of caution with sound and voice, etc. Points of many changes include blind spots such as sidewalks, crosswalks, and areas near the shadows of buildings, as well as traffic participants other than vehicles that may behave unexpectedly, such as pedestrians.

[0040] An example of adding such information is shown in Fig. 4. Fig. 4 is a diagram showing the display screen of the display device DP, and shows a sidewalk SW and a crosswalk PC as examples of areas with many changing points. Fig. 4 shows an example in which the area of ​​the sidewalk SW is surrounded by a frame FL, and the area of ​​the crosswalk PC is a colored area CL. Also, the sentence CH saying [Watch out for the "crosswalk."] is displayed in the corner of the display screen.

[0041] <Effects> As described above, the remote operation support device 100 of embodiment 1 provides information to assist in understanding information about a failure, in addition to the data recorded in the post-failure state recording unit 6, making it easier to understand information about the surroundings of the autonomous operating system 32 when a failure occurs, and can assist the remote operator in recovering the autonomous operating system.

[0042] In addition, the post-fault state recording unit 6 records the surrounding environment information 33 of the autonomous operating system 32 and information about the autonomous operating system 32 acquired by the state detection unit 2 before and after the occurrence of a fault, and presents the information to the remote operator, so that the cause of the fault can be grasped in a short period of time.

[0043] <Embodiment 2> <Device configuration> 5 is a block diagram showing a configuration of a remote operation support device 200 according to a second embodiment of the present disclosure. The remote operation support device 200 has a configuration in which a playback speed calculation unit 8 is further provided in addition to the configuration of the remote operation support device 100 shown in FIG. 2. In addition to the playback speed calculation unit 8, the remote operation support device 200 differs from the remote operation support device 100 in that the post-fault state recording unit 6 records information acquired from the state detection unit 2 from immediately before the occurrence of the fault to the latest (current) state based on the notification of the fault occurrence transmitted from the fault occurrence determination unit 4, and the recorded information is also transmitted to the playback speed calculation unit 8. In addition, in FIG. 5, the same components as those of the remote operation support device 100 described using FIG. 2 are denoted by the same reference numerals, and duplicated descriptions are omitted.

[0044] The playback speed calculation unit 8 calculates the playback speed of the video on the display device DP when providing the information recorded in the post-failure state recording unit 6 to the remote operator, and adjusts the playback speed of the video on the display device DP via the post-failure situation presentation unit 7. That is, it receives operation information for identifying the cause of the remote operator's failure from the operation interface 3, and calculates the playback speed based on the received operation information.

[0045] More specifically, the time during which the remote operator requests information is obtained from the operation information of the information presentation for the remote operator to grasp the status of the failure, the time from immediately before the occurrence of the failure until the remote operator requests the information is defined as a first time t1, and the time from when the remote operator requests the information and plays back the information to the present is defined as a second time t2, and the playback speed is calculated so that the sum of the first time t1 and the second time t2 is sufficiently smaller than the time t2.

[0046] For example, if time t1 is 4 seconds, and the playback speed is set to double, video for time t1 can be provided in 2 seconds. In this case, assuming that there is no time lag between when the remote operator requests information and when the information is provided, 2 seconds have passed since the information was provided, so time t2 is 2 seconds. If this information for time t2 is continuously provided at double the playback speed, it can be provided in 1 second. Similarly, if that 1 second's worth of information is provided at double the playback speed, it will take 0.5 seconds. By repeating this, the time will be extremely close to the current time, and the cause of the problem can be identified in a short amount of time.

[0047] The playback speed does not have to be constant; for example, the playback speed of the information around the time of the failure is slowed down, and the playback speed is increased as the time approaches the present. For example, the playback speed of the information around the time of the failure is set to 1x (slow), and the playback speed of the information closer to the present is set to 4x (fast). However, this speed can be changed depending on the situation.

[0048] <Operation> Next, the operation of the remote operation support device 200 will be described with reference to the flowchart shown in Fig. 6. Note that steps S11 to S15 and S17 in Fig. 6 are the same as steps S11 to S15 and S17 in the flowchart shown in Fig. 3, and therefore description thereof will be omitted.

[0049] In step S161 following step S15 in FIG. 6, the post-fault state recording unit 6 records information obtained from the state detection unit 2 from just before the fault occurred to the latest (present) information based on the notification of the fault occurrence sent from the fault occurrence determination unit 4.

[0050] When the remote operator notices the occurrence of a failure and requests the remote operation support device 100 to present information for grasping the current status of the failure of the autonomous operation system 32, the operation interface 3 accepts the request (step S17).

[0051] The operation interface 3 transmits information about the requested operation to the post-failure situation presenting unit 7 and the playback speed calculating unit 8 (step S20).

[0052] Based on the information on the requested operation and the recorded information obtained from the post-failure state recording unit 6, the playback speed calculation unit 8 calculates the playback speed of the video on the display device DP when providing the recorded information to the remote operator (step S21).

[0053] Based on the information of the requested operation, the post-failure state presenting unit 7 displays the information recorded by the post-failure state recording unit 6 on the display device DP according to the playback speed calculated by the playback speed calculation unit 8 (step S22).

[0054] At this time, information for assisting in grasping information about the failure is added to the data recorded in the post-failure state recording unit 6 and provided, similarly to the remote operation support device 100 of the first embodiment.

[0055] <Effects> As described above, in the remote operation support device 200 of the second embodiment, the post-failure state recording unit 6 records the information obtained from the state detection unit 2 from immediately before the occurrence of the failure to the latest information based on the notification of the failure transmitted from the failure occurrence determination unit 4, and transmits the recorded information to the playback speed calculation unit 8. The playback speed calculation unit 8 calculates the playback speed of the video on the display device DP when providing the information from immediately before the failure occurrence to the latest information recorded in the post-failure state recording unit 6 to the remote operator, and adjusts the playback speed of the video on the display device DP via the post-failure situation presentation unit 7.

[0056] For this reason, the remote operator can determine whether the cause of the failure is still having an effect. For example, if the cause of the failure is a person running out into the road, information on "where the person is currently located," such as whether the person has already moved far away from the vehicle to a range where driving is not affected, or whether the person has moved into the blind spot of the vehicle within a range where driving is affected and is still there, is important for resuming operation of the autonomous operation system 32, and can be used to grasp such information.

[0057] <Embodiment 3> <Device configuration> Fig. 7 is a block diagram showing a configuration of a remote operation support device 300 according to a third embodiment of the present disclosure. The remote operation support device 300 has basically the same configuration as the remote operation support device 200 shown in Fig. 5, but in the remote operation support device 300, the failure occurrence determination unit 4 has a function of estimating the cause of the failure occurrence in addition to the function of determining the occurrence of a failure in the autonomous operation system 32, and transmits the determination result and the estimated cause of the failure occurrence to the post-failure state recording unit 6. In addition, in Fig. 7, the same components as those of the remote operation support device 200 described using Fig. 5 are assigned the same reference numerals, and duplicated descriptions will be omitted.

[0058] The fault occurrence determination unit 4 determines whether or not a fault has occurred in the autonomous operating system 32 based on information about the autonomous operating system 32 and information about the surrounding environment of the autonomous operating system 32, and estimates the cause of the fault, transmits the determination result to the post-fault state recording unit 6, and transmits the estimated cause of the fault to the playback speed calculation unit 8.

[0059] For example, the fault occurrence determination unit 4 determines whether or not a fault has occurred based on the result of the determination as to whether or not the autonomous operating system 32 is capable of self-propelling, which is transmitted from the autonomous operating system 32. As a specific example, when the autonomous operating system 32 is running outside the ODD (Operational Design Domain) setting, for example, when the autonomous operating system 32 cannot recognize surrounding vehicles and roads due to bad weather caused by heavy rain, the autonomous operating system 32 determines that it cannot self-propel, and the determination result is transmitted to the fault occurrence determination unit 4 via the state detection unit 2, and it is determined that a fault has occurred in the autonomous operating system 32.

[0060] To estimate the cause of the failure, the information used when the autonomous operation system 32 determines that it cannot drive on its own, such as the recognition results of white lines on the road, guardrails, road surfaces, etc., the recognition results of other traffic participants, the detection results of obstacles to driving, the status of sensors and equipment, the calculation results of automatic driving route generation, and information from the emergency button installed in the vehicle, are estimated as causes of the failure.

[0061] More specifically, based on the results of the recognition of the white lines on the road, it can be inferred that the cause of the accident was the vehicle's inability to recognize the white lines and therefore being unable to move independently.

[0062] From the recognition results of other traffic participants, such as pedestrians, it can be inferred that the cause of the obstruction was "a pedestrian approaching and preventing the vehicle from moving independently."

[0063] From the detection results of obstacles to travel, it can be inferred that "obstacles on the road" are the cause of the accident.

[0064] Based on information from the emergency button installed inside the vehicle, it can be inferred that the cause of the malfunction was "the emergency button inside the vehicle being pressed, rendering the vehicle unable to move under its own power."

[0065] Based on the status of the sensor and equipment, it is estimated that the cause of the failure is "a sensor failure that made it impossible to obtain information."

[0066] Furthermore, it is also possible to estimate that information that is a value different from normal in the information of the autonomous operation system 32 and the surrounding environment information 33 of the autonomous operation system 32 is the cause of the occurrence of a fault.

[0067] More specifically, when the relative distance between another vehicle and another vehicle becomes different from normal, a malfunction of the other vehicle or an operating error by the driver of the other vehicle may be considered as a cause of the problem, and therefore the presence of the other vehicle can be presumed to be a cause of the problem.

[0068] In addition, when a sensor value, for example the value of an illuminance sensor, becomes different from normal, a malfunction or dirt in the illuminance sensor can be considered as a cause of the fault, and therefore the illuminance sensor can be presumed to be the cause of the fault.

[0069] When the post-fault status presentation unit 7 displays the information recorded in the post-fault status recording unit 6 on the display device DP, it is possible to add information to the data recorded in the post-fault status recording unit 6 to assist in understanding the points related to the cause of the fault.

[0070] In other words, based on the information on the inferred cause of a failure obtained from the failure occurrence determination unit 4, it is also possible to add information related to the inferred result of the cause of a failure, such as coloring the information, surrounding it with a frame, indicating points of caution in text, or calling attention to points of caution with sound and voice.

[0071] An example of adding such information is shown in Fig. 8. Fig. 8 is a diagram showing the display screen of the display device DP, illustrating an example in which "approaching pedestrians" is estimated as a cause. Fig. 8 shows an example in which a pedestrian WM is surrounded by a frame FL and is colored as a region CL. Also, a sentence CH saying "Approaching pedestrians has been estimated as a cause" is displayed in the corner of the display screen, and further shows an example in which attention is drawn to the sound VO saying "Approaching pedestrians has been estimated as a cause" from the speaker SP.

[0072] Furthermore, in order to identify areas related to the cause of a failure, information on the location of the failure and information on the causes of past failures can be used to add information such as coloring the information on the estimated causes of past failures, surrounding it with a frame, or alerting users to areas with sound or voice.

[0073] An example of such addition of information is shown in Fig. 9. Fig. 9 is a diagram showing the display screen of the display device DP, illustrating an example in which a "pedestrian crossing" is estimated as a factor. Fig. 9 shows an example in which the area of ​​the pedestrian crossing PC is surrounded by a frame FL and is colored as a colored area CL. Also, a sentence CH saying "A "pedestrian crossing" has been estimated as a factor" is displayed in the corner of the display screen, and further shows an example in which attention is drawn by a voice VO saying "A "pedestrian crossing" has been estimated as a factor" from the speaker SP.

[0074] Here, in order to present information on causes of past faults as shown in Fig. 9, the following conditions are necessary: ​​location information and facility information, etc., must be input in advance to the post-fault state recording unit 6, past information must be recorded in the post-fault situation presentation unit 7, and the time period and other situations such as the relative positions of other traffic participants must be similar.

[0075] Further, examples of adding information to identify locations related to the cause of the problem include, when driving to or from an intersection near an elementary school, surrounding with a frame the blind spot on the road in the direction of the elementary school in the video captured by the in-vehicle camera, especially the lower part of the screen, and, when driving near a hospital, displaying a message to watch out for emergency vehicles, or alerting with sound and voice.

[0076] The playback speed calculation unit 8 calculates the playback speed using information on the cause of the failure obtained from the failure occurrence determination unit 4. In this case, the playback speed is calculated to be slowed down while the cause of the failure is displayed. For example, if the cause of the failure is the detection of a travel obstruction, the playback speed is slowed down only during the time period when the travel obstruction is captured in the vehicle-mounted camera video, and the playback speed is increased during other time periods.

[0077] <Operation> Next, the operation of the remote operation support device 300 will be described with reference to the flowchart shown in Fig. 10. Note that steps S11 to S14 in Fig. 10 are the same as steps S11 to S14 in the flowchart shown in Fig. 3, and therefore description thereof will be omitted.

[0078] If it is determined in step S14 that a fault has occurred in the autonomous operation system 32 (Yes), the process proceeds to step S31, where the fault occurrence determination unit 4 estimates the cause of the fault occurrence.

[0079] The fault occurrence determination unit 4 transmits a notification of the fault occurrence to the post-fault state recording unit 6, and transmits an estimate of the cause of the fault occurrence to the replay speed calculation unit 8 (step S32).

[0080] The post-failure state recording unit 6 records the information acquired from the state detection unit 2 from immediately before the failure occurrence to the latest information based on the notification of the failure occurrence transmitted from the failure occurrence determination unit 4 (step S33).

[0081] When the remote operator notices the occurrence of a fault and requests the remote operation support device 300 to present information for grasping the current fault condition of the autonomous operation system 32, the operation interface 3 accepts the request (step S34).

[0082] The operation interface 3 transmits information about the requested operation to the post-failure situation presenting unit 7 and the playback speed calculating unit 8 (step S35).

[0083] Based on the information on the estimated cause of the fault occurrence and the recorded information obtained from the post-fault state recording unit 6, the playback speed calculation unit 8 calculates the playback speed in the display device DP when providing the recorded information to the remote operator (step S36).

[0084] The post-failure state presenting section 7 displays the information recorded by the post-failure state recording section 6 on the display device DP according to the playback speed calculated by the playback speed calculation section 8 based on the information of the requested operation.

[0085] <Effects> As described above, in the remote operation assistance device 300 of embodiment 3, the fault occurrence determination unit 4 determines whether a fault has occurred in the autonomous operating system 32 based on the information about the autonomous operating system 32 and the surrounding environment information 33 about the autonomous operating system 32, and estimates the cause of the fault, transmits the determination result to the post-fault state recording unit 6, and transmits the estimated cause of the fault to the playback speed calculation unit 8.

[0086] The playback speed calculation unit 8 calculates the playback speed on the display device DP when providing the recorded information to the remote operator, based on the information on the estimated cause of the fault obtained from the fault occurrence determination unit 4 and the recorded information obtained from the post-fault state recording unit 6.

[0087] This makes it easier to grasp the cause of the failure, and helps the remote operator avoid and eliminate the cause of the failure when restoring the autonomous operation system 32.

[0088] In addition, the remote operation support device 300 of embodiment 3 provides information to assist in identifying the points related to the cause of the failure, added to the data recorded in the post-failure state recording unit 6, making it easier to identify the cause of the failure and assisting the remote operator in recovering the autonomous operating system 32.

[0089] <Fourth embodiment> <Device configuration> FIG. 11 is a block diagram showing a configuration of a remote operation support device 400 according to a fourth embodiment of the present disclosure. The remote operation support device 400 further includes a remote operator state detection unit 9 in addition to the configuration of the remote operation support device 200 shown in FIG. 5, and the remote operator state detection unit 9 acquires state information of the remote operator, such as whether the remote operator pays attention to information presented by the remote operation support device 400. The playback speed calculation unit 8 changes the playback speed using the state information of the remote operator acquired by the remote operator state detection unit 9. In addition, in FIG. 11, the same components as those of the remote operation support device 200 described using FIG. 5 are assigned the same reference numerals, and duplicated explanations are omitted.

[0090] The remote operator state detection unit 9 acquires the line of sight of the remote operator RM based on information from a line of sight detection device such as an eye tracker and a camera installed in a remote operation room where the remote operation support device 400 is installed, and detects whether the remote operator RM is looking at the screen of the display device DP presenting information. Technology for detecting line of sight is well known, and is installed in vehicles and the like to monitor whether the driver is dozing.

[0091] Also, using devices such as an indoor camera, a microphone, and a motion sensor, the voice and body movements of the remote operator RM are acquired, and it is detected whether the remote operator is performing any other actions, such as calling, speaking, writing, etc., other than operating the remote operation support device 400 while looking at the screen of the display device DP. The technology for acquiring the voice and body movements of the remote operator RM is publicly known, and is installed in vehicles, etc., to monitor the driving state of the driver.

[0092] The remote operator state detection unit 9 judges whether the remote operator RM is making a call or talking based on the voice information and the operation status of the remote operation system. For example, if there is an utterance while using a call function, which is a function of the remote operation system, it is judged that the remote operator RM is operating the remote operation support device, i.e., making a call using the remote operation support device 400. In other circumstances, it is judged that the remote operator RM is performing an action other than operating the remote operation support device.

[0093] With regard to actions such as writing, it is determined whether or not the user is holding any device or tool other than the remote operation support device 400, and whether or not the user is performing an expected action that has been input in advance in the operation of the remote operation support device 400. For example, any action other than mouse operation, keyboard operation, or viewing the screen of the display device DP is determined to be not an expected action.

[0094] The status information of the remote operator RM acquired by the remote operator status detection unit 9 is transmitted to the playback speed calculation unit 8, and the playback speed of the video is changed depending on whether the remote operator is paying attention to the information presented by the remote operation support device 400. For example, when the remote operator RM is not paying attention to the screen of the display device DP when information is presented by the remote operation support device 400, the playback speed is slowed down. Also, when the remote operator RM's line of sight is on the screen of the display device DP but he / she is performing another action, for example, a call such as a telephone call, a conversation with another person, writing, etc., the playback speed of the video is slowed down, and in other cases the playback speed is increased.

[0095] <Operation> Next, the operation of the remote operation support device 400 will be described with reference to the flowchart shown in Fig. 12. The flowchart shown in Fig. 12 shows steps following the flowchart of the remote operation support device 200 of the second embodiment shown in Fig. 6, and the flowchart shown in Fig. 6 is also the same in the operation of the remote operation support device 400, with steps S41 and S42 in Fig. 12 being added after step S22 in Fig. 6 in the operation of the remote operation support device 400.

[0096] In step S22 of FIG. 6, the post-fault situation presentation unit 7 displays the information recorded in the post-fault state recording unit 6 on the display device DP according to the playback speed calculated by the playback speed calculation unit 8 based on the information of the requested operation. Then, as shown in step S41 of FIG. 12, the remote operator state detection unit 9 acquires state information of the remote operator RM and transmits it to the playback speed calculation unit 8.

[0097] The playback speed calculation unit 8 changes the playback speed of the display device DP when providing the recorded information to the remote operator RM, based on the state information of the remote operator RM and the recorded information obtained from the post-failure state recording unit 6 (step S42).

[0098] <Effects> As described above, in the remote operation support device 400 of embodiment 4, the post-fault situation presentation unit 7 displays the information recorded in the post-fault state recording unit 6 on the display device DP according to the playback speed calculated by the playback speed calculation unit 8 based on the information of the requested operation, and then changes the playback speed on the display device DP based on the status information of the remote operator RM acquired by the remote operator status detection unit 9.

[0099] Therefore, if the remote operator RM is not paying attention to the information presented on the display device DP, the playback speed of the video can be slowed down, thereby preventing the remote operator RM from missing the video.

[0100] <Embodiment 5> Fig. 13 is a block diagram showing a configuration of a remote operation support device 500 according to a fifth embodiment of the present disclosure. The remote operation support device 500 has basically the same configuration as the remote operation support device 200 shown in Fig. 5, but in the remote operation support device 500, the playback speed calculation unit 8 calculates the playback speed using information recorded in the post-failure state recording unit 6, which is different from the remote operation support device 200. In addition, the remote operation support device 500 differs from the remote operation support device 200 in that the determination result in the failure occurrence determination unit 4 is transmitted to the post-failure situation presentation unit 7. In addition, in Fig. 13, the same components as those of the remote operation support device 200 described using Fig. 5 are assigned the same reference numerals, and duplicated descriptions are omitted.

[0101] The playback speed calculation unit 8 analyzes the information recorded in the post-fault state recording unit 6, and slows down the playback speed of parts where the information has changed significantly or where there are values ​​that are different from normal. The playback speed of other parts is set to be faster. The analysis method is to predict the normal value range using a time series statistical model constructed from past data, and any value that is significantly outside that range is identified as a part where there was an abnormal value.

[0102] <Operation> Next, the operation of the remote operation support device 500 will be described with reference to the flowchart shown in Fig. 14. Steps S11 to S17 in Fig. 14 are the same as steps S11 to S17 in the flowchart shown in Fig. 6, and therefore description thereof will be omitted.

[0103] In step S17, when the remote operator notices the occurrence of a fault and requests the remote operation support device 500 to present information for grasping the current fault condition of the autonomous operation system 32, the operation interface 3 accepts the request.

[0104] The operation interface 3 transmits information about the requested operation to the post-failure situation presentation unit 7 (step S51).

[0105] Based on the recorded information obtained from the post-fault state recording unit 6, the playback speed calculation unit 8 calculates the playback speed on the display device when providing the recorded information to the remote operator (step S52). At this time, the information recorded in the post-fault state recording unit 6 is analyzed, and the playback speed is slowed down for parts where the information has changed significantly or where there are values ​​that are different from normal.

[0106] Based on the information of the requested operation, the post-failure state presenting unit 7 displays the information recorded by the post-failure state recording unit 6 on the display device DP according to the playback speed calculated by the playback speed calculation unit 8 (step S53).

[0107] <Effects> As described above, in the remote operation support device 500 of the fourth embodiment, the playback speed calculation unit 8 analyzes the information recorded in the post-fault state recording unit 6, and slows down the playback speed of parts where the information has changed significantly or where there are values ​​that are different from normal.

[0108] This makes it easier to grasp the cause of the failure, and helps the remote operator avoid and eliminate the cause of the failure when restoring the autonomous operation system 32.

[0109] <Hardware configuration> Each of the components of the remote operation support devices 100 to 500 according to the first to fifth embodiments described above can be configured using a computer, and is realized by the computer executing a program. That is, the remote operation support devices 100 to 500 are realized by, for example, a processing circuit 60 shown in Fig. 15. A processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor) is applied to the processing circuit 60, and the function of each part is realized by executing a program stored in a storage device.

[0110] Dedicated hardware may be applied to the processing circuit 60. When the processing circuit 60 is dedicated hardware, the processing circuit 60 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of these.

[0111] The functions of the components of the remote operation support devices 100 to 500 can be realized by individual processing circuits, or the functions can be realized collectively by one processing circuit.

[0112] 16 shows a hardware configuration in the case where the processing circuit 60 is configured using a processor. In this case, the functions of each part of the remote operation support devices 100 to 500 are realized by a combination of software, etc. (software, firmware, or software and firmware). The software, etc. are described as a program and stored in the memory 62. The processor 61 functioning as the processing circuit 60 realizes the functions of each part by reading and executing the program stored in the memory 62 (storage device). In other words, it can be said that this program causes a computer to execute the procedure and method of operation of the components of the remote operation support devices 100 to 500.

[0113] Here, the memory 62 may be, for example, a non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), HDD (Hard Disk Drive), magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc) and its drive device, or any storage medium to be used in the future.

[0114] The above describes a configuration in which the functions of the components of the remote operation support devices 100 to 500 are realized by either hardware or software. However, this is not limited to this, and some of the components of the remote operation support devices 100 to 500 can be realized by dedicated hardware, and other components can be realized by software, etc. For example, it is possible to realize the functions of some of the components by the processing circuit 60 as dedicated hardware, and realize the functions of other components by the processing circuit 60 as the processor 61 reading and executing a program stored in the memory 62.

[0115] As described above, the remote operation support devices 100 to 500 can realize the above-mentioned functions by hardware, software, or a combination of these.

[0116] Although the present disclosure has been described in detail, the above description is illustrative in all respects and does not limit the present disclosure. It is understood that countless variations not illustrated can be assumed without departing from the scope of the present disclosure.

[0117] In addition, within the scope of the present disclosure, it is possible to freely combine the various embodiments, and to modify or omit the various embodiments as appropriate.

Claims

1. A remote operation support device that temporarily operates an autonomous operation system from a remote location, an operation interface for receiving input from a remote operator; A state detection unit that detects surrounding environment information of the autonomous operating system and system information of the autonomous operating system; a fault occurrence determination unit that acquires the surrounding environment information and the system information of the autonomous operation system from the state detection unit, determines whether or not a fault has occurred in the autonomous operation system, and outputs the determined information as fault occurrence information relating to the occurrence of a fault; a post-fault state recording unit that acquires the surrounding environment information and the system information of the autonomous operating system from the state detection unit, acquires the fault occurrence information from the fault occurrence determination unit, and records the information as information from at least immediately before the fault occurrence to the present; a post-failure status presentation unit that presents the information recorded by the post-failure status recording unit to the remote operator; a playback speed calculation unit that calculates a playback speed of the information recorded in the post-fault state recording unit in accordance with the operation status of the remote operator received by the operation interface, The post-failure situation presentation unit is a remote operation support device that presents the information recorded by the post-fault state recording unit to the remote operator using the playback speed calculated by the playback speed calculation unit.

2. The operation status of the remote operator is a time during which the remote operator requests the information recorded in the post-fault status recording unit, The playback speed calculation unit a first time period from immediately before the occurrence of the failure until the remote operator requests the information; 2. The remote operation support device according to claim 1, wherein the playback speed is calculated so that, when the time from when the remote operator played back the information from just before the occurrence of the failure to the present is a second time, the sum of the first time and the second time is smaller than the second time.

3. A remote operation support device for temporarily operating an autonomous operating system from a remote location, comprising: an operation interface for receiving input from a remote operator; A state detection unit that detects surrounding environment information of the autonomous operating system and system information of the autonomous operating system; a fault occurrence determination unit that acquires the surrounding environment information and the system information of the autonomous operation system from the state detection unit, determines whether or not a fault has occurred in the autonomous operation system, estimates a cause of the fault occurrence, and outputs the result as fault occurrence information regarding the fault occurrence; a post-fault state recording unit that acquires the surrounding environment information and the system information of the autonomous operating system from the state detection unit, acquires the fault occurrence information from the fault occurrence determination unit, and records the information as information from at least immediately before the fault occurrence to the present; a post-failure status presentation unit that presents the information recorded by the post-failure status recording unit to the remote operator; a reproduction speed calculation unit that calculates a reproduction speed of the information recorded in the post-fault state recording unit by using the cause of the fault occurrence estimated by the fault occurrence determination unit, The post-failure situation presentation unit is A remote operation support device that presents the information recorded by the post-fault state recording unit to the remote operator using the playback speed calculated by the playback speed calculation unit.

4. A remote operation support device for temporarily operating an autonomous operating system from a remote location, comprising: an operation interface for receiving input from a remote operator; A state detection unit that detects surrounding environment information of the autonomous operating system and system information of the autonomous operating system; a fault occurrence determination unit that acquires the surrounding environment information and the system information of the autonomous operation system from the state detection unit, determines whether or not a fault has occurred in the autonomous operation system, and outputs the determined information as fault occurrence information relating to the occurrence of a fault; a post-fault state recording unit that acquires the surrounding environment information and the system information of the autonomous operating system from the state detection unit, acquires the fault occurrence information from the fault occurrence determination unit, and records the information as information from at least immediately before the fault occurrence to the present; a post-failure status presentation unit that presents the information recorded by the post-failure status recording unit to the remote operator; a remote operator state detection unit that acquires and outputs a state of the remote operator; a playback speed calculation unit that calculates a playback speed of the information recorded in the post-fault state recording unit according to the state of the remote operator acquired from the remote operator state detection unit, The post-failure situation presentation unit is A remote operation support device that presents the information recorded by the post-fault state recording unit to the remote operator using the playback speed calculated by the playback speed calculation unit.

5. The post-failure situation presentation unit is The remote operation support device according to claim 1 , further comprising: information for assisting in grasping the fault information added to the information recorded in the post-fault state recording unit and presented.

6. The information for assisting in grasping the failure information is 6. The remote operation support device of claim 5, further comprising one of coloring and framing areas where there are many changes in the surrounding environment information of the autonomous operating system, displaying points of caution in text, and alerting users to points of caution using sound and voice.

7. The post-failure situation presentation unit is 4. The remote operation support device according to claim 3, further comprising: information for assisting in grasping a part related to a cause of the occurrence of the failure, added to the information recorded in the post-failure state recording section and presented.

8. The information to assist in understanding the parts related to the cause of the failure is as follows:

8. The remote operation support device according to claim 7, further comprising one of coloring and framing the inferred result of the cause of the failure, displaying points of caution in text, and calling attention to points of caution by sound and voice.