Remote operation assistance device

Abstract

The present disclosure relates to a remote operation assistance device provided with: an operation interface that receives input from a remote operator; a state detection unit that detects surrounding environment information of an autonomous operating system and information about the autonomous operating system; a failure factor estimation unit that acquires the surrounding environment information of the autonomous operating system and the information about the autonomous operating system from the state detection unit, and estimates a failure factor preventing the autonomous operation of the autonomous operating system; a database that stores remote operation information for restoring the autonomous operating system to autonomous operation; a failure handling unit that acquires remote operation information using the failure factor acquired from the failure factor estimation unit, the surrounding environment information and the information about the autonomous operating system acquired from the state detection unit, and the database, and calculates an operation instruction for remotely operating the autonomous operating system on the basis of the remote operation information; and a control unit that controls the autonomous operating system according to the operation instruction.

Description

Remote operation support device

[0001] The present disclosure relates to a remote operation support device, and more particularly to a remote operation support device that supports recovery work of an autonomous operating system when an event occurs in which the autonomous operating system cannot continue autonomous operation.

[0002] In a system in which, if some kind of failure occurs in an autonomous operating system and causes it to stop operating autonomously, a remote operator at a remote location will determine the cause of the failure and then remotely operate the system to return it to autonomous operation.It is expected that the remote operator will be responsible for multiple autonomous operating systems at the same time.

[0003] In this case, the remote operator is required to instantly grasp the status of multiple different autonomous operating systems, which places a heavy cognitive load on the operator. Furthermore, as autonomous operating systems become more sophisticated, the number and types of sensors installed tend to increase, and the amount of data handled tends to increase. Therefore, it is not easy to understand the surrounding environment information of the autonomous operating system and the status of the autonomous operating system based on this information, and to consider methods to eliminate causes of failures.

[0004] Patent Document 1 discloses a technology that reduces the burden on a remote operator in a system in which the remote operator makes decisions in response to requests from a vehicle by automating some of the decisions rather than having the remote operator make all of the decisions.

[0005] Japanese Patent Application Laid-Open No. 2022-85735

[0006] Patent Document 1 discloses the use of learning data to automate remote assistance, but it only uses predetermined learning data, and if the quality of the learning data is low, it is difficult to automate the tasks of the remote operator.

[0007] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a remote operation support device that automates the tasks of a remote operator.

[0008] The remote operation assistance device according to the present disclosure is a remote operation assistance device that temporarily operates an autonomously operating autonomous operation system from a remote location, and includes an operation interface that accepts input from a remote operator, a state detection unit that detects ambient environment information of the autonomous operation system and information about the autonomous operation system, a failure factor estimation unit that acquires the ambient environment information of the autonomous operation system and the information about the autonomous operation system from the state detection unit and estimates a failure factor that is hindering the autonomous operation of the autonomous operation system, a database in which remote operation information for restoring the autonomous operation system to its autonomous operation is stored, a failure response unit that acquires the remote operation information using the failure factor acquired from the failure factor estimation unit, the ambient environment information and the information about the autonomous operation system acquired from the state detection unit, and the database, and calculates operation instructions for remotely operating the autonomous operation system based on the remote operation information, and a control unit that controls the autonomous operation system in accordance with the operation instructions output from the failure response unit.

[0009] According to the remote operation assistance device of the present disclosure, the cause of the failure can be identified by analyzing the collected information, operation instructions can be calculated based on a database, and the autonomous operating system can be restored to autonomous operation based on the operation instructions, thereby automating the tasks of the remote operator and reducing the burden on the remote operator.

[0010] 1 is a flowchart showing a flow of operations of a remote operator operating a remote operation support device according to the present disclosure. FIG. 1 is a block diagram showing a configuration of a remote operation support device according to a first embodiment of the present disclosure. FIG. 2 is a diagram showing an example of information stored in a correspondence database in the form of a table. FIG. 3 is a flowchart showing operations of a remote operation support device according to the first embodiment of the present disclosure. FIG. 4 is a block diagram showing a configuration of a remote operation support device according to a second embodiment of the present disclosure. FIG. 5 is a flowchart showing operations of a remote operation support device according to the second embodiment of the present disclosure. FIG. 6 is a block diagram showing a configuration of a remote operation support device according to a third embodiment of the present disclosure. FIG. 7 is a flowchart showing operations of a remote operation support device according to the third embodiment of the present disclosure. FIG. 8 is a flowchart showing operations of an intervening operation sharing recording unit. FIG. 9 is a block diagram showing a configuration of a remote operation support device according to a fourth embodiment of the present disclosure. FIG. 10 is a flowchart showing operations of a remote operation support device according to the fourth embodiment of the present disclosure. FIG. 11 is a block diagram showing a configuration of a remote operation support device according to a fifth embodiment of the present disclosure. FIG. 12 is a diagram showing an example of information in an extended correspondence database in the form of a table. FIG. 13 is a diagram showing an example of information in an extended correspondence database in the form of a table. FIG. 14 is a flowchart showing operations of a remote operation support device according to the fifth embodiment of the present disclosure. FIG. 15 is a diagram showing a hardware configuration for realizing the remote operation support devices according to the first to fifth embodiments of the present disclosure. FIG. 16 is a diagram showing a hardware configuration for realizing the remote operation support devices according to the first to fifth embodiments of the present disclosure.

[0011] Introduction FIG. 1 is a flowchart showing the flow of actions of a remote operator operating a remote operation support device according to the present disclosure.

[0012] 1, the autonomous operating system notifies the remote operation support device of the occurrence of a fault. This notification is made, for example, via a monitor device for the autonomous operating system that is provided separately from the remote operation support device.

[0013] The remote operator who has received the notification checks the current state of the corresponding autonomous operating system (step S2), which is also checked via a monitor device or the like for the autonomous operating system.

[0014] Next, the remote operator grasps the details of the current fault in the autonomous operation system via the remote operation support device (step S3).

[0015] Next, the remote operator uses the remote operation support device to consider a means of recovering from the current failure of the autonomous operation system (step S4).

[0016] Finally, the remote operator restores the fault by remotely operating the autonomous operating system based on the restoration means (step S5), thereby completing the series of operations.

[0017] In the following explanation, we will use a self-driving vehicle as an example of an autonomous operating system, but this is not limited to this and examples include automatically moving artificial objects such as flying objects, drones, probes, and mobility robots.

[0018] 2 is a block diagram showing the configuration of a remote operation assistance device 100 according to a first embodiment of the present disclosure. The remote operation assistance device 100 is used when temporarily controlling an autonomous operation system 32, such as an autonomously driven vehicle, from a remote location, and includes a control unit 1, a state detection unit 2, a fault cause estimation unit 4, a fault response unit 6, and a response database (DB) 7.

[0019] The state detection unit 2 detects surrounding environment information 33 of the autonomous operation system 32 and information about the autonomous operation system 32. Examples of surrounding environment information 33 of the autonomous operation system 32 include road information including the state of the road surface, the state of the road such as the presence or absence of obstacles on the road, the positional relationship with pedestrians, the state of other traffic participants such as the state of pedestrians, weather conditions such as sunny, rainy, cloudy, snowy, etc., surrounding sounds, road type, the state of the road surface such as whether the road is paved or not, the presence or absence of pedestrian crossings, the presence or absence of traffic lights, etc. Furthermore, the information may be acquired by multiple sensors equipped in the autonomous operation system 32, or may be information acquired from an infrastructure system installed on the road.

[0020] The information of the autonomous operating system 32 includes information determined by the autonomous operating system 32, information on the devices and sensors installed in the autonomous operating system 32, and status information inside the vehicle.

[0021] More specifically, the information determined by the autonomous operation system 32 includes the determination results of the autonomous operation system 32. Information from devices and sensors mounted on the autonomous operation system 32 includes speed, acceleration, engine RPM, horn information, steering information, accelerator information, brake information, turn signal information, position information, etc. In-vehicle status information includes status information of objects present in the vehicle, such as occupants and luggage, and environmental information about the in-vehicle space, such as sound and temperature.

[0022] The surrounding environment information 33 and the information of the autonomous operation system 32 are transmitted to the failure cause estimation unit 4 and the failure response unit 6 .

[0023] The fault factor estimation unit 4 estimates the factor (fault factor) that is hindering the autonomous operation of the autonomous operating system 32 from the surrounding environment information 33 and information about the autonomous operating system 32, and transmits the estimated fault factor to the fault response unit 6.

[0024] For example, the failure factor estimation unit 4 estimates the failure factor based on the result of a determination as to whether the autonomous operation system 32 is capable of self-propelling, which is transmitted from the autonomous operation system 32. As a specific example, when the autonomous operation system 32 is traveling outside the ODD (Operational Design Domain) setting, for example, when it is unable to recognize surrounding vehicles and roads due to bad weather such as heavy rain, the autonomous operation system 32 determines that it cannot self-propel, and transmits the determination result to the failure factor estimation unit 4 via the state detection unit 2.

[0025] To estimate the cause of the obstruction, the autonomous operating system 32 uses information used when it determines that it cannot drive on its own, such as the recognition results of white lines on the road, guardrails, road surfaces, etc., the recognition results of other traffic participants, the detection results of obstacles to driving, the status of sensors and equipment, the calculation results of automatic driving route generation, and information from the emergency button installed in the vehicle.

[0026] More specifically, based on the results of the recognition of the white lines on the road, it can be inferred that the cause of the problem was that the white lines could not be recognized and the vehicle was unable to move independently.

[0027] From the recognition results of other traffic participants, such as pedestrians, it can be inferred that the obstacle was that the vehicle was unable to move forward due to the approach of a pedestrian.

[0028] From the detection results of obstacles to travel, it can be inferred that "obstacles on the road" are the cause of the obstruction.

[0029] Based on information from the emergency button installed inside the vehicle, it can be inferred that the cause of the failure was "an accident or a passenger problem causing the emergency button inside the vehicle to be pressed, making it unable to move under its own power."

[0030] Based on the status of the sensor and equipment, it is estimated that the cause of the failure is that the sensor has broken down and information can no longer be obtained.

[0031] Furthermore, it is also possible to estimate that information that is different from normal values ​​in the information about the autonomous operating system 32 and the surrounding environment information 33 about the autonomous operating system 32 is a cause of a failure.

[0032] More specifically, if the relative distance to another vehicle is different from normal, the presence of another vehicle can be inferred as an obstruction factor, such as a malfunction of the other vehicle or an operating error by the driver of the other vehicle.

[0033] Furthermore, if a sensor value, for example, the value of an illuminance sensor, becomes different from normal, the illuminance sensor may be broken or dirty, and the illuminance sensor may be considered to be the cause of the problem.

[0034] The correspondence DB7 stores remote operation information that is associated with surrounding environment information 33, information on the autonomous operating system 32, and information on the cause of the failure that is preventing the autonomous operating system 32 from operating autonomously, and that is used to remotely operate the autonomous operating system 32 in order to restore the autonomous operating system 32 to autonomous operation.

[0035] 3 is a table showing an example of information stored in the correspondence DB 7. As shown in FIG. 3, the correspondence DB 7 shows, from top to bottom, "crosswalk" and "fallen tree" as surrounding environment information 33, "approaching pedestrian" and "obstacle detected" as information about the autonomous operation system 32, and shows, from top to bottom, "approaching pedestrian at crosswalk" and "obstacle detected" as obstacle factors hindering autonomous operation. Regarding the surrounding environment information 33 and the information about the autonomous operation system 32, it is also possible to store only information related to the recovery of the autonomous operation system 32, rather than all of the information.

[0036] As remote operation information for remotely operating the autonomous operating system 32, if the cause is "approach of a pedestrian at a crosswalk," a "control instruction to encourage pedestrians to cross" is associated, and if the cause is "detection of an obstacle," a "control instruction to drive in a way that avoids the obstacle" is associated.

[0037] "Control instructions to encourage pedestrians to cross" include controls such as displaying images or text encouraging pedestrians to cross on a display projected onto the windshield, or playing audio from an external speaker encouraging pedestrians to cross, and operations previously performed by a remote operator are digitized.

[0038] The "control instructions for driving to avoid obstacles" are digitized numerical information such as the steering angle, accelerator depression amount, and brake depression amount, which are set in accordance with pre-determined rules based on the position information of the obstacle and the position information of the autonomous operation system 32.

[0039] Here, a remote operator is a person who, when some kind of failure occurs in the autonomous operating system 32 and autonomous operation stops, remotely operates the autonomous operating system 32 from a remote location in accordance with requests from the autonomous operating system 32 and the remote operation support device 100, thereby restoring autonomous operation.

[0040] The failure response unit 6 calculates operation instructions to remotely operate the autonomous operating system 32 to restore the autonomous operating system 32 to autonomous operation based on the failure cause transmitted from the failure cause estimation unit 4, the surrounding environment information 33 and information on the autonomous operating system 32 transmitted from the state detection unit 2, and the remote operation information in the correspondence DB 7.

[0041] That is, the failure response unit 6 compares the acquired information on the failure cause, surrounding environment information 33, and autonomous operating system 32 with the correspondence DB 7, and acquires from the correspondence DB 7 remote operation information associated with the information on the failure cause, surrounding environment information, and autonomous operating system that has been acquired in advance and stored in the correspondence DB 7.

[0042] For example, if the acquired surrounding environment information 33, information from the autonomous operating system 32, and the cause of the obstacle are "fallen tree," "obstacle detected," and "obstacle detected," respectively, then by comparing with the corresponding DB 7, "control instructions to travel while avoiding the obstacle" are acquired as remote control information.

[0043] By using the correspondence DB 7, remote operation information for restoring the autonomous operation system 32 to autonomous operation can be obtained in a short time.

[0044] Then, in order to execute the "control instruction to travel so as to avoid obstacles," the fault response unit 6 calculates, as operation instructions, numerical information necessary for operating the vehicle, such as the steering angle, the amount of accelerator depression, the amount of brake depression, and information on the direction of the turn signal.

[0045] The control unit 1 controls the autonomous operating system 32 in accordance with operation instructions for remotely operating the autonomous operating system 32 transmitted from the failure response unit 6 .

[0046] <Operation> Next, the operation of the remote operation support device 100 will be described with reference to the flowchart shown in Fig. 4. When the remote operation support device 100 is operated, first, the state detection unit 2 acquires the ambient environment information 33 of the autonomous operating system 32 and information about the autonomous operating system 32 (step S11).

[0047] Using the surrounding environment information 33 and information on the autonomous operating system 32 acquired from the state detection unit 2, the fault factor estimation unit 4 estimates the fault factor that is hindering the autonomous operation of the autonomous operating system 32 and transmits it to the fault response unit 6 (step S12).

[0048] The failure response unit 6 uses the cause of the failure, the surrounding environment information 33, and the information on the autonomous operating system 32 to calculate an operation instruction for the autonomous operating system 32 from the response DB 7 and transmits it to the control unit 1 (step S13).

[0049] Based on the operation instruction to the autonomous operating system 32, the control unit 1 remotely controls the autonomous operating system 32 to restore autonomous operation (step S14), and ends the series of processes.

[0050] <Effects> As described above, the remote operation assistance device 100 of embodiment 1 identifies the cause of the failure by having the failure cause estimation unit 4 analyze the collected information, calculates operation instructions for remotely operating the autonomous operating system 32 from the correspondence DB so that the autonomous operating system 32 will return to autonomous operation, and can restore the autonomous operating system 32 to autonomous operation based on the operation instructions, thereby making it possible to automate the tasks of the remote operator and reduce the burden on the remote operator.

[0051] <Second Embodiment> <Device Configuration> Fig. 5 is a block diagram showing the configuration of a remote operation support device 200 according to a second embodiment of the present disclosure. The remote operation support device 200 has a configuration in which a state presenting unit 5, an intervention operation interface 3, the state presenting unit 5, and an intervention operation learning unit 8 are further included in the configuration of the remote operation support device 100 shown in Fig. 2. Note that in the remote operation support device 200, when the failure response unit 6 determines that it is not possible to restore the autonomous operating system 32 to autonomous operation, it displays a response request to the remote operator on the display device DP used by the remote operator.

[0052] The control unit 1 not only follows instructions from the failure response unit 6, but also receives, via the intervention operation interface 3, intervention operation instructions for remotely controlling the autonomous operating system 32 input by the remote operator via the input device 31, and transmits the intervention operation instructions to the intervention operation learning unit 8, and controls the autonomous operating system 32 in accordance with the intervention operation instructions from the remote operator. 5 , the same components as those of the remote operation assistance device 100 described using FIG. 2 are denoted by the same reference numerals, and redundant description will be omitted.

[0053] The state presenting unit 5 provides the remote operator with surrounding environment information 33 of the autonomous operating system 32 transmitted from the state detecting unit 2 and information about the autonomous operating system 32 via the display device DP. The remote operator looks at the display device DP and inputs an intervention operation instruction using the input device 31.

[0054] The intervention operation learning unit 8 learns the remote operation performed by the remote operator to operate the autonomous operation system 32 in order to restore the autonomous operation system 32 to autonomous operation, using the intervention operation instructions sent from the control unit 1 in which the remote operator remotely operates the autonomous operation system 32, the failure factors interfering with the autonomous operation of the autonomous operation system 32 sent from the failure factor estimation unit 4, and the surrounding environment information 33 and information on the autonomous operation system 32 sent from the state detection unit 2. Here, learning the intervention operation means associating the intervention operation instructions when the remote operator remotely operates the autonomous operation system 32 with various newly acquired information and storing them in the correspondence DB 7 as remote operation information, and as a result, the amount of data stored in the correspondence DB 7 can be increased.

[0055] 3 as an example, the "surrounding environment information" stores the surrounding environment information 33 sent from the state detection unit 2 as input information. Note that the information may be processed, such as by object recognition, before being stored.

[0056] "Information about the autonomous operating system" stores information about the autonomous operating system 32 sent from the state detection unit 2 as input information.

[0057] The "fault factor hindering autonomous operation" stores the fault factor hindering the autonomous operation of the autonomous operation system 32 sent from the fault factor estimation unit 4 as input information.

[0058] The "remote operation information for remotely operating the autonomous operating system" is input information that is sent from the control unit 1 and is an intervention operation instruction sent when a remote operator remotely operates the autonomous operating system 32, and this information is generalized and stored.

[0059] For example, if the "obstacle factor interfering with autonomous operation" is an obstacle and the remote operator remotely operates the autonomous operation system 32 to avoid the obstacle, the steering angle, accelerator depression amount, and brake depression amount used in that operation are converted into numerical data according to pre-established rules and stored.

[0060] In this way, by learning the intervention operation, it is possible to increase the amount of data stored in the correspondence DB 7, and it is possible to increase the number of events that the remote operation assistance device 200 can handle.

[0061] The intervention operation learning unit 8 can use the intervention operation instructions when a remote operator remotely operates the autonomous operating system 32 as input information, and can generalize and store this as ``remote operation information for remotely operating the autonomous operating system'' through machine learning using artificial intelligence (AI).

[0062] <Operation> Next, the operation of the remote operation support device 200 will be described with reference to the flowchart shown in Fig. 6. Note that steps S11 to S13 in Fig. 6 are the same as steps S11 to S13 in the flowchart shown in Fig. 4, and therefore description thereof will be omitted.

[0063] In step S15 following step S13 in FIG. 6, the failure response unit 6 checks whether or not a corresponding operation instruction exists in the response DB 7, and if a corresponding operation instruction exists (if Yes), the control unit 1 remotely controls the autonomous operating system 32 based on the instruction to the autonomous operating system 32 to restore autonomous operation (step S16), and the series of processes ends.

[0064] On the other hand, if the corresponding operation instruction does not exist in the response DB 7 (No), the failure response unit 6 transmits a request for failure response to the remote operator via the display device DP (step S17).

[0065] The control unit 1 controls the autonomous operation system 32 based on the intervention operation of the remote operator via the intervention operation interface 3 to restore the autonomous operation (step S18).

[0066] The intervention operation learning unit 8 learns the intervention operation using information on the autonomous operation system 32 based on the intervention operation of the remote operator, and information on the surrounding environment information 33 and the autonomous operation system 32 (step S19).

[0067] The intervening operation learning unit 8 stores the learned intervening operation in the correspondence DB 7 (step S20), and ends the series of processes.

[0068] <Effects> As described above, in the remote operation support device 200 of embodiment 2, when an event occurs that is difficult for the remote operation support device 200 to handle, the remote operator performs an intervening operation, and the intervening operation learning unit 8 learns the intervening operation of the remote operator, so that the number of events that the remote operation support device 200 can handle can be increased, and the automation of the remote operator's tasks can be further ensured.

[0069] <Embodiment 3> <Device Configuration> Figure 7 is a block diagram showing the configuration of a remote operation support device 300 according to embodiment 3 of the present disclosure. The remote operation support device 300 has basically the same configuration as the remote operation support device 200 shown in Figure 5, but in the remote operation support device 300, the intervention operation learning unit 8 transmits to an external intervention operation shared recording unit IO surrounding environment information 33, information on the autonomous operating system 32, information on obstacles preventing the autonomous operation of the autonomous operating system, and intervention operation instructions when a remote operator remotely operates the autonomously operating system. In Figure 7, the same components as those of the remote operation support device 200 described using Figure 5 are denoted by the same reference numerals, and redundant description will be omitted.

[0070] The intervening operation sharing and recording unit IO is provided, for example, in a server computer constituting a cloud environment, and is accessible to the remote operation support device 300 and a system other than the remote operation support device 300, so that information transmitted from the intervening operation learning unit 8 can be shared with another system, such as another remote operation support device. The other system may include, for example, a simulator. The simulator can reproduce situations requiring an intervening operation, and can be used for remote operator learning and for experienced remote operators to provide corrections and guidance.

[0071] <Operation> Next, the operation of the remote operation support device 300 will be described with reference to the flowchart shown in Fig. 8. Note that steps S11 to S13 and S15 to S20 in Fig. 8 are the same as steps S11 to S13 and S15 to S20 in the flowchart shown in Fig. 6, and therefore description thereof will be omitted.

[0072] In step S21 following step S20 in FIG. 8, the information acquired by the intervening operation learning unit 8 is transmitted to the intervening operation sharing recording unit IO, and the series of processes is completed.

[0073] 9 is a flowchart showing the operation of the intervening operation shared recording unit 10. The intervening operation shared recording unit 10 receives a request to acquire information recorded in the intervening operation shared recording unit 10 from another system (step S31).

[0074] In response to a request from another system, the intervening operation sharing recording unit IO transmits the information recorded in the intervening operation sharing recording unit IO (step S32). The intervening operation sharing recording unit IO is always in operation, and repeats the processes of steps S31 and S32.

[0075] <Effects> As described above, the remote operation assistance device 300 of the third embodiment is configured so that the information acquired by the intervening operation learning unit 8 is transmitted to the intervening operation shared recording unit IO, and is recorded in the intervening operation shared recording unit IO, thereby enabling another system to use the recorded information. Therefore, the recorded information can be used, for example, to reproduce a situation requiring an intervening operation in a simulator, and can be used for learning by a remote operator, correction guidance by an experienced remote operator, and the like.

[0076] <Fourth Embodiment> <Device Configuration> Fig. 10 is a block diagram showing the configuration of a remote operation assistance device 400 according to a fourth embodiment of the present disclosure. The remote operation assistance device 400 has basically the same configuration as the remote operation assistance device 300 shown in Fig. 7 , but differs in that, in response to a request from the remote operation assistance device 400, the intervening operation sharing recording unit IO has a function of transmitting information about an intervening operation obtained from a system other than the remote operation assistance device 400, for example, from another remote operation assistance device operated by another remote operator, to the intervening operation learning unit 8, and the intervening operation learning unit 8 updates the correspondence DB 7 using the transmitted information.

[0077] The intervention operation learning unit 8 updates the correspondence DB7 based on external information transmitted from the intervention operation sharing recording unit IO, i.e., information on the surrounding environment of another autonomous operating system acquired by another remote operation support device, information on the other autonomous operating system, information on obstacle factors that are hindering the autonomous operation of the other autonomous operating system, and intervention operation instructions when another remote operator remotely operates another autonomous operating system.

[0078] In this way, by updating the correspondence DB 7 using not only information obtained by a single remote operation support device 400 but also external information obtained by another remote operation support device, the amount of information in the correspondence DB 7 increases. Also, since another remote support device is responsible for a different autonomous operation system, the surrounding environment and the cases of failure that occur are different for each autonomous operation system that it is responsible for, and therefore the types of information become more abundant, the types of failures that can be handled also increase, and the correspondence DB 7 can be further enriched.

[0079] Furthermore, since the number of remote operators in a single remote operation support device 400 is limited, learning how to make decisions regarding intervention operations is affected by the proficiency level of the remote operator, etc. However, by sharing information between systems as in the remote operation support device 400, it is possible to reduce the difference in the quality of learning in the intervention operation learning unit 8.

[0080] <Operation> Next, the operation of the remote operation support device 400 will be described using the flowchart shown in Fig. 11. Note that the operation of the remote operation support device 400 is basically the same as that of the remote operation support device 300 described using the flowchart in Fig. 8, and therefore a duplicated description will be omitted. The flowchart shown in Fig. 11 shows a flow in which the remote operation support device 400 requests information acquired from another system and recorded in the intervening operation sharing recording unit IO, and the intervening operation learning unit 8 uses the information to update the correspondence DB 7.

[0081] First, the intervening operation learning unit 8 of the remote operation support device 400 requests the recorded information acquired from another system from the intervening operation sharing recording unit IO (step S41).

[0082] The intervening operation learning unit 8 acquires the information transmitted from the intervening operation sharing recording unit IO (step S42).

[0083] The intervening operation learning unit 8 updates the correspondence DB 7 using the information transmitted from the intervening operation sharing recording unit IO (step S43), and ends the series of processes.

[0084] In this way, in the remote operation support device 400, the correspondence DB7 is updated using information obtained from another system by the intervention operation sharing recording unit IO, thereby further enriching the correspondence DB7 and reducing the difference in the quality of learning in the intervention operation learning unit 8.

[0085] <Fifth Embodiment> <Device Configuration> Fig. 12 is a block diagram showing the configuration of a remote operation support device 500 according to a fifth embodiment of the present disclosure. The remote operation support device 500 has a configuration in which, in addition to the configuration of the remote operation support device 400 shown in Fig. 10, a remote operator state detection unit 9 is added, which acquires the state of the remote operator RM to know where the remote operator RM is looking and what information the remote operator RM is using to make a decision, and transmits the acquired state to the failure factor estimation unit 4 and the intervention operation learning unit 8. In addition, the intervention operation interface 3 has a function of acquiring analysis information input by the remote operator RM via the input device 31 and transmitting the analysis information to the intervention operation learning unit 8. In Fig. 12, the same components as those of the remote operation support device 400 described using Fig. 10 are denoted by the same reference numerals, and redundant description will be omitted.

[0086] The remote operator state detection unit 9 acquires the gaze of the remote operator RM based on information from a gaze detection device such as an eye tracker and a camera installed in the remote operation room where the remote operation support device 500 is installed, and detects where the remote operator RM is looking on the screen of the display device DP that displays information. Technology for detecting gaze is well known, and is installed in vehicles, etc. to monitor whether the driver is drowsy.

[0087] The intervening operation learning unit 8 uses the state of the remote operator RM transmitted from the remote operator state detection unit 9 and the analysis information of the remote operator RM transmitted from the intervening operation interface 3 to learn the intervening operation.

[0088] That is, the status information of the remote operator RM acquired by the remote operator status detection unit 9 is transmitted to the fault factor estimation unit 4 and used to increase the certainty of the fault factor that is hindering the autonomous operation of the autonomous operation system 32.

[0089] For example, if "a bicycle suddenly appears" and "a pedestrian approaches" are raised as possible obstacle causes, and if the remote operator RM is paying more attention to the "bicycle," the certainty of "a bicycle suddenly appears" will be increased.

[0090] This certainty information is stored in the correspondence DB 7 and expands the information in the correspondence DB 7. FIG. 13 is a diagram showing an example of the expanded information in the correspondence DB 7 in the form of a table. As shown in FIG. 13, the correspondence DB 7 shows, from top to bottom, "crosswalk," "crosswalk," and "fallen tree" as surrounding environment information 33, and, from top to bottom, "approaching pedestrian," "approaching pedestrian," and "obstacle detected" as information on the autonomous operation system 32, and, from top to bottom, shows "approaching pedestrian at crosswalk," "bicycle jumping out," and "obstacle detected" as obstacle factors interfering with autonomous operation. Then, "low," "high," and "medium" are shown as the certainty levels of the "obstacle factors interfering with autonomous operation."

[0091] As remote operation information for remotely operating the autonomous operating system 32, if the cause is "approach of a pedestrian at a crosswalk," "start off while driving slowly" is associated; if the cause is "a bicycle suddenly appearing," "start off while driving slowly" is associated; and if the cause is "detection of an obstacle," "intervention operation instruction to drive in a way that avoids the obstacle" is associated.

[0092] Of the obstacle factors interfering with autonomous operation, the certainty level of "approaching pedestrians at a crosswalk" is set to "low," while the certainty level of "bicycles running out in front of the vehicle" is set to "high." This is because the gaze information of the remote operator RM in the state information of the remote operator RM is "gazing at the bicycle," and the certainty level of the candidate obstacle factors interfering with autonomous operation that matches the information that the remote operator RM was gazing at is set to "high."

[0093] In this way, the intervention operation learning unit 8 sets a degree of certainty for each obstacle that is hindering autonomous operation based on the status information of the remote operator RM, and learns that the obstacle with the higher degree of certainty is the obstacle that is hindering autonomous operation, thereby making it possible to further ensure the automation of the remote operator's tasks.

[0094] Furthermore, the analysis information of the remote operator RM sent from the intervention operation interface 3 is sent to the intervention operation learning unit 8 and used for learning the intervention operation.

[0095] That is, the analysis information of the remote operator RM is information on the review of the intervention operation created by the remote operator RM, and includes the basis for the judgment, concerns, questions, alternatives, and the like.

[0096] An example of a basis for a decision would be if you input "I see an obstacle ahead," an example of a concern would be if you input "I judged I could pass the obstacle this time, but it may have been a close call," and an example of an alternative would be if you input "I'll change my driving route."

[0097] The intervention operation learning unit 8 uses the input grounds for judgment to set the degree of certainty of the "obstacle factor that is hindering autonomous operation." Furthermore, using the input concerns or questions, the information in the correspondence DB 7 can be expanded by adding concerns about the "remote operation information for remotely operating the autonomous operation system" and alternatives to the "remote operation information for remotely operating the autonomous operation system," in addition to the "remote operation information for remotely operating the autonomous operation system."

[0098] 14 is a diagram showing an example of information in the expanded correspondence DB 7 in the form of a table. As shown in FIG. 14, the correspondence DB 7 shows, from top to bottom, "crosswalk" and "fallen tree" as surrounding environment information 33, "approaching pedestrian" and "detected obstacle" as information about the autonomous operation system 32, and shows, from top to bottom, "approaching pedestrian at crosswalk" and "detected obstacle" as obstacle factors interfering with autonomous operation. Furthermore, "high" and "medium" are shown as the certainty factors of the "obstacle factors interfering with autonomous operation." This is because the "obstacle factor interfering with autonomous operation" is "detected obstacle," which is the same as the basis for the judgment, and therefore the certainty factor of "approaching pedestrian at crosswalk" is set to "high."

[0099] As remote operation information for remotely operating the autonomous operating system 32, if the cause is "approach of a pedestrian at a crosswalk," a "control instruction to encourage pedestrians to cross" is associated, and if the cause is "detection of an obstacle," a "control instruction to drive in a way that avoids the obstacle" is associated.

[0100] Additionally, as a concern for the added "remote operation information for remotely operating the autonomous operation system," the "control instructions to encourage pedestrians to cross" was marked as "no concern," while the "control instructions to drive to avoid obstacles" was marked as "concern." This is because the concern was entered as "I judged that I could pass by the obstacle this time, but it may have been a close call," so the concern for "control instructions to drive to avoid obstacles" was marked as "concern."

[0101] Additionally, "Change driving route" is shown in "Control instructions for driving to avoid obstacles" as an alternative to "Remote operation information for remotely operating the autonomous operating system." This is because "Change driving route" was input as an alternative, and therefore "Change driving route" is set as an alternative to "Remote operation information for remotely operating the autonomous operating system."

[0102] In this way, the intervention operation learning unit 8 sets a degree of certainty for each obstacle that is hindering autonomous operation based on the analysis information of the remote operator RM, and learns that the obstacle with the higher degree of certainty is the obstacle that is hindering autonomous operation, thereby making it possible to further ensure the automation of the remote operator's tasks.

[0103] <Operation> Next, the operation of the remote operation support device 500 will be described with reference to the flowchart shown in Fig. 15. Note that steps S11 to S13 and S15 to S17 in Fig. 15 are the same as steps S11 to S13 and S15 to S17 in the flowchart shown in Fig. 6, and therefore description thereof will be omitted.

[0104] In step S22 following step S17 in FIG. 15, the remote operator state detection unit 9 acquires state information of the remote operator RM.

[0105] The control unit 1 controls the autonomous operation system 32 based on the intervention operation of the remote operator via the intervention operation interface 3 to restore the autonomous operation (step S23).

[0106] The intervening operation learning unit 8 acquires the analysis information input by the remote operator RM via the intervening operation interface 3 (step S24).

[0107] The intervention operation learning unit 8 learns the intervention operation using control information of the autonomous operating system 32 based on the intervention operation of the remote operator, information on the surrounding environment 33 and the autonomous operating system 32, status information of the remote operator RM during operation, and analysis information of the remote operator RM (step S25).

[0108] The intervening operation learning unit 8 stores the learned intervening operation in the correspondence DB 7 (step S26), and ends the series of processes.

[0109] In this way, in the remote operation assistance device 500, the intervention operation learning unit 8 uses the inputted basis for judgment to set the degree of certainty of the "obstacle factors that are hindering autonomous operation," and also uses the inputted concerns and questions to add concerns about the "remote operation information for remotely operating the autonomous operating system" and alternatives to the "remote operation information for remotely operating the autonomous operating system," thereby expanding the information in the correspondence DB 7 and further ensuring the automation of the remote operator's tasks.

[0110] <Hardware Configuration> Each of the components of the remote operation support devices 100 to 500 according to the first to fifth embodiments described above can be configured using a computer and is realized by the computer executing a program. That is, the remote operation support devices 100 to 500 are realized by, for example, a processing circuit 60 shown in FIG. 16. A processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor) is applied to the processing circuit 60, and the functions of each part are realized by executing a program stored in a storage device.

[0111] Dedicated hardware may be applied to the processing circuitry 60. When the processing circuitry 60 is dedicated hardware, the processing circuitry 60 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0112] The functions of the components of the remote operation support devices 100 to 500 can be realized by individual processing circuits, or the functions can be realized collectively by a single processing circuit.

[0113] 17 shows a hardware configuration in the case where the processing circuitry 60 is configured using a processor. In this case, the functions of each unit of the remote operation support devices 100 to 500 are realized by a combination of software, etc. (software, firmware, or software and firmware). The software, etc. is written as a program and stored in memory 62. The processor 61 functioning as the processing circuitry 60 realizes the functions of each unit by reading and executing the program stored in memory 62 (storage device). In other words, it can be said that this program causes a computer to execute the procedures and methods of operation of the components of the remote operation support devices 100 to 500.

[0114] Here, the memory 62 may be, for example, a non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), HDD (Hard Disk Drive), magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc) and its drive device, or any storage medium that will be used in the future.

[0115] The above describes a configuration in which the functions of each component of the remote operation support devices 100 to 500 are realized either by hardware or software, etc. However, this is not limited to this, and some of the components of the remote operation support devices 100 to 500 can be realized by dedicated hardware, and other components can be realized by software, etc. For example, some of the components can be realized by the processing circuit 60 as dedicated hardware, and other components can be realized by the processing circuit 60 as the processor 61 reading and executing a program stored in the memory 62.

[0116] As described above, the remote operation support devices 100 to 500 can realize the above-mentioned functions by hardware, software, or a combination of these.

[0117] Although the present disclosure has been described in detail, the above description is illustrative in all respects and does not limit the present disclosure to these. It is understood that countless variations not illustrated can be envisioned without departing from the scope of the present disclosure.

[0118] It should be noted that, within the scope of the present disclosure, the embodiments can be freely combined, and the embodiments can be modified or omitted as appropriate.

Claims

1. A remote operation support device that temporarily operates an autonomous operation system from a remote location, an operation interface for receiving input from a remote operator; A state detection unit that detects surrounding environment information of the autonomous operating system and information of the autonomous operating system; an obstacle factor estimation unit that acquires the surrounding environment information of the autonomous operation system and the information of the autonomous operation system from the state detection unit and estimates an obstacle factor that is impeding the autonomous operation of the autonomous operation system; a failure response unit that acquires remote operation information using the failure factor acquired from the failure factor estimation unit, the surrounding environment information and the information of the autonomous operation system acquired from the state detection unit, and a database in which remote operation information for restoring the autonomous operation system to the autonomous operation is stored, and calculates an operation instruction for remotely operating the autonomous operation system based on the remote operation information; a control unit that controls the autonomous operation system in accordance with the operation instruction output from the failure response unit, A remote operation support device, wherein the information of the autonomous operating system is at least one of information regarding the results determined by the autonomous operating system, information regarding whether or not a sensor installed in the autonomous operating system has a malfunction, and information regarding the interior of the vehicle if the autonomous operating system is an autonomously driven vehicle.

2. A remote operation support device for temporarily operating an autonomously operating system from a remote location, comprising: an operation interface for receiving input from a remote operator; A state detection unit that detects surrounding environment information of the autonomous operating system and information of the autonomous operating system; an obstacle factor estimation unit that acquires the surrounding environment information of the autonomous operation system and the information of the autonomous operation system from the state detection unit and estimates an obstacle factor that is impeding the autonomous operation of the autonomous operation system; a failure response unit that acquires remote operation information using the failure factor acquired from the failure factor estimation unit, the surrounding environment information and the information of the autonomous operation system acquired from the state detection unit, and a database in which remote operation information for restoring the autonomous operation system to the autonomous operation is stored, and calculates an operation instruction for remotely operating the autonomous operation system based on the remote operation information; a control unit that controls the autonomous operation system in accordance with the operation instruction output from the failure response unit, The failure response unit is If the autonomous operation system cannot be restored to the autonomous operation, requesting the remote operator to perform an intervention operation to remotely control the autonomous operation system; a state presentation unit that acquires the surrounding environment information and the information of the autonomous operation system from the state detection unit and presents the information to the remote operator; an intervention operation interface that receives an intervention operation instruction by the intervention operation of the remote operator and inputs the instruction to the control unit; an intervention operation learning unit that learns the intervention operation using the intervention operation instruction acquired from the control unit, the failure factor acquired from the failure factor estimation unit, and the surrounding environment information and the information of the autonomous operation system acquired from the state detection unit, and transmits the intervention operation instruction, the failure factor, the surrounding environment information, and the information of the autonomous operation system to an intervention operation sharing and recording unit outside the remote operation assistance device; The control unit is When the intervention operation instruction is input, the autonomous operation system is controlled in accordance with the intervention operation instruction; The intervening operation sharing recording unit includes: A remote operation assistance device that shares the intervention operation instruction, the failure cause, the surrounding environment information, and the information of the autonomous operation system transmitted from the intervention operation learning unit with a system other than the remote operation assistance device.

3. The remote operation information is The fault factor, the surrounding environment information, and the information of the autonomous operation system are associated with each other and stored in the database, the information being acquired and stored in advance; The failure response unit is The remote operation assistance device according to claim 1 , further comprising: a controller for controlling a remote operation of the autonomous system in accordance with a first embodiment of the present invention;

4. The failure response unit is If the autonomous operation system cannot be restored to the autonomous operation, requesting the remote operator to perform an intervention operation to remotely control the autonomous operation system; a state presentation unit that acquires the surrounding environment information and the information of the autonomous operation system from the state detection unit and presents the information to the remote operator; an intervention operation interface that receives an intervention operation instruction by the intervention operation of the remote operator and inputs the instruction to the control unit; An intervention operation learning unit that learns the intervention operation using the intervention operation instruction acquired from the control unit, the failure factor acquired from the failure factor estimation unit, and the surrounding environment information and the information of the autonomous operation system acquired from the state detection unit, The control unit is The remote operation support device according to claim 1 , wherein when the intervention operation instruction is input, the device controls the autonomous operation system in accordance with the intervention operation instruction.

5. The intervention operation learning unit includes:

5. The remote operation assistance device according to claim 4, wherein the intervention operation is learned by associating the intervention operation instruction acquired from the control unit with the fault factor acquired from the fault factor estimation unit, the surrounding environment information acquired from the state detection unit, and the information of the autonomous operation system.

6. The intervention operation learning unit includes: Transmitting the intervention operation instruction, the fault cause, the surrounding environment information, and the information of the autonomous operation system to an intervention operation sharing and recording unit outside the remote operation assistance device; The intervening operation sharing recording unit includes: The remote operation assistance device according to claim 4 , wherein the intervention operation instruction, the fault cause, the surrounding environment information, and the information of the autonomous operating system transmitted from the intervention operation learning unit are shared with a system other than the remote operation assistance device.

7. The intervening operation sharing recording unit includes: In response to a request from the intervention operation learning unit, transmitting external information obtained from the other system to the intervention operation learning unit; The intervention operation learning unit includes: The remote operation support device according to claim 6 , further comprising: a processor configured to update the database based on the external information transmitted from the intervening operation sharing recording unit.

8. A remote operator status detection unit that acquires and outputs a status of the remote operator, The intervention operation interface includes: Accepting analysis information input by the remote operator and inputting it to the intervention operation learning unit; The intervention operation learning unit includes: The remote operation assistance device according to claim 4 , wherein the state of the remote operator output from the remote operator state detection unit and the analysis information input from the intervention operation interface are used to learn the intervention operation.

9. The state of the remote operator is The remote operator's line of sight information is included, The analysis information input by the remote operator is The remote operator's decision on the intervention operation includes information on grounds for the intervention operation, concerns, and alternatives, The intervention operation learning unit includes: setting a certainty level of the obstacle factor based on the line of sight information or the grounds for the judgment; determining whether or not there is a concern about the remote operation information based on the concern; The remote operation assistance device according to claim 8 , wherein the intervening operation is learned by setting the alternative to the remote operation information based on the alternative.