Remote control support device

Abstract

The present disclosure relates to a remote operation assistance device provided with: an operation interface that receives input from a remote operator; a state detection unit that detects surrounding environment information of an autonomous operating system and information about the autonomous operating system; a failure factor estimation unit that acquires the surrounding environment information of the autonomous operating system and the information about the autonomous operating system from the state detection unit, and estimates a failure factor preventing the autonomous operation of the autonomous operating system; a database that stores remote operation information for restoring the autonomous operating system to autonomous operation; a failure handling unit that acquires remote operation information using the failure factor acquired from the failure factor estimation unit, the surrounding environment information and the information about the autonomous operating system acquired from the state detection unit, and the database, and calculates an operation instruction for remotely operating the autonomous operating system on the basis of the remote operation information; and a control unit that controls the autonomous operating system according to the operation instruction.

Description

[Technical Field]

[0001] The present disclosure relates to a remote operation support device, and particularly to a remote operation support device that supports recovery work of an autonomous operation system when an event occurs in which the autonomously operating autonomous operation system cannot continue autonomous operation. [Background Art]

[0002] In a system where, when some failure occurs in an autonomous operation system and autonomous operation is stopped, a remote operator at a remote location determines the cause of the failure, performs remote operation, and restores the system to autonomous operation again, it is assumed that a remote operator handles a plurality of autonomous operation systems at the same time.

[0003] In this case, the remote operator is required to instantaneously grasp the statuses of a plurality of different autonomous operation systems, which imposes a high cognitive load. Furthermore, in increasingly sophisticated autonomous operation systems, the number and types of installed sensors are large, and the amount of data handled tends to increase, so it is not easy to grasp the surrounding environment information of the autonomous operation system and the status of the autonomous operation system based on such information, and to study methods for eliminating the cause of the failure.

[0004] Patent Document 1 discloses a technology for reducing the burden on a remote operator in a system in which the remote operator makes judgments in response to requests from vehicles, by automating some judgments instead of having the remote operator make all judgments. [Prior Art Documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Unexamined Patent Publication No. 2022-85735 [Summary of the Invention] [Problem to be Solved by the Invention]

[0006] Patent Document 1 discloses the use of training data for automating remote support, but it only uses predetermined training data, and if the quality of the training data is low, it is difficult to automate the tasks of the remote operator.

[0007] This disclosure is made to solve the problems described above and aims to provide a remote operation support device that automates the tasks of the remote operator. [Means for solving the problem]

[0008] The remote operation support device according to this disclosure is a remote operation support device for temporarily operating an autonomous operating system from a remote location, comprising: an operation interface that accepts input from a remote operator; a state detection unit that detects the surrounding environment information of the autonomous operating system and the information of the autonomous operating system; a fault factor estimation unit that acquires the surrounding environment information of the autonomous operating system and the information of the autonomous operating system from the state detection unit and estimates fault factors that are hindering the autonomous operation of the autonomous operating system; a fault response unit that acquires the fault factors acquired from the fault factor estimation unit, the surrounding environment information and the information of the autonomous operating system acquired from the state detection unit, and a database storing remote operation information for restoring the autonomous operating system to autonomous operation, and calculates operation instructions for remotely operating the autonomous operating system based on the remote operation information; and a control unit that controls the autonomous operating system in accordance with the operation instructions output from the fault response unit. If the fault response unit is unable to restore the autonomous operating system to autonomous operation, it requests the remote operator to perform an intervention operation to remotely control the autonomous operating system, and the state presentation unit obtains the surrounding environment information and the information of the autonomous operating system from the state detection unit and presents it to the remote operator, and the intervention operation interface receives the intervention operation instruction from the remote operator and inputs it to the control unit, the intervention operation instruction obtained from the control unit, the fault factor obtained from the fault factor estimation unit, the surrounding environment information and the The control unit further comprises an intervention operation learning unit that learns the intervention operation using the information of the autonomous operating system and transmits the intervention operation instruction, the obstacle factor, the surrounding environment information, and the information of the autonomous operating system to an intervention operation sharing recording unit outside the remote control support device, wherein when the intervention operation instruction is input, the control unit controls the autonomous operating system in accordance with the intervention operation instruction, and the intervention operation sharing recording unit shares the intervention operation instruction, the obstacle factor, the surrounding environment information, and the information of the autonomous operating system transmitted from the intervention operation learning unit with a system other than the remote control support device. . [Effects of the Invention]

[0009] According to the remote operation support device described herein, by analyzing the collected information, the cause of the malfunction can be identified, operation instructions can be calculated based on the database, and the autonomous operation system can be restored to autonomous operation based on those operation instructions. This automates the tasks of the remote operator and reduces the burden on the remote operator. [Brief explanation of the drawing]

[0010] [Figure 1] This flowchart shows the flow of actions taken by a remote operator operating the remote control support device related to this disclosure. [Figure 2] This is a block diagram showing the configuration of the remote control support device of Embodiment 1 according to this disclosure. [Figure 3] This diagram shows an example of the information stored in the corresponding database, presented in a table format. [Figure 4] This is a flowchart showing the operation of the remote control support device of Embodiment 1 according to this disclosure. [Figure 5] This is a block diagram showing the configuration of the remote control support device of Embodiment 2 according to this disclosure. [Figure 6] This flowchart shows the operation of the remote control support device of Embodiment 2 according to this disclosure. [Figure 7] This is a block diagram showing the configuration of the remote control support device of Embodiment 3 according to this disclosure. [Figure 8] This is a flowchart showing the operation of the remote control support device of Embodiment 3 according to this disclosure. [Figure 9] This flowchart shows the operation of the intervention operation sharing recording unit. [Figure 10] This is a block diagram showing the configuration of the remote control support device of Embodiment 4 according to this disclosure. [Figure 11] This is a flowchart showing the operation of the remote control support device of Embodiment 4 according to this disclosure. [Figure 12] This is a block diagram showing the configuration of the remote control support device of Embodiment 5 according to this disclosure. [Figure 13] This diagram shows an example of information from the expanded supported database in a table format. [Figure 14] This diagram shows an example of information from the expanded supported database in a table format. [Figure 15] This flowchart shows the operation of the remote control support device of Embodiment 5 according to this disclosure. [Figure 16]It is a diagram showing a hardware configuration that implements the remote operation assistance device according to the first to fifth embodiments of the present disclosure. [Figure 17] It is a diagram showing a hardware configuration that implements the remote operation assistance device according to the first to fifth embodiments of the present disclosure. Mode for Carrying Out the Invention

[0011] <Introduction> FIG. 1 is a flowchart showing an operation flow of a remote operator who operates the remote operation assistance device according to the present disclosure.

[0012] In step S1 shown in FIG. 1, a fault occurrence is notified from an autonomously operating autonomous operation system. This notification is provided, for example, via a monitoring device of the autonomous operation system provided separately from the remote operation assistance device.

[0013] The remote operator who has received the notification checks the current state of the corresponding autonomous operation system (step S2). This check is also performed via a monitoring device or the like of the autonomous operation system.

[0014] Next, the remote operator grasps the content of the current fault of the autonomous operation system via the remote operation assistance device (step S3).

[0015] Next, the remote operator uses the remote operation assistance device to consider recovery means for the current fault of the autonomous operation system (step S4).

[0016] Finally, the remote operator remotely operates the autonomous operation system based on the recovery means to recover from the fault (step S5), and ends the series of operations.

[0017] Note that, in the following description, an autonomous driving vehicle is used as an example of the autonomous operation system, but the present disclosure is not limited thereto, and automatically moving artifacts such as flying vehicles, drones, exploration vehicles, and mobility robots can be cited.

[0018] <Embodiment 1> <Device configuration> Figure 2 is a block diagram showing the configuration of the remote control support device 100 according to Embodiment 1 of the present disclosure. The remote control support device 100 is used to temporarily control an autonomous operating system 32, such as an autonomous vehicle, from a remote location, and includes a control unit 1, a state detection unit 2, a fault cause estimation unit 4, a fault response unit 6, and a response database (DB) 7.

[0019] The state detection unit 2 detects the surrounding environment information 33 of the autonomous operating system 32 and information about the autonomous operating system 32. The surrounding environment information 33 of the autonomous operating system 32 can include road information such as the condition of the road surface, the presence or absence of obstacles on the road, the positional relationship with pedestrians, the status of other road participants such as pedestrians, the weather conditions such as sunny, rainy, cloudy, and snowy, ambient sounds, road type, road surface conditions such as whether it is paved or not, the presence or absence of pedestrian crossings, and the presence or absence of traffic lights. It can also be information acquired by multiple sensors equipped in the autonomous operating system 32, or information acquired from infrastructure systems installed on the road.

[0020] Information from the autonomous operating system 32 includes information determined by the autonomous operating system 32, information from equipment and sensors installed in the autonomous operating system 32, and information about the state inside the vehicle.

[0021] More specifically, the information determined by the autonomous operating system 32 includes the results of the autonomous operating system 32's decisions. Information from the equipment and sensors mounted on the autonomous operating system 32 includes speed, acceleration, engine speed, horn information, steering information, accelerator information, brake information, turn signal information, and position information. Information about the state inside the vehicle includes the state of the occupants, luggage and other objects inside the vehicle, and environmental information of the interior space such as sound and temperature.

[0022] The surrounding environment information 33 and the information from the autonomous operating system 32 are transmitted to the fault cause estimation unit 4 and the fault response unit 6.

[0023] The fault factor estimation unit 4 estimates the factors (fault factors) that are hindering the autonomous operation of the autonomous operation system 32 from the surrounding environment information 33 and the information of the autonomous operation system 32, and transmits the estimated fault factors to the fault response unit 6.

[0024] For example, the fault factor estimation unit 4 estimates fault factors based on the result of determining whether the autonomous operating system 32 is capable of self-propulsion, which is transmitted from the autonomous operating system 32. In a specific example, if the autonomous operating system 32 is operating outside of the ODD (Operational Design Domain) setting, for example, if it cannot recognize surrounding vehicles and the road due to bad weather such as heavy rain, the autonomous operating system 32 will determine that it is not capable of self-propulsion, and the determination result will be transmitted to the fault factor estimation unit 4 via the state detection unit 2.

[0025] To estimate the cause of the failure, information used when the autonomous operating system 32 determined that it could not move on its own is used, such as the recognition results of white lines on the road, guardrails, road surface, etc., the recognition results of other traffic participants, the detection results of obstacles to driving, the status of sensors and equipment, the calculation results of automatic driving route generation, and information from the emergency button installed inside the vehicle are estimated as the cause of the failure.

[0026] More specifically, the results of recognizing the white lines on the track suggest that "the inability to recognize the white lines and thus the inability to move independently" was a contributing factor to the problem.

[0027] Based on the perceptions of other road users, such as pedestrians, it can be inferred that "the approaching pedestrian prevented the vehicle from moving under its own power" was a contributing factor to the obstruction.

[0028] The detection results for obstacles to driving suggest that "obstacles on the road" are the cause of the obstruction.

[0029] Information from the emergency button installed inside the vehicle suggests that the cause of the malfunction was that "the emergency button inside the vehicle was pressed due to an accident or a problem with the occupants, rendering the vehicle unable to move under its own power."

[0030] Based on the status of the sensors and equipment, it is presumed that the cause of the failure was "a sensor malfunction that prevented information from being acquired."

[0031] Furthermore, information from the autonomous operating system 32 and the surrounding environment information 33 of the autonomous operating system 32 that shows values ​​different from those under normal conditions can also be used to estimate potential failure factors.

[0032] More specifically, regarding the relative distance to other vehicles, if the relative distance differs from the normal state, it is possible that malfunctions in other vehicles or operational errors by the drivers of other vehicles are considered to be contributing factors to the problem, and therefore the presence of other vehicles can be estimated as a contributing factor to the problem.

[0033] Furthermore, if sensor values, such as those from an illuminance sensor, become different from normal, malfunctions or contamination of the illuminance sensor can be considered as contributing factors to the problem, allowing the illuminance sensor to be identified as the cause of the malfunction.

[0034] The corresponding DB7 stores information associated with the surrounding environment 33, information about the autonomous operating system 32, and information about fault factors that are preventing the autonomous operation of the autonomous operating system 32, as well as remote operation information used to remotely operate the autonomous operating system 32 in order to restore autonomous operation to the autonomous operating system 32.

[0035] Figure 3 is a table showing an example of the information stored in the corresponding DB7. As shown in Figure 3, the corresponding DB7 shows, from top to bottom, "pedestrian crossing" and "fallen tree" as surrounding environment information 33, "pedestrian approaching" and "obstacle detected" as information for the autonomous operating system 32, and "pedestrian approaching at pedestrian crossing" and "obstacle detected" as obstacles that are hindering autonomous operation. Regarding the surrounding environment information 33 and the information for the autonomous operating system 32, it is also possible to store only the information related to the recovery of the autonomous operating system 32, rather than all information.

[0036] Furthermore, as remote control information for remotely operating the autonomous operating system 32, if the cause is "approaching pedestrians at a crosswalk," a "control instruction to encourage pedestrians to cross" is associated, and if the cause is "detection of an obstacle," a "control instruction to drive in a manner that avoids the obstacle" is associated.

[0037] Examples of "control instructions to encourage pedestrians to cross" include displaying images or text encouraging crossing on a screen projected onto the windshield, or playing audio prompts from external speakers. Data on operations previously performed by remote operators has been compiled.

[0038] The "control instructions to drive while avoiding obstacles" include numerical information such as steering angle, accelerator pedal depression amount, and brake pedal depression amount, which are set according to predetermined rules based on the location information of the obstacles and the location information of the autonomous operating system 32.

[0039] Here, a remote operator is a person who, in response to requests from the autonomous operating system 32 and the remote operation support device 100, remotely operates the autonomous operating system 32 from a remote location to restore autonomous operation when an error occurs in the autonomous operating system 32 and autonomous operation stops.

[0040] The fault response unit 6 calculates operation instructions to remotely operate the autonomous operating system 32 in order to restore the autonomous operating system 32 to autonomous operation, based on the fault cause transmitted from the fault cause estimation unit 4, the surrounding environment information 33 and information on the autonomous operating system 32 transmitted from the state detection unit 2, and the remote operation information from the corresponding DB 7.

[0041] In other words, the fault response unit 6 compares the acquired fault cause, surrounding environment information 33, and autonomous operating system information 32 with the corresponding DB 7, and retrieves remote control information from the corresponding DB 7 that has been previously acquired and stored in the corresponding DB 7 and associated with the fault cause, surrounding environment information, and autonomous operating system information.

[0042] For example, if the acquired surrounding environment information 33, the information of the autonomous operating system 32, and the fault cause are "fallen tree," "obstacle detected," and "obstacle detected," respectively, then by comparing them with the corresponding DB7, a "control instruction to drive while avoiding the obstacle" is obtained as remote operation information.

[0043] By using the corresponding DB7, remote control information for restoring the autonomous operating system 32 to autonomous operation can be obtained in a short time.

[0044] The fault response unit 6 then calculates numerical information necessary for vehicle operation, such as steering angle, accelerator pedal depression amount, brake pedal depression amount, and turn signal direction information, as operation instructions in order to execute a "control instruction to drive while avoiding obstacles."

[0045] The control unit 1 controls the autonomous operating system 32 in accordance with the operation instructions for remotely operating the autonomous operating system 32 transmitted from the fault response unit 6.

[0046] <Operation> Next, the operation of the remote control support device 100 will be explained using the flowchart shown in Figure 4. When the remote control support device 100 is activated, first the state detection unit 2 acquires the surrounding environment information 33 of the autonomous operating system 32 and information about the autonomous operating system 32 (step S11).

[0047] Using the surrounding environment information 33 acquired from the state detection unit 2 and the information from the autonomous operating system 32, the fault factor estimation unit 4 estimates the fault factors that are hindering the autonomous operation of the autonomous operating system 32 and transmits them to the fault response unit 6 (step S12).

[0048] The fault response unit 6 uses the fault cause, surrounding environment information 33, and information from the autonomous operating system 32 to calculate an operation instruction for the autonomous operating system 32 from the response DB 7 and transmit it to the control unit 1 (step S13).

[0049] Based on the operation instructions given to the autonomous operation system 32, the control unit 1 remotely controls the autonomous operation system 32 to restore autonomous operation (step S14), and then terminates the series of processes.

[0050] <Effects> As described above, the remote operation support device 100 of Embodiment 1 identifies the cause of a failure by having the failure cause estimation unit 4 analyze the collected information, calculates operation instructions for remotely operating the autonomous operation system 32 to restore autonomous operation from the corresponding DB, and restores the autonomous operation system 32 to autonomous operation based on said operation instructions. Therefore, the tasks of the remote operator can be automated and the burden on the remote operator can be reduced.

[0051] <Embodiment 2> <Device configuration> Figure 5 is a block diagram showing the configuration of the remote operation support device 200 according to Embodiment 2 of this disclosure. The remote operation support device 200 has a configuration that further includes a state presentation unit 5, an intervention operation interface 3, a state presentation unit 5, and an intervention operation learning unit 8 in addition to the configuration of the remote operation support device 100 shown in Figure 2. In the remote operation support device 200, if the fault response unit 6 determines that it is not possible to restore the autonomous operation system 32 to autonomous operation, it displays a request for assistance to the remote operator on the display device DP used by the remote operator.

[0052] The control unit 1 not only follows instructions from the fault response unit 6, but also receives intervention operation instructions for remotely controlling the autonomous operating system 32 input by the remote operator via the input device 31 through the intervention operation interface 3, transmits them to the intervention operation learning unit 8, and controls the autonomous operating system 32 in accordance with the intervention operation instructions from the remote operator. In addition, in Figure 5, the same reference numerals are used for components identical to those of the remote operation support device 100 described using Figure 2, and redundant explanations are omitted.

[0053] The state presentation unit 5 provides the remote operator with information about the surrounding environment 33 of the autonomous operating system 32 transmitted from the state detection unit 2, and information about the autonomous operating system 32, via the display device DP. The remote operator views the display device DP and inputs intervention operation instructions using the input device 31.

[0054] The intervention operation learning unit 8 learns the remote operation performed by the remote operator on the autonomous operating system 32 in order to restore the autonomous operating system 32 to autonomous operation. This learning unit uses the intervention operation instructions transmitted from the control unit 1 for the remote operator to remotely operate the autonomous operating system 32, the fault factors that are hindering the autonomous operation of the autonomous operating system 32 transmitted from the fault factor estimation unit 4, and the surrounding environment information 33 and autonomous operating system 32 information transmitted from the state detection unit 2. Here, learning an intervention operation means associating the intervention operation instructions when the remote operator remotely operates the autonomous operating system 32 with newly acquired information and storing them in the corresponding DB7 as remote operation information, thereby increasing the amount of data stored in the corresponding DB7.

[0055] Using the corresponding DB7 table shown in Figure 3 as an example, the "Surrounding Environment Information" field stores the surrounding environment information 33 transmitted from the state detection unit 2 as input information. It is also possible to process the information, such as recognizing objects, before storing it.

[0056] The "Autonomous Operating System Information" stores the information of the autonomous operating system 32 transmitted from the state detection unit 2 as input information.

[0057] The "fault factors hindering autonomous operation" field stores the fault factors that are hindering the autonomous operation of the autonomous operation system 32, as input information, transmitted from the fault factor estimation unit 4.

[0058] The "remote operation information for remotely controlling the autonomous operating system" is stored by generalizing the intervention operation instructions transmitted from the control unit 1 when a remote operator remotely controls the autonomous operating system 32.

[0059] For example, if the "obstacle factor preventing autonomous operation" is an obstacle, and a remote operator remotely controls the autonomous operation system 32 to avoid the obstacle, the steering angle, accelerator pedal depression amount, and brake pedal depression amount used in that operation are stored as numerical data according to predetermined rules.

[0060] In this way, by learning intervention operations, the amount of data stored in the corresponding DB7 can be increased, and the number of events that the remote operation support device 200 can handle can be increased.

[0061] The intervention operation learning unit 8 can also take the intervention operation instructions given when a remote operator remotely controls the autonomous operating system 32 as input information, and generalize it into "remote operation information for remotely controlling the autonomous operating system" using machine learning by artificial intelligence (AI) and store it.

[0062] <Operation> Next, the operation of the remote control support device 200 will be explained using the flowchart shown in Figure 6. Note that steps S11 to S13 in Figure 6 are the same as steps S11 to S13 in the flowchart shown in Figure 4, and therefore their explanation will be omitted.

[0063] In step S15, following step S13 in Figure 6, the fault response unit 6 checks whether there is an operation instruction corresponding to the corresponding DB7. If there is an operation instruction corresponding to the corresponding DB7 (Yes), the control unit 1 remotely controls the autonomous operation system 32 based on the instruction to the autonomous operation system 32 to restore autonomous operation (step S16), and the series of processes ends.

[0064] On the other hand, if there is no operation instruction corresponding to the corresponding DB7 (in the case of No), the fault response unit 6 sends a fault response request to the remote operator via the display device DP (step S17).

[0065] The control unit 1 controls the autonomous operation system 32 based on the intervention operation of the remote operator via the intervention operation interface 3 to restore autonomous operation (step S18).

[0066] The intervention operation learning unit 8 learns intervention operations using information from the autonomous operating system 32 based on the remote operator's intervention operations, as well as surrounding environment information 33 and information from the autonomous operating system 32 (step S19).

[0067] The intervention operation learning unit 8 stores the learned intervention operation in the corresponding DB7 (step S20), and the series of processes ends.

[0068] <Effects> As described above, in the remote operation support device 200 of Embodiment 2, if an event occurs that is difficult for the remote operation support device 200 to handle, the remote operator will perform an intervention operation, and the intervention operation learning unit 8 will learn the intervention operation performed by the remote operator. This increases the number of events that the remote operation support device 200 can handle and further ensures the automation of the remote operator's tasks.

[0069] <Embodiment 3> <Device configuration> Figure 7 is a block diagram showing the configuration of the remote control support device 300 according to Embodiment 3 of the present disclosure. The remote control support device 300 has basically the same configuration as the remote control support device 200 shown in Figure 5, but in the remote control support device 300, the intervention operation learning unit 8 transmits surrounding environment information 33, information on the autonomous operating system 32, information on fault factors that are hindering the autonomous operation of the autonomous operating system, and intervention operation instructions when a remote operator remotely operates the autonomously operating system to an externally provided intervention operation sharing record unit IO. In addition, in Figure 7, the same reference numerals are used for components that are the same as those of the remote control support device 200 described using Figure 5, and redundant explanations are omitted.

[0070] The intervention operation sharing record unit IO is, for example, located within a server computer that constitutes a cloud environment, and is accessible to the remote operation support device 300 and other systems, thereby allowing information transmitted from the intervention operation learning unit 8 to be shared with other systems, such as other remote operation support devices. These other systems may include, for example, simulators. The simulator can reproduce situations requiring intervention operations and can be used for remote operator training and for correction and guidance by experienced remote operators.

[0071] <Operation> Next, the operation of the remote control support device 300 will be explained using the flowchart shown in Figure 8. Note that steps S11-S13 and S15-S20 in Figure 8 are the same as steps S11-S13 and S15-S20 in the flowchart shown in Figure 6, so their explanation will be omitted.

[0072] In step S21, following step S20 in Figure 8, the information acquired by the intervention operation learning unit 8 is transmitted to the intervention operation sharing record unit IO, and the series of processes is completed.

[0073] Figure 9 is a flowchart showing the operation of the Intervention Operation Shared Record Unit IO. The Intervention Operation Shared Record Unit IO receives a request from another system to retrieve information recorded in the Intervention Operation Shared Record Unit IO (step S31).

[0074] The Intervention Operation Shared Record Unit IO transmits the information it has recorded in the Intervention Operation Shared Record Unit IO in response to a request from another system (step S32). The Intervention Operation Shared Record Unit IO is always in operation and repeats the processes in steps S31 and S32.

[0075] <Effects> As described above, the remote operation support device 300 of Embodiment 3 has a configuration in which the information acquired by the intervention operation learning unit 8 is transmitted to the intervention operation sharing record unit IO, and the information is recorded by the intervention operation sharing record unit IO, so that the recorded information can be used by another system. For this reason, the recorded information can be used, for example, to reproduce a situation in which an intervention operation is required in a simulator, and to be used for remote operator training and correction guidance by skilled remote operators.

[0076] <Embodiment 4> <Device configuration> Figure 10 is a block diagram showing the configuration of the remote operation support device 400 according to Embodiment 4 of the present disclosure. The remote operation support device 400 has basically the same configuration as the remote operation support device 300 shown in Figure 7, but differs in that, in response to a request from the remote operation support device 400, the intervention operation sharing record unit IO transmits information on intervention operations obtained from a system other than the remote operation support device 400, for example, another remote operation support device operated by another remote operator, to the intervention operation learning unit 8, and the intervention operation learning unit 8 updates the corresponding DB7 using the transmitted information.

[0077] The intervention operation learning unit 8 updates the corresponding DB7 based on external information transmitted from the intervention operation sharing record unit IO, namely, surrounding environment information of another autonomous operating system acquired by another remote operation support device, information of another autonomous operating system, information on fault factors that are hindering the autonomous operation of another autonomous operating system, and intervention operation instructions when another remote operator remotely operates another autonomous operating system.

[0078] In this way, by updating the corresponding DB7 using not only information obtained by a single remote control support device 400, but also external information obtained by another remote control support device, the amount of information in the corresponding DB7 increases. Furthermore, since each remote control support device is responsible for a different autonomous operating system, the surrounding environment and the types of failures that occur differ for each autonomous operating system it is responsible for. This results in a richer variety of information, an increase in the types of failures that can be addressed, and further enrichment of the corresponding DB7.

[0079] Furthermore, since the number of remote operators for a single remote control support device 400 is limited, the learning of judgments in intervention operations is affected by the proficiency level of the remote operators. However, by sharing information between systems, as in the remote control support device 400, it is possible to reduce the differences in the quality of learning in the intervention operation learning unit 8.

[0080] <Operation> Next, the operation of the remote control support device 400 will be explained using the flowchart shown in Figure 11. Note that the operation of the remote control support device 400 is basically the same as that of the remote control support device 300 explained using the flowchart in Figure 8, so redundant explanations will be omitted. The flowchart shown in Figure 11 shows the flow in which the remote control support device 400 requests information acquired from another system, which is recorded in the intervention operation sharing record unit IO, and the intervention operation learning unit 8 updates the corresponding DB7 using that information.

[0081] First, the intervention operation learning unit 8 of the remote operation support device 400 requests the intervention operation sharing recording unit IO to record information acquired from another system (step S41).

[0082] The intervention operation learning unit 8 acquires information transmitted from the intervention operation sharing recording unit IO (step S42).

[0083] The intervention operation learning unit 8 updates the corresponding DB7 using the information transmitted from the intervention operation sharing record unit IO (step S43), and then terminates the series of processes.

[0084] Thus, in the remote operation support device 400, the corresponding DB7 is updated using information acquired from another system via the intervention operation sharing record unit IO, which allows for further enhancement of the corresponding DB7 and reduces differences in the quality of learning in the intervention operation learning unit 8.

[0085] <Embodiment 5> <Device configuration> Figure 12 is a block diagram showing the configuration of the remote operation support device 500 according to Embodiment 5 of the present disclosure. The remote operation support device 500 has the configuration of the remote operation support device 400 shown in Figure 10, plus a remote operator status detection unit 9 that acquires the status of the remote operator RM in order to know where the remote operator RM is looking and what information the remote operator RM is using to make decisions, and transmits this information to the failure factor estimation unit 4 and the intervention operation learning unit 8. In addition, the intervention operation interface 3 has the function of acquiring analysis information input by the remote operator RM via the input device 31 and transmitting it to the intervention operation learning unit 8. Furthermore, in Figure 12, components identical to those of the remote operation support device 400 described using Figure 10 are denoted by the same reference numerals, and redundant explanations are omitted.

[0086] The remote operator status detection unit 9 acquires the gaze of the remote operator RM based on information from an eye-tracking device and camera installed in the remote control room where the remote operation support device 500 is located, and detects where the remote operator RM is looking on the screen of the display device DP that is presenting the information. The technology for detecting gaze is well known and is installed in vehicles and the like to monitor driver drowsiness.

[0087] The intervention operation learning unit 8 uses the remote operator RM's status transmitted from the remote operator status detection unit 9 and the remote operator RM's analysis information transmitted from the intervention operation interface 3 to learn intervention operations.

[0088] In other words, the remote operator status information acquired by the remote operator status detection unit 9 is transmitted to the failure factor estimation unit 4 and used to increase the confidence level of the failure factors that are hindering the autonomous operation of the autonomous operating system 32.

[0089] For example, if "a bicycle suddenly appearing" and "a pedestrian approaching" are identified as potential obstacles, the remote operator (RM) will be more focused on the "bicycle," thus increasing their confidence in the "bicycle suddenly appearing" event.

[0090] This confidence level information is stored in the corresponding DB7 and expands the information in the corresponding DB7. Figure 13 is a table showing an example of the expanded information in the corresponding DB7. As shown in Figure 13, the corresponding DB7 shows, from top to bottom, "pedestrian crossing," "pedestrian crossing," and "fallen tree" as surrounding environment information 33, and from top to bottom, "pedestrian approaching," "pedestrian approaching," and "obstacle detected" as information for the autonomous operation system 32, and from top to bottom, "pedestrian approaching at pedestrian crossing," "bicycle suddenly appearing," and "obstacle detected" as obstacle factors that hinder autonomous operation. The confidence levels for "obstacle factors that hinder autonomous operation" are shown from top to bottom as "low," "high," and "medium."

[0091] Furthermore, as remote control information for remotely operating the autonomous operating system 32, if the cause is "pedestrian approaching at a crosswalk," the command "start moving slowly" is associated; if the cause is "bicycle suddenly appearing," the command "start moving slowly" is associated; and if the cause is "obstacle detection," the command "intervention operation instruction to drive in a manner that avoids the obstacle" is associated.

[0092] Among the obstacles that hinder autonomous operation, "approaching pedestrians at a crosswalk" was given a "low" confidence level, while "a bicycle suddenly appearing" was given a "high" confidence level. This is because the remote operator's (RM) gaze information in the remote operator's (RM) state information indicated "gazing at a bicycle," and therefore, the confidence level for the candidate obstacles that hinder autonomous operation that matched the information the remote operator (RM) was gazing at was set to "high."

[0093] In this way, the intervention operation learning unit 8 sets a confidence level for each obstacle that hinders autonomous operation based on the state information of the remote operator RM, and learns that the obstacle with the higher confidence level is the obstacle that hinders autonomous operation, thereby making the automation of the remote operator's tasks even more reliable.

[0094] Furthermore, the analysis information of the remote operator RM transmitted from the intervention operation interface 3 is sent to the intervention operation learning unit 8 and used for learning the intervention operation.

[0095] In other words, the analysis information from the remote operator (RM) is a reflection on the intervention operation created by the remote operator (RM), and includes the basis for judgment, concerns, questions, and alternatives.

[0096] An example of the basis for the decision is entering "Obstacles confirmed ahead," an example of a concern is entering "I judged that I could pass by the side of the obstacle this time, but it might have been a close call," and an example of an alternative is entering "Change the driving route."

[0097] The intervention operation learning unit 8 sets the confidence level of the "impediment factors hindering autonomous operation" using the basis for the input judgment. In addition, the information in the corresponding DB7 can be expanded by adding concerns about the "remote operation information for remotely operating the autonomous operation system" and alternatives to the "remote operation information for remotely operating the autonomous operation system" using the input concerns or questions.

[0098] Figure 14 is a table showing an example of the information in the extended corresponding DB7. As shown in Figure 14, the corresponding DB7 shows, from top to bottom, "pedestrian crossing" and "fallen tree" as surrounding environment information 33, "pedestrian approaching" and "obstacle detected" as information in the autonomous operation system 32, and "pedestrian approaching at pedestrian crossing" and "obstacle detected" as obstacle factors that hinder autonomous operation. Furthermore, the confidence level for "obstacle factors that hinder autonomous operation" is shown from top to bottom as "high" and "medium". This is because the confidence level for "pedestrian approaching at pedestrian crossing" is set to "high" because "obstacle detected" is an obstacle factor that hinders autonomous operation, and the basis for the judgment is the same.

[0099] Furthermore, as remote control information for remotely operating the autonomous operating system 32, if the cause is "approaching pedestrians at a crosswalk," a "control instruction to encourage pedestrians to cross" is associated, and if the cause is "detection of an obstacle," a "control instruction to drive in a manner that avoids the obstacle" is associated.

[0100] Furthermore, regarding the added "remote control information for remotely operating the autonomous operating system," there are concerns. For "control instructions to prompt pedestrians to cross," the concern is listed as "none," while for "control instructions to drive while avoiding obstacles," the concern is listed as "yes." This is because the input for the concern was, "This time, it was judged that it could pass by the side of the obstacle, but it might have been a close call," which is why the concern for "control instructions to drive while avoiding obstacles" was set to "yes."

[0101] Furthermore, as an alternative to "remote control information for remotely operating the autonomous operating system," "change the driving route" is indicated for "control instructions to drive while avoiding obstacles." This is because "change the driving route" was entered as an alternative, so "change the driving route" is set as the alternative to "remote control information for remotely operating the autonomous operating system."

[0102] In this way, the intervention operation learning unit 8 sets a confidence level for each obstacle that hinders autonomous operation based on the analysis information of the remote operator RM, and learns that the obstacle with the higher confidence level is the obstacle that hinders autonomous operation, thereby making the automation of the remote operator's task even more reliable.

[0103] <Operation> Next, the operation of the remote control support device 500 will be explained using the flowchart shown in Figure 15. Steps S11-S13 and S15-S17 in Figure 15 are the same as steps S11-S13 and S15-S17 in the flowchart shown in Figure 6, so their explanation will be omitted.

[0104] In step S22, following step S17 in Figure 15, the remote operator status detection unit 9 acquires status information of the remote operator RM.

[0105] The control unit 1 controls the autonomous operation system 32 based on the intervention operation of the remote operator via the intervention operation interface 3, and restores it to autonomous operation (step S23).

[0106] The intervention operation learning unit 8 acquires analysis information input by the remote operator RM from the intervention operation interface 3 (step S24).

[0107] The intervention operation learning unit 8 learns the intervention operation using the control information of the autonomous operating system 32 based on the remote operator's intervention operation, the surrounding environment information 33 and the information of the autonomous operating system 32, the state information of the remote operator RM during the operation, and the analysis information of the remote operator RM (step S25).

[0108] The intervention operation learning unit 8 stores the learned intervention operation in the corresponding DB7 (step S26), and then terminates the series of processes.

[0109] Thus, in the remote operation support device 500, the intervention operation learning unit 8 can set the confidence level of "obstacle factors hindering autonomous operation" using the basis for the input judgment, and can also expand the information in the corresponding DB7 by adding concerns and questions about "remote operation information for remotely operating the autonomous operation system" and alternatives to "remote operation information for remotely operating the autonomous operation system" using the input concerns and questions, thereby making the automation of the remote operator's tasks even more reliable.

[0110] <Hardware Configuration> Each component of the remote control support devices 100 to 500 described above in Embodiments 1 to 5 can be configured using a computer and is realized by the computer executing a program. That is, the remote control support devices 100 to 500 are realized, for example, by the processing circuit 60 shown in Figure 16. The processing circuit 60 is fitted with a processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), and the functions of each part are realized by executing a program stored in a memory device.

[0111] Furthermore, dedicated hardware may be applied to the processing circuit 60. If the processing circuit 60 is dedicated hardware, it may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0112] The remote control support devices 100-500 can either have each component function implemented by a separate processing circuit, or they can be implemented together by a single processing circuit.

[0113] Figure 17 also shows the hardware configuration when the processing circuit 60 is configured using a processor. In this case, the functions of each part of the remote control support devices 100 to 500 are realized by a combination of software, firmware, or software and firmware. The software is written as a program and stored in memory 62. The processor 61, which functions as the processing circuit 60, realizes the functions of each part by reading and executing the program stored in memory 62 (storage device). In other words, this program can be said to cause the computer to execute the procedures and methods of operation of the components of the remote control support devices 100 to 500.

[0114] Here, memory 62 can be, for example, non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), HDD (Hard Disk Drive), magnetic disk, flexible disk, optical disk, compact disk, minidisc, DVD (Digital Versatile Disc) and its drive device, or any storage medium that may be used in the future.

[0115] The above describes a configuration in which the functions of each component of the remote control support device 100-500 are realized by either hardware or software. However, this is not the only configuration; some components of the remote control support device 100-500 can be realized by dedicated hardware, while other components can be realized by software. For example, some components can have their functions realized by a processing circuit 60 as dedicated hardware, while other components can have their functions realized by a processing circuit 60 acting as a processor 61 reading and executing a program stored in memory 62.

[0116] As described above, the remote control support devices 100 to 500 can realize each of the above-mentioned functions through hardware, software, etc., or a combination thereof.

[0117] Although this disclosure has been described in detail, the above description is illustrative in all respects and does not limit this disclosure. It is understood that countless variations not illustrated may be conceivable without falling outside the scope of this disclosure.

[0118] Furthermore, within the scope of this disclosure, it is possible to freely combine each embodiment, or to modify or omit each embodiment as appropriate.

Claims

1. A remote control support device for temporarily operating an autonomous operating system from a remote location, An operating interface that accepts input from a remote operator, A state detection unit that detects the surrounding environment information of the autonomous operating system and information of the autonomous operating system, A failure factor estimation unit obtains the surrounding environment information and the information of the autonomous operating system from the state detection unit and estimates the failure factors that are hindering the autonomous operation of the autonomous operating system. A fault response unit acquires remote operation information using the fault factors obtained from the fault factor estimation unit, the surrounding environment information and the information of the autonomous operating system obtained from the state detection unit, and a database storing remote operation information for restoring the autonomous operating system to autonomous operation, and calculates operation instructions for remotely operating the autonomous operating system based on the remote operation information. The system comprises a control unit that controls the autonomous operating system in accordance with the operation instructions output from the fault response unit, The aforementioned fault response unit, If the autonomous operating system cannot be restored to autonomous operation, the remote operator is requested to perform an intervention operation to remotely control the autonomous operating system. A state presentation unit that acquires the surrounding environment information and the information of the autonomous operating system from the state detection unit and presents it to the remote operator, An intervention operation interface that receives intervention operation instructions from the remote operator and inputs them to the control unit, The device further comprises an intervention operation learning unit that learns the intervention operation using the intervention operation instruction obtained from the control unit, the failure factor obtained from the failure factor estimation unit, the surrounding environment information and the information of the autonomous operating system obtained from the state detection unit, and transmits the intervention operation instruction, the failure factor, the surrounding environment information and the information of the autonomous operating system to an intervention operation sharing recording unit outside the remote operation support device, The control unit, When the intervention operation instruction is input, the autonomous operating system is controlled in accordance with the intervention operation instruction. The intervention operation sharing recording unit is: A remote operation support device that shares the intervention operation instructions, the failure factors, the surrounding environment information, and the information of the autonomous operating system transmitted from the intervention operation learning unit with a system other than the remote operation support device.

2. The intervention operation sharing recording unit is: In response to a request from the intervention operation learning unit, external information obtained from the other system is transmitted to the intervention operation learning unit. The intervention operation learning unit is, The remote operation support device according to claim 1, which updates the database based on the external information transmitted from the intervention operation sharing record unit.

3. The system further includes a remote operator status detection unit that acquires and outputs the status of the remote operator, The intervention operation interface is, The analysis information entered by the remote operator is received and input into the intervention operation learning unit. The intervention operation learning unit is, The remote operation support device according to claim 2, wherein the state of the remote operator output from the remote operator state detection unit and the analysis information input from the intervention operation interface are used for learning the intervention operation.

4. The aforementioned state of the remote operator is, Including the gaze information of the remote operator, The analysis information entered by the remote operator is: This includes the basis for the remote operator's judgment regarding the intervention operation, concerns, and information on alternatives, The intervention operation learning unit is, Based on the aforementioned gaze information or the basis for the aforementioned judgment, the degree of confidence in the aforementioned obstacle factor is set. Based on the aforementioned concerns, the presence or absence of the aforementioned concerns in the remote control information is determined. The remote control support device according to claim 3, which learns the intervention operation by setting the alternative remote control information based on the aforementioned alternative.

Images