Remote operation support device

Abstract

The present disclosure relates to a remote operation assistance device, comprising: a state detection unit that detects surrounding environment information of an autonomous operating system and information about the autonomous operating system; an operation interface that accepts input from a remote operator including an intervention operation to remotely control the autonomous operating system, and outputs it as operation information; a state presentation unit that acquires surrounding environment information and information about the autonomous operating system from the state detection unit and presents it to the remote operator; a remote operator state recognition unit that acquires state information of the remote operator and recognizes the state of the remote operator using the state information and the operation information acquired from the operation interface; a display setting dynamic change unit that dynamically determines settings of information to be presented to the remote operator in accordance with the state of the remote operator recognized by the remote operator state recognition unit, and outputs them as display settings; and a fault response auxiliary information presentation unit that displays on a display device as auxiliary information to assist the remote operator in making decisions, based on the display setting obtained from the display setting dynamic change unit.

Description

[Technical field]

[0001] The present disclosure relates to a remote operation support device, and more particularly to a remote operation support device that supports recovery work of an autonomous operating system when an event occurs in which the autonomous operating system cannot continue its autonomous operation. [Background technology]

[0002] In the event that some kind of fault occurs in an autonomous operating system causing it to stop operating autonomously, a remote operator at a remote location will determine the cause of the fault and then remotely operate the system to return it to autonomous operation.It is anticipated that the remote operator will be responsible for multiple autonomous operating systems at the same time.

[0003] In this case, the remote operator is required to instantly grasp the status of multiple different autonomous operating systems, which places a large cognitive load on the operator. Furthermore, as autonomous operating systems become more advanced, the number and types of sensors installed tend to increase, and the amount of data handled tends to increase. Therefore, it is not easy to understand the surrounding environment information of the autonomous operating system and the status of the autonomous operating system based on that information, and to consider methods to remove the causes of failures.

[0004] Patent Document 1 discloses a computer operation management system that can perform efficient maintenance and rapid failure recovery without relying on the skill level of individual workers. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2001-265474 A Summary of the Invention [Problem to be solved by the invention]

[0006] Patent Document 1 discloses a technology that supports workers' response by displaying operational management procedures as a flowchart on a screen, displaying related information and response processing procedure information, and sequentially displaying the related information as sub-blocks.

[0007] However, even if the response procedure is known, the cognitive load on the operator is not reduced, and it is not easy to quickly grasp the situation and consider countermeasures.

[0008] The present disclosure has been made to solve the above problems, and aims to provide a remote operation support device that can reduce the cognitive load on a remote operator. [Means for solving the problem]

[0009] A remote operation support device according to the present disclosure is a remote operation support device that temporarily operates an autonomously operating autonomous system from a remote location, and includes a state detection unit that detects surrounding environment information of the autonomous operation system and information of the autonomous operation system, an operation interface that receives input from a remote operator including an intervention operation to remotely control the autonomous operation system and outputs it as operation information, a state presentation unit that acquires the surrounding environment information and the information of the autonomous operation system from the state detection unit and presents it to the remote operator, and a remote operator state display unit that acquires state information of the remote operator and recognizes the state of the remote operator using the state information and the operation information acquired from the operation interface. a state recognition unit; a display setting dynamic change unit that dynamically determines a setting of information to be presented to the remote operator according to the state of the remote operator recognized by the remote operator state recognition unit and outputs the setting as a display setting; a failure response auxiliary information presentation unit that displays, on a display device, the display setting obtained from the display setting dynamic change unit as auxiliary information that assists the judgment of the remote operator; and a remote operator state presentation unit that notifies the remote operator of the state of the remote operator recognized by the remote operator state recognition unit and notifies the remote operator of an intention of the change when the display setting is dynamically changed, wherein the auxiliary information includes information to be confirmed in order to grasp the state of the autonomous operating system when a failure occurs.The remote operator state recognition unit estimates at least one of the busyness of the remote operator and the fatigue level of the remote operator and recognizes the estimated busyness and fatigue level of the remote operator as the state of the remote operator. do. Effect of the Invention

[0010] According to the remote operation assistance device of the present disclosure, the settings of the information to be presented to the remote operator are dynamically determined based on the state of the remote operator and output as display settings, and based on the display settings, the fault response auxiliary information presentation unit displays the information on the display device as auxiliary information to assist the remote operator in making decisions, thereby reducing the cognitive load on the remote operator. [Brief description of the drawings]

[0011] [Figure 1] 10 is a flowchart showing a flow of an action of a remote operator operating a remote operation assistance device according to the present disclosure. [Diagram 2] 1 is a block diagram showing a configuration of a remote operation assistance device according to a first embodiment of the present disclosure. [Diagram 3] FIG. 11 is a diagram illustrating an example of a state of a remote operator in the form of a table. [Figure 4] FIG. 13 is a diagram illustrating an example of a table showing accumulated work contents handled in the past. [Diagram 5] FIG. 13 is a diagram showing an example of a case where information is presented to a remote operator in a flow chart format. [Figure 6] FIG. 13 is a diagram showing an example of a case where information is presented to a remote operator in a Yes / No format. [Figure 7] 5 is a flowchart showing an operation of the remote operation assistance device according to the first embodiment of the present disclosure. [Figure 8] 11 is a block diagram showing a configuration of a remote operation assistance device according to a second embodiment of the present disclosure. FIG. [Figure 9] 10 is a flowchart showing an operation of a remote operation assistance device according to a second embodiment of the present disclosure. [Figure 10] FIG. 11 is a block diagram showing a configuration of a remote operation assistance device according to a third embodiment of the present disclosure. [Figure 11] 13 is a flowchart showing an operation of the remote operation assistance device according to the third embodiment of the present disclosure. [Figure 12] FIG. 11 is a block diagram showing a configuration of a remote operation assistance device according to a fourth embodiment of the present disclosure. [Figure 13] 13 is a flowchart showing an operation of the remote operation assistance device according to the fourth embodiment of the present disclosure. [Figure 14] FIG. 11 is a block diagram showing a configuration of a remote operation assistance device according to a fourth embodiment of the present disclosure. [Figure 15] FIG. 11 is a diagram illustrating an example of a state of a remote operator in the form of a table. [Figure 16] 13 is a flowchart showing an operation of the remote operation assistance device according to the fourth embodiment of the present disclosure. [Figure 17] FIG. 1 is a diagram illustrating a hardware configuration for implementing a remote operation support device according to first to fifth embodiments of the present disclosure. [Figure 18] FIG. 1 is a diagram illustrating a hardware configuration for implementing a remote operation support device according to first to fifth embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] <Introduction> 1 is a flowchart showing a flow of operations of a remote operator who operates a remote operation support device according to the present disclosure. Hereinafter, the role of the remote operation support device according to the present disclosure will be described with reference to FIG.

[0013] 1, the autonomous operating system notifies the occurrence of a fault from the autonomous operating system. This notification is made, for example, via a monitor device for the autonomous operating system provided separately from the remote operation support device.

[0014] The remote operator who has received the notification checks the current state of the relevant autonomous operation system via the remote operation support device (step S2).

[0015] Next, the remote operator grasps the details of the current fault in the autonomous operation system via the remote operation support device (step S3).

[0016] Next, the remote operator uses the remote operation support device to consider a means for recovering from the current failure of the autonomous operation system (step S4).

[0017] Finally, the remote operator restores the failure by remotely operating the autonomous operation system based on the restoration means (step S5), thereby completing the series of operations.

[0018] In the following explanation, a self-driving vehicle will be used as an example of an autonomous operating system; however, this is not limited to this and examples include automatically moving man-made objects such as flying objects, drones, probes, and mobility robots.

[0019] <Embodiment 1> <Device configuration> 2 is a block diagram showing a configuration of a remote operation support device 100 according to a first embodiment of the present disclosure. The remote operation support device 100 is used when temporarily controlling an autonomous operation system 32, such as an autonomous driving vehicle, from a remote location, and includes a control unit 1, a state detection unit 2, an operation interface 3, a remote operator state recognition unit 4, a state presentation unit 5, a display setting dynamic change unit 6, and a failure response auxiliary information presentation unit 7.

[0020] The state detection unit 2 detects surrounding environment information 33 of the autonomous operation system 32 and information about the autonomous operation system 32. Examples of the surrounding environment information 33 of the autonomous operation system 32 include road surface conditions, roadway conditions such as the presence or absence of obstacles on the roadway, positional relationships with pedestrians, the states of other traffic participants such as the state of pedestrians, weather conditions such as sunny, rainy, cloudy, snowy, etc., surrounding sounds, road types, road surface conditions such as whether the road is paved or not, the presence or absence of pedestrian crossings, the presence or absence of traffic lights, etc. The information may also be acquired by multiple sensors provided in the autonomous operation system 32, or may also be acquired from an infrastructure system installed on the road.

[0021] The information of the autonomous operation system 32 includes information determined by the autonomous operation system 32, information of the devices and sensors installed in the autonomous operation system 32, and status information inside the vehicle.

[0022] More specifically, the information determined by the autonomous operation system 32 includes the determination result of the autonomous operation system 32. The information of the devices and sensors mounted on the autonomous operation system 32 includes speed, acceleration, engine RPM, horn information, steering information, accelerator information, brake information, turn signal information, position information, etc. The status information inside the vehicle includes status information of objects present inside the vehicle such as passengers and luggage, and environmental information of the vehicle interior space such as sound and temperature.

[0023] The surrounding environment information 33 and the information of the autonomous operation system 32 are transmitted to the status presenting unit 5 and the fault response auxiliary information presenting unit 7 .

[0024] The state presenting unit 5 provides the remote operator with surrounding environment information 33 of the autonomous operation system 32 transmitted from the state detecting unit 2 and information on the autonomous operation system 32 via the display device DP. The remote operator looks at the display device DP and inputs an intervention operation instruction using the input device 31.

[0025] Here, the remote operator is a person who, when some kind of failure occurs in the autonomous operating system 32 and autonomous operation stops, remotely controls the autonomous operating system 32 from a remote location in accordance with requests from the autonomous operating system 32 and the remote operation support device 100, thereby restoring the autonomous operating system 32 to autonomous operation again.

[0026] Furthermore, the operations input by the remote operator using the input device 31 include operations for controlling the remote operation support device 100 in addition to intervention operations for remotely controlling the autonomous operation system 32. For example, there is an operation for changing the content displayed by the remote operation support device 100 so that the remote operator can identify the cause of a failure or grasp the status of the autonomous operation system 32.

[0027] In addition, operations by the remote operator other than the intervention operation of remotely controlling the autonomous operation system 32 include communication between staff and communication with the crew. Communication between staff includes operations for consulting and requesting work from other staff involved in the operation of the autonomous operation system 32, such as a vehicle maintenance worker or manager, when a case arises that the remote operator cannot handle. In addition, communication with the crew includes operations for calling the crew aboard the autonomous operation system 32, such as a case where an emergency button is pressed in the autonomous operation system 32.

[0028] The operation interface 3 transmits operation information input by the remote operator via the input device 31 to the control unit 1 and the remote operator state recognition unit 4. In addition, the state presentation unit 5 and the failure response auxiliary information presentation unit 7 receive operation instructions from the operation interface 3 for changing the display content of the remote operation gaze device 100.

[0029] The control unit 1 accepts, via the operation interface 3, intervention operation instructions for remotely controlling the autonomous operating system 32 from the operation information input by the remote operator via the input device 31, and controls the autonomous operating system 32 in accordance with the intervention operation instructions from the remote operator.

[0030] The remote operator state recognition unit 4 recognizes the state of the remote operator from the operation information of the remote operator and the state information of the remote operator obtained from the operation interface 3.

[0031] To obtain the status information of the remote operator, a gaze detection device such as an eye tracker and a camera installed in the remote operation room where the remote operation support device 100 is installed, a heart rate sensor worn by the remote operator, etc. can be used. Since the viewpoint information of the remote operator is obtained from the eye tracker and the camera, the gaze of the remote operator is obtained and it is detected where the remote operator is looking on the screen of the display device DP presenting the information. The technology for detecting the gaze is publicly known and is installed in vehicles, etc. to monitor the driver's drowsiness.

[0032] In addition, since the heart rate sensor can obtain information on the remote operator's heart rate, the remote operator's fatigue level can be estimated from the interval between heart rates. Techniques for estimating fatigue level from heart rate are well known, and for example, the remote operator's fatigue level can be estimated by comparing the average resting heart rate of the remote operator with the heart rate during work.

[0033] Furthermore, the remote operator state recognition unit 4 estimates the remote operator's work proficiency and how busy he or she is from the operation of the remote operator for controlling the remote operation support device 100 and the intervention operation instruction for remotely controlling the autonomous operation system 32, among the operation information input by the remote operator via the input device 31, and creates a work history. Information obtained from the autonomous operation system 32 can also be used to create the work history.

[0034] The level of proficiency can be expressed by the remote operator's years of service, cumulative working hours, and the results of learning and training using a simulator or the like.

[0035] Busyness can be expressed by the number of cases handled by a remote operator per unit time, the working hours, which is the cumulative time spent on handling cases, or the difficulty of the cases, which is expressed as whether remote operation is difficult or not required.

[0036] The work history is a record of the remote operator's past work, and includes the work content and the location information of the corresponding autonomous operation system.

[0037] In addition, the remote operator status recognition unit 4 can not only recognize the remote operator from the operation information and the remote operator's status information, but also utilize status information about the remote operator inputted by the remote operator. In particular, by configuring the remote operator to input information such as years of service and results of learning and training in advance, useful information can be obtained.

[0038] Proficiency, busyness, and fatigue can be expressed in stages. For example, for proficiency, the formula for calculating proficiency is: Years of service x X + cumulative working hours x Y + results of learning and training x Z = proficiency, and the calculation result can be defined in five stages. Here, X, Y, and Z are the ratios of the importance of each item, for example, X = 40%, Y = 40%, and Z = 20%. The stages are 0-1, 2-4, 5-7, 8-10, and 10 or more. If a remote operator works 4 hours a day, 20 days a month, and has 10 years of service, the cumulative working hours are 4 hours / day x 20 days x 12 months x 10 years = 9.6 (thousand hours). If the result of learning and training is 10, the proficiency is 10 x 40% + 9.6 x 40% + 10 x 20% = 9.84, and the proficiency stage is 4.

[0039] In addition, if a remote operator works 2 hours a day, 20 days a month, and has 20 years of service, the cumulative work hours will be 2 hours / day x 20 days x 12 months x 10 years = 9.6 (thousand hours). If the result of learning and training is 10, the proficiency level will be 20 x 40% + 9.6 x 40% + 10 x 20% = 13.84, and the proficiency level will be 5.

[0040] Busyness can be expressed by the workload of the remote operator, for example, by "difficulty of the task handled" x "task time". If the sum of "difficulty of the task handled" x "task time (minutes)" of the tasks handled per hour is 0 to 10, the busyness level is set to 1, if it is 11 to 20, the busyness level is set to 2, if it is 21 to 30, the busyness level is set to 3, if it is 41 to 50, the busyness level is set to 4, and if it is 51 or more, the busyness level is set to 5. For example, if 5 minutes of a task of difficulty level 2 and 3 minutes of a task of difficulty level 1 are performed in an hour, then 2 x 5 + 1 x 3 = 13, so the busyness level is set to 2.

[0041] Regarding the work history, the most recent work content, the location information of the autonomous operation system, the time taken for the work, etc. can be displayed in list form. It can also be calculated as a cumulative total of the work content handled in the past.

[0042] Fig. 3 shows in a table an example of the state of the remote operator recognized by the remote operator state recognition unit 4. Fig. 3 shows the proficiency level, busyness level, fatigue level, and work history, with the proficiency level being 5, the busyness level being 3, and the fatigue level being 2. The work history includes, for example, [XX street intersection, remote operation, 5 minutes] and [XY bus stop, conversation with passenger, 5 minutes].

[0043] In addition, Figure 4 shows a table of an example of accumulated work contents that have been handled in the past. In Figure 4, the response contents are listed as emergency vehicle response, pedestrians running out into the street, bus stop response, etc., and the number of responses for each is 10 times for emergency vehicle response, 20 times for pedestrians running out into the street, and 400 times for bus stop response.

[0044] The work content can use operation information from a remote operator or the result of judgment by the autonomous operation system 32. As an example of using the result of judgment by the autonomous operation system 32, information used when the autonomous operation system 32 judges that it cannot run on its own, for example, "insufficient recognition of white lines on the road", the work of the remote operator requires "remote operation", so "remote operation" is set as the work content.

[0045] In this case, the "judgment result of the autonomous operation system" and the "task of the remote operator" are associated in advance, and when "failure to recognize white lines on the road" is acquired from the autonomous operation system 32, the remote operator state recognition unit 4 calculates "remote operation" as the task content. Note that when this method is adopted, the remote operator state recognition unit 4 is configured to acquire the judgment result of the autonomous operation system 32 via the state detection unit 2.

[0046] The remote operator state recognition unit 4 can also use the remote operator's state information, the remote operator's operation for controlling the remote operation support device 100, and the intervention operation instruction when the remote operator remotely operates the autonomous operation system 32 as input information, and calculate the proficiency level, busyness level, and fatigue level by machine learning using artificial intelligence (AI). Using AI enables faster processing.

[0047] The display setting dynamic change unit 6 dynamically determines the settings of the form and amount of information to be presented to the remote operator via the display device DP based on the state of the remote operator recognized by the remote operator state recognition unit 4, i.e., proficiency, busyness, and fatigue level, and transmits it to the fault response auxiliary information presentation unit 7 as display settings.

[0048] The form of information available includes a flow chart format, a Yes / No format, and other formats with detailed guides, as well as a format that displays only the check items that need to be confirmed.

[0049] Figure 5 shows an example of a case where information is presented to a remote operator in a flow chart format. In Figure 5, first, it is confirmed whether there has been a collision of the vehicle, and if there has been no collision, it is confirmed whether the equipment on the vehicle is normal. If the equipment is normal, it moves on to checking whether there are people or objects around the vehicle.

[0050] When checking whether or not there has been a vehicle collision, if there has been a collision, the driver is instructed to secure the vehicle's safety and then to contact relevant parties.

[0051] When checking the vehicle's equipment, if any abnormalities are found in the equipment, you will be instructed to ensure the safety of the vehicle and then to contact the relevant parties.

[0052] If there are people or objects around the vehicle, the driver will proceed to check whether it is possible to avoid them and continue driving. If this is possible, the driver will be instructed to control the vehicle remotely. If this is not possible, the driver will be instructed to ensure the safety of the vehicle and then to contact relevant parties.

[0053] For convenience, only a part of the flowchart is shown in FIG. 5, and the remote operator can take a series of necessary measures by carrying out confirmation operations in accordance with the flowchart displayed on the display device DP.

[0054] Fig. 6 shows an example of a case where information is presented to a remote operator in a Yes / No format. In Fig. 6, first, a screen SC1 for confirming whether or not there has been a vehicle collision is displayed on the display device DP. If Yes is selected on screen SC1, the screen transitions to screen SC2, and if No is selected, the screen transitions to screen SC3. On screen SC3, instructions are given to ensure the safety of the vehicle.

[0055] The screen SC2 is a screen for confirming whether the vehicle's equipment is normal. If Yes is selected, the screen transitions to screen SC4, and if No is selected, the screen transitions to screen SC5.

[0056] Screen SC4 instructs the driver to ensure the safety of the vehicle, and screen SC5 is a screen for checking whether there are people or objects around the vehicle.

[0057] For convenience, FIG. 6 shows only a portion of the Yes / No display, and the remote operator can take a series of necessary measures by performing confirmation tasks in accordance with the Yes / No display displayed on the display device DP.

[0058] For example, by presenting information in a flow chart format to a remote operator with high proficiency and in a Yes / No format to a remote operator with low proficiency, information presentation that may be bothersome depending on the remote operator's proficiency, or conversely, unfriendly to an inexperienced remote operator, can be provided that reflects the characteristics of each remote operator. Therefore, appropriate information can be presented in an appropriate amount, and judgment errors by the remote operator can be reduced.

[0059] The amount of information is the amount of supplementary information, and is set by the amount of text to be written in the flow charts and the like.

[0060] More specifically, the settings for the form and amount of information are dynamically changed depending on the state of the remote operator. For example, depending on the level of proficiency, an expert may be shown only the minimum number of items that need to be checked, while a flowchart with detailed explanations is displayed for an inexperienced remote operator. Furthermore, the amount of information in the explanation is adjusted depending on how busy the operator is. With regard to the work history, if it is the first time the operator is dealing with a location, detailed information is displayed, but if a similar task has been performed recently, the display is simplified. Also, if the same task has been performed on multiple machines in succession, the display is simplified, but if there are some differences, information is added to draw attention to the differences.

[0061] The display setting dynamic change unit 6 can dynamically determine the format and amount of information to be presented to the remote operator using machine learning by AI, based on the input information of the remote operator's state recognized by the remote operator state recognition unit 4, which includes proficiency, busyness, and fatigue. Using AI enables faster processing.

[0062] The fault response auxiliary information presentation unit 7 creates information to assist the remote operator in making a decision, and presents it on the display device DP together with the information from the status presentation unit 5. That is, based on the display settings obtained from the display setting dynamic change unit 6, the form and amount of information are adjusted, and the information is displayed as auxiliary information to assist the remote operator in making a decision.

[0063] As auxiliary information to assist the remote operator in making decisions, information that the remote operator should check to identify the cause of the failure or to understand the status of the autonomous operation system 32 and the operating procedures of the remote operation support device 100, information that should be checked regarding measures to restore autonomous operation and the operating procedures of the remote operation support device 100, etc. are created.

[0064] With regard to the information to be confirmed and the operating procedures for the remote operation support device, preset information such as the flow chart format and Yes / No format described above is used, while other information to be provided based on individual events, such as the recognition result of the autonomous operation system 32 and surrounding environment information 33 of the autonomous operation system 32, is created independently. For example, if the recognition result of the autonomous operation system 32 is "poor recognition of white lines," it displays "The system's recognition result was 'poor recognition of white lines'."

[0065] When creating auxiliary information to assist the remote operator in making decisions, and when creating display content according to the display settings, the surrounding environment information 33 of the autonomous operating system 32 acquired from the status acquisition unit 2 and information about the autonomous operating system 32 can be used.

[0066] The location and highlighting of information can also be adjusted when presented to the remote operator.

[0067] <Operation> Next, the operation of the remote operation support device 100 will be described with reference to the flowchart shown in Fig. 7. When the remote operation support device 100 is operated, first, the state detection unit 2 acquires the surrounding environment information 33 of the autonomous operation system 32 and information about the autonomous operation system 32 (step S11).

[0068] The remote operator state recognition unit 4 acquires the state information of the remote operator (step S12).

[0069] The remote operator state recognition unit 4 recognizes the state of the remote operator based on the state information of the remote operator and the operation information of the remote operator (step S13).

[0070] The display setting dynamic change unit 6 dynamically determines the setting of the display contents for the remote operator based on the state information of the remote operator (step S14).

[0071] The fault response auxiliary information presentation unit 7 creates information to assist in fault response based on the surrounding environment information 33 of the autonomous operating system 32 and information about the autonomous operating system 32, and creates content to be presented to the remote operator based on the display content settings determined by the display setting dynamic change unit 6 (step S15).

[0072] The fault response auxiliary information presentation unit 7 displays the surrounding environment information 33 of the autonomous operating system 32, the information of the autonomous operating system 32, and information for assisting in fault response on the display device DP (step S16), and the series of processes ends.

[0073] <Effects> As described above, in the remote operation support device 100 of embodiment 1, the remote operator status recognition unit 4 recognizes the status of the remote operator from the remote operator's operation information and the remote operator's status information, the display setting dynamic change unit 6 dynamically determines the settings for the form and amount of information to be presented to the remote operator based on the status of the remote operator recognized by the remote operator status recognition unit 4 to create display settings, and the fault response auxiliary information presentation unit 7 presents this to the remote operator as information to assist the remote operator in making decisions, thereby reducing the cognitive load on the remote operator.

[0074] <Embodiment 2> <Device configuration> Fig. 8 is a block diagram showing a configuration of a remote operation support device 200 according to a second embodiment of the present disclosure. The remote operation support device 200 has a configuration in which a remote operator state presentation unit 8 that displays the state of the remote operator recognized by the remote operator state recognition unit 4 on a display device is further provided in addition to the configuration of the remote operation support device 100 shown in Fig. 2. In addition, in Fig. 8, the same components as those of the remote operation support device 100 described using Fig. 2 are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0075] The remote operator status presentation unit 8 notifies the remote operator of the status of the remote operator recognized by the remote operator status recognition unit 4, and conveys to the remote operator the intention of the change when the display on the display device DP is dynamically changed, thereby reducing the sense of discomfort felt by the remote operator when the display is dynamically changed.

[0076] That is, when the amount of information displayed on the display device DP changes, for example, when a detailed flow chart is suddenly displayed and the information is presented in a simplified manner, the remote operator may be confused as to why the display has suddenly changed. This also applies when the information presented changes from a simplified form to a complex form.

[0077] Therefore, by displaying the intention to change the display in combination with the state of the remote operator, the remote operator can be given a sense of satisfaction regarding the change in the display. For example, by notifying the remote operator that "Your proficiency has improved, so we have simplified the guide" or "Your fatigue level is increasing, so we will make the guide more detailed," the sense of discomfort caused by a sudden change in the display can be reduced.

[0078] <Operation> Next, the operation of the remote operation support device 200 will be described with reference to the flowchart shown in Fig. 9. Note that steps S11 to S16 in Fig. 9 are the same as steps S11 to S16 in the flowchart shown in Fig. 7, and therefore description thereof will be omitted.

[0079] In step S17 following step S16 in FIG. 9, the remote operator status presenting unit 8 displays the intention to change the display and the status of the remote operator on the display device DP, and then the series of processes ends.

[0080] <Effects> As described above, in the remote operation support device 200 of embodiment 2, the remote operator status presentation unit 8 combines the intention to change the display with the status of the remote operator and displays it on the display device DP. Therefore, even if the form and amount of information presented to the remote operator changes, the remote operator can be given a sense of satisfaction regarding the change in display, and the sense of discomfort caused by a sudden change in display can be reduced.

[0081] <Embodiment 3> <Device configuration> FIG. 10 is a block diagram showing a configuration of a remote operation support device 300 according to a third embodiment of the present disclosure. The remote operation support device 300 has basically the same configuration as the remote operation support device 200 shown in FIG. 8, but in the remote operation support device 300, the display setting dynamic change unit 6 has a function of acquiring information of the autonomous operating system 32, for example, information of a sensor mounted on the autonomous operating system 32, via the state detection unit 2, and dynamically changes the display setting according to the acquired sensor information. For example, when the temperature sensor acquires data in a range different from that in normal times, a process such as highlighting is performed on the display item related to the temperature sensor. This process allows the remote operator to pay attention to the target sensor, and assists in determining the necessity of an intervention operation.

[0082] <Operation> Next, the operation of the remote operation support device 300 will be described with reference to the flowchart shown in Fig. 11. Note that steps S11 to S13, S15, and S16 in Fig. 11 are the same as steps S11 to S13, S15, and S16 in the flowchart shown in Fig. 9, and therefore will not be described.

[0083] In step S141 following step S13 in FIG. 11, the display setting dynamic change unit 6 dynamically determines the setting of the display content for the remote operator based on the state information of the remote operator and the information of the sensor mounted on the autonomous operation system 32.

[0084] <Effects> As described above, in the remote operation support device 300 of embodiment 3, the display setting dynamic change unit 6 acquires information from the sensors mounted on the autonomous operation system 32 and dynamically changes the display settings in accordance with the acquired sensor information, thereby increasing the amount of information to assist the remote operator in making decisions and reducing the cognitive load on the remote operator.

[0085] <Fourth embodiment> <Device configuration> FIG. 12 is a block diagram showing a configuration of a remote operation support device 400 according to a fourth embodiment of the present disclosure. The remote operation support device 400 has basically the same configuration as the remote operation support device 300 shown in FIG. 10, but in the remote operation support device 400, the remote operator state recognition unit 4 has a function of acquiring viewpoint information of the remote operator and recognizing where the remote operator is looking on the screen of the display device DP. In addition, the display setting dynamic change unit 6 has a function of changing the display position based on the screen information of the screen that the remote operator is looking at, recognized by the remote operator state recognition unit 4. In addition, the failure response auxiliary information presentation unit 7 presents auxiliary information by reflecting the movement of the remote operator's line of sight on the screen of the display device DP that the remote operator is looking at.

[0086] The remote operator state recognition unit 4 acquires the viewpoint information of the remote operator using a gaze detection device such as an eye tracker and a camera installed in the remote operation room where the remote operation support device 100 is placed. Since the viewpoint information of the remote operator is obtained from the eye tracker and the camera, it is possible to acquire the gaze of the remote operator and recognize where on the screen of the display device DP the remote operator is looking. Note that the screen information of the screen the remote operator is currently looking at is added as the remote operator's state information.

[0087] <Operation> Next, the operation of the remote operation support device 400 will be described with reference to the flowchart shown in Fig. 13. Note that steps S11, and S14 to S16 in Fig. 13 are the same as steps S11, and S14 to S16 in the flowchart shown in Fig. 9, and therefore description thereof will be omitted.

[0088] In step S121 following step S11 in FIG. 13, the remote operator state recognition unit 4 acquires state information of the remote operator including viewpoint information of the remote operator.

[0089] The remote operator state recognition unit 4 recognizes the state of the remote operator, including where the remote operator is looking on the screen of the display device DP, based on the remote operator state information and the remote operator operation information (step S131).

[0090] <Effects> As described above, in the remote operation support device 400 of the fourth embodiment, the remote operator state recognition unit 4 recognizes where the remote operator is looking on the screen of the display device DP, and the display setting dynamic change unit 6 changes the display position based on the screen information of the screen the remote operator is looking at recognized by the remote operator state recognition unit 4, so that the fault response auxiliary information presentation unit 7 can reflect the screen information of the display device DP that the remote operator is looking at in the display setting. When there are multiple screens that the remote operator looks at on the display device DP, important information can be displayed in a tracked manner on the screen that the remote operator is currently looking at, thereby improving the convenience of the remote operator.

[0091] <Embodiment 5> <Device configuration> Fig. 14 is a block diagram showing a configuration of a remote operation support device 500 according to a fifth embodiment of the present disclosure. The remote operation support device 500 has basically the same configuration as the remote operation support device 400 shown in Fig. 12, but in the remote operation support device 500, the remote operator state recognition unit 4 has a function of recognizing where the remote operator is looking on the screen of the display device DP, and has a function of adding attention information that the remote operator paid attention to during the intervention operation to the work history. In addition, the display setting dynamic change unit 6 sets information to be highlighted using the attention information that the remote operator paid attention to.

[0092] Fig. 15 shows an example of the state of the remote operator recognized by the remote operator state recognition unit 4 in a table. Fig. 15 shows the proficiency level, busyness level, fatigue level, the screen currently being viewed by the remote operator, and the work history, with the proficiency level being 5, busyness level being 3, fatigue level being 2, and the screen currently being viewed being the left screen. The work history includes [XX street intersection, remote operation, 5 minutes, video from the vehicle-mounted camera], [XY bus stop, conversation with passenger, 5 minutes, sensor operation status], etc.

[0093] 15, when the remote operator remotely operates the autonomous operation system 32 at the intersection of XX street, the remote operator watches the image from the onboard camera on the screen of the display device DP and talks to the passenger at the bus stop XY, while paying attention to the operating status of the sensor on the screen of the display device DP. In this way, the attention information that the remote operator paid particular attention to can be added to the work history.

[0094] The display setting dynamic change unit 6 can reflect the information that the remote operator focused on during the intervention operation in the display settings. For example, when the same work is required, the information that was focused on in similar cases in the past, for example, when remotely operating the autonomous operation system 32 at an intersection on XX street, is highlighted to show the image from the onboard camera, and also, at the XY bus stop, to check the operating status of the sensor, thereby presenting the operation that the remote operator should take, and improving the convenience of the remote operator.

[0095] <Operation> Next, the operation of the remote operation support device 500 will be described with reference to the flowchart shown in Fig. 16. Note that steps S11, S12, and S14 to S16 in Fig. 16 are the same as steps S11, S12, and S14 to S16 in the flowchart shown in Fig. 9, and therefore description thereof will be omitted.

[0096] In step S132 following step S12 in FIG. 16, the remote operator state recognition unit 4 recognizes the state of the remote operator, including the work history having information that the remote operator was paying attention to at the time of the intervening operation, based on the remote operator's state information and the remote operator's operation information.

[0097] <Effects> As described above, in the remote operation support device 500 of embodiment 5, the remote operator state recognition unit 4 adds the information that the remote operator was paying attention to during the intervention operation to the work history, and the display setting dynamic change unit 6 reflects the information that the remote operator was paying attention to during the intervention operation in the display settings, thereby improving convenience for the remote operator.

[0098] <Hardware configuration> Each of the components of the remote operation support devices 100 to 500 according to the first to fifth embodiments described above can be configured using a computer, and is realized by the computer executing a program. That is, the remote operation support devices 100 to 500 are realized, for example, by a processing circuit 60 shown in Fig. 17. A processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor) is applied to the processing circuit 60, and the function of each part is realized by executing a program stored in a storage device.

[0099] Dedicated hardware may be applied to the processing circuit 60. When the processing circuit 60 is dedicated hardware, the processing circuit 60 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of these.

[0100] The functions of the components of the remote operation support devices 100 to 500 can be realized by individual processing circuits, or these functions can be realized collectively by one processing circuit.

[0101] 18 shows a hardware configuration in the case where the processing circuit 60 is configured using a processor. In this case, the functions of each part of the remote operation support devices 100 to 500 are realized by a combination of software, etc. (software, firmware, or software and firmware). The software, etc. are described as a program and stored in the memory 62. The processor 61 functioning as the processing circuit 60 realizes the functions of each part by reading and executing the program stored in the memory 62 (storage device). In other words, it can be said that this program causes a computer to execute the procedure and method of operation of the components of the remote operation support devices 100 to 500.

[0102] Here, the memory 62 may be, for example, a non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), HDD (Hard Disk Drive), magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc) and its drive device, or any storage medium to be used in the future.

[0103] The above describes a configuration in which the functions of the components of the remote operation support devices 100 to 500 are realized by either hardware or software. However, this is not limited to this, and some of the components of the remote operation support devices 100 to 500 can be realized by dedicated hardware, and other components can be realized by software, etc. For example, it is possible to realize the functions of some of the components by the processing circuit 60 as dedicated hardware, and realize the functions of other components by the processing circuit 60 as the processor 61 reading and executing a program stored in the memory 62.

[0104] As described above, the remote operation support devices 100 to 500 can realize the above-mentioned functions by hardware, software, or a combination of these.

[0105] Although the present disclosure has been described in detail, the above description is illustrative in all respects and does not limit the present disclosure. It is understood that countless variations not illustrated can be assumed without departing from the scope of the present disclosure.

[0106] In addition, within the scope of the present disclosure, it is possible to freely combine the various embodiments, and to modify or omit the various embodiments as appropriate.

Claims

1. A remote operation support device that temporarily operates an autonomous operating system from a remote location, comprising: A state detection unit that detects surrounding environment information of the autonomous operating system and information of the autonomous operating system; an operation interface that receives input from a remote operator including an intervention operation for remotely controlling the autonomous operation system and outputs the input as operation information; a state presentation unit that acquires the surrounding environment information and the information of the autonomous operation system from the state detection unit and presents the information to the remote operator; a remote operator state recognition unit that acquires state information of the remote operator and recognizes a state of the remote operator using the state information and the operation information acquired from the operation interface; a display setting dynamic change unit that dynamically determines a setting of information to be presented to the remote operator according to the state of the remote operator recognized by the remote operator state recognition unit, and outputs the setting as a display setting; a fault response auxiliary information presenting unit that displays, on a display device, auxiliary information for assisting a judgment by the remote operator based on the display setting obtained by the display setting dynamic changing unit; A remote operation support device, wherein the auxiliary information includes information that should be confirmed in order to understand the status of the autonomous operating system when a failure occurs.

2. A remote operation assistance device as described in claim 1, wherein the auxiliary information includes information that the remote operator should check to identify the cause of the failure.

3. A remote operation assistance device as described in claim 1, wherein the auxiliary information includes information to be confirmed regarding measures to be taken to restore autonomous operation.

4. A remote operation assistance device as described in claim 1, wherein the auxiliary information includes operating procedures for the remote operation assistance device for restoring autonomous operation.

5. The remote operator state recognition unit is 2. The remote operation support device according to claim 1, wherein the remote operator's level of proficiency in the task, the busyness of the remote operator, and the fatigue level of the remote operator are estimated and recognized as the state of the remote operator.

6. The display setting dynamic change unit is Dynamically determining the format and amount of information to be presented to the remote operator; The fault response auxiliary information presentation unit 2. The remote operation support device according to claim 1, wherein the form and amount of the information obtained from the dynamic display setting change unit are adjusted and displayed on the display device.

7. The remote operation support device according to claim 1 , further comprising a remote operator status presentation unit that notifies the remote operator of the status of the remote operator recognized by the remote operator status recognition unit, and that notifies the remote operator of an intention for change when the display settings are dynamically changed.

8. The display setting dynamic change unit is The remote operation support device according to claim 1 , wherein the display settings are dynamically determined using information about the autonomous operating system acquired from the state detection unit.

9. The remote operator state recognition unit is Obtaining viewpoint information of the remote operator, and recognizing where on the screen of the display device the remote operator is looking; The display setting dynamic change unit is 2. The remote operation support device according to claim 1, wherein screen information of the screen viewed by the remote operator recognized by the remote operator state recognition unit is dynamically reflected in the display settings and used as the auxiliary information.

10. The remote operator state recognition unit is Obtaining viewpoint information of the remote operator, and recognizing where on the screen of the display device the remote operator is looking; Recognizing, as attention information, screen information of the screen on which the remote operator was paying attention when the remote operator performed the intervention operation; The display setting dynamic change unit is 2. The remote operation support device according to claim 1, wherein the attention information of the remote operator recognized by the remote operator state recognition unit is dynamically reflected in the display settings to be used as the auxiliary information.

11. The remote operator state recognition unit The remote operation support device according to claim 5 , wherein the state information and the operation information are input information, and the proficiency level, the busyness level, and the fatigue level of the remote operator are calculated by machine learning using artificial intelligence.

12. The display setting dynamic change unit is The remote operation support device according to claim 11, wherein the proficiency level, the busyness level, and the fatigue level of the remote operator calculated by the remote operator state recognition unit are used as input information, and the settings of the information to be presented to the remote operator are dynamically determined by the machine learning by the artificial intelligence.

Images