Behavior support device, behavior support system, and behavior support method

Abstract

To provide a behavior support device capable of preventing reduction of entire efficiency even in a case where a behavior of a target person is not supported any more, by supporting the behavior of the target person while improving autonomy and a skill level of the target person.SOLUTION: The present invention relates to a behavior support device 10 which supports a behavior of a target person according to an environment around the target person. The behavior support device 10 comprises a learning section 11, a prediction section 12 and a behavior support section 13. The learning section 11 generates a behavior model BM of the target person by learning environment information EI relating to the environment and behavior information BI relating to the behavior of the target person with respect to the environment. The prediction section 12 generates a predictive behavior PB of the target person based on the behavior model BM and the environment information EI. The behavior support section 13 generates behavior support information BSI1-BSI4 for the target person depending on a matching level of an optimal behavior OB of the target person based on the environment information EI and the predictive behavior PB of the target person.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an action support device, an action support system, and an action support method. [Background technology]

[0002] Conventionally, there have been known guidance programs for providing guidance information to a target person and guiding the target person appropriately. The guidance management program described in the following Patent Document 1 has a transmission / reception means and a reversible printing means (Patent Document 1, claim 5, etc.). The transmission / reception means and the reversible printing means perform reversible printing on a reversible display section of an information display medium having a reversible display section on which information is visually reversibly displayed and a data storage means, and transmit and receive data to and from the data storage means.

[0003] This conventional guidance management program uses a management computer connected to an action schedule storage means and a guidance data storage means to manage guidance of the subject. Here, the action schedule storage means is a means for recording action schedule data relating to the node terminal installed in the node and the subject. Also, the guidance data storage means is a means for recording guidance data for guiding the subject to the next node to which he or she should proceed. This conventional guidance management program causes the management computer to function as a node identification means, an acquisition means, and an output means.

[0004] The node identification means acquires behavior identification data recorded in the data storage means of the information display medium via the node terminal, and identifies the next node to which the subject will advance from the behavior schedule storage means based on the behavior identification data. The acquisition means acquires, from the guidance data storage means, guidance data for guiding the subject from the node where the reversible printing means of the node terminal is installed to the next node to which the subject will advance. The output means displays the acquired guidance data on the reversible display unit of the information display medium via the node terminal.

[0005] According to this conventional guidance management program, the subject can be guided to the next node by the guidance printed on the reversible display part of the information display medium. Therefore, for example, the subject can be guided in sequence to nodes approaching the final destination, so that the subject can be guided accurately. Also, each time the subject arrives at a node, guidance to the next node is provided. Therefore, the management computer can grasp the subject's progress at each node and manage the subject's behavior process (Patent Document 1, paragraph 0011, etc.). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2006-126173 A Summary of the Invention [Problem to be solved by the invention]

[0007] The above-mentioned conventional guidance management program can accurately guide the subject by sequentially guiding the subject to nodes approaching the final destination, but the subject may become overly dependent on the guidance, which may reduce the subject's autonomy and hinder the subject's mastery. Therefore, for example, if a malfunction occurs in the guidance management program, the subject may not be able to take appropriate action according to the surrounding environment, and overall efficiency may decrease.

[0008] The present disclosure provides a behavior support device, a behavior support system, and a behavior support method that can support a subject's behavior while improving the subject's autonomy and proficiency, thereby preventing a decrease in overall efficiency even when support for the subject's behavior is no longer available. [Means for solving the problem]

[0009] One aspect of the present disclosure is a behavior support device that supports the behavior of a subject in accordance with the environment surrounding the subject, comprising: a learning unit that learns environmental information related to the environment and behavioral information related to the subject's behavior relative to the environment to generate a behavior model of the subject; a prediction unit that generates a predicted behavior of the subject based on the behavior model and the environmental information; and a behavior support unit that generates behavior support information for the subject in accordance with the degree of match between the subject's optimal behavior based on the environmental information and the predicted behavior of the subject. Effect of the Invention

[0010] According to the above-described aspect of the present disclosure, it is possible to provide a behavior support device that can support the subject's behavior while improving the subject's autonomy and proficiency, thereby preventing a decrease in overall efficiency even when support for the subject's behavior is no longer available. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an embodiment of a behavior support device and system according to the present disclosure. [Diagram 2] 2 is a perspective view showing an example of the environment around a subject supported by the action support device of FIG. 1; [Diagram 3] 3 is a plan view showing a movement path of a moving object in the environment of FIG. 2. [Figure 4] 1 is a flow diagram showing an embodiment of a behavior support method according to the present disclosure. [Diagram 5] FIG. 5 is a flow diagram showing details of a process for acquiring environmental information around the subject shown in FIG. 4. [Figure 6] FIG. 5 is a flow diagram showing details of the process of acquiring behavioral information of the subject shown in FIG. 4. [Figure 7] FIG. 5 is a flow diagram showing details of the process of generating a behavioral model of the subject shown in FIG. 4. [Figure 8] FIG. 5 is a flow diagram showing details of the process of generating a predicted behavior of the subject shown in FIG. 4 . [Figure 9] FIG. 5 is a flow chart showing details of a process for generating action support information for the subject shown in FIG. 4. [Figure 10] FIG. 5 is a flow chart showing details of a process for providing behavioral support information to the subject of FIG. 4. [Figure 11] 2 is an image diagram showing an example of behavioral support information provided to a subject by the UI of FIG. 1. [Figure 12] 2 is an image diagram showing an example of behavioral support information provided to a subject by the UI of FIG. 1. [Figure 13] 2 is an image diagram showing an example of behavioral support information provided to a subject by the UI of FIG. 1. [Figure 14] 2 is an image diagram showing an example of behavioral support information provided to a subject by the UI of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the action support device, action support system, and action support method of the present disclosure will be described with reference to the drawings.

[0013] Fig. 1 is a block diagram showing an embodiment of a behavior support device and behavior support system according to the present disclosure. Fig. 2 is a perspective view showing an example of a target person OP and a surrounding environment E for behavior support by the behavior support device 10 and the behavior support system 100 of Fig. 1.

[0014] The action support device 10 of this embodiment is a device that supports the action of the subject OP according to the environment E surrounding the subject OP, and is a device that generates action support information BSI1-BSI4 for the subject OP. The environment E is, for example, an environment such as a logistics warehouse where autonomous machines M1, M2, ... such as autonomously traveling unmanned forklifts and workers who are the subject OP work simultaneously in the same space.

[0015] The action support system 100 of this embodiment includes an action support device 10 and a user interface (UI) 20 that provides a subject OP with action support information generated by the action support device 10. The action support system 100 may also include, for example, an external sensor 30 and a communication device 40.

[0016] The action support device 10 is a device that supports the action of the subject OP in accordance with the surrounding environment E of the subject OP. The action support device 10 can be configured, for example, by a central processing unit (CPU), memories such as RAM and ROM, a timer, and one or more microcontrollers or firmware equipped with an input / output unit, or a computer.

[0017] The action support device 10 includes, for example, a learning unit 11, a prediction unit 12, and an action support unit 13. The action support device 10 may also include, for example, a action information acquisition unit 14, an environmental information acquisition unit 15, and an overall action planning unit 16. The action information acquisition unit 14, the environmental information acquisition unit 15, and the overall action planning unit 16 may be configured as devices separate from the action support device 10.

[0018] Each part of the action support device 10 shown in Fig. 1 represents each function of the action support device 10 that is realized, for example, by executing a program recorded in memory by a CPU. Each part of the action support device 10 shown in Fig. 1 may be configured by a separate device, two or more parts may be configured by the same device, or all parts may be configured by a single device.

[0019] The UI20 is, for example, an augmented reality (AR) glass or a smart glass that can be worn by the subject OP. The UI20 receives, for example, the action support information BSI1-BSI4 transmitted from the action support device 10 via the communication device 40, and provides the action support information BSI1-BSI4 to the subject OP by displaying the action support information BSI1-BSI4 in the field of view of the subject OP without significantly obstructing the field of view of the subject OP. Note that the UI20 is not limited to smart glasses, and may be, for example, a mobile information terminal such as a smartphone, or a device that displays information in the environment E surrounding the subject OP, such as a digital signage or a projector.

[0020] The external sensor 30 includes, for example, at least one of one or more cameras 31 and one or more LiDARs 32. The external sensor 30 detects external information including the positions, speeds, and moving directions of the subject OP and the autonomous machines M1, M2, ..., the presence or absence of a container box CB, and the like, in an application area A set in the environment E, and outputs the information to the action support device 10. In the example shown in FIG. 2, two cameras 31 and two LiDARs 32 are installed adjacent to each other in the application area A.

[0021] The communication device 40 is, for example, a wireless communication device capable of wireless communication with the UI 20. The communication device 40 is, for example, connected to the action support apparatus 10 via a wired communication line or a wireless communication line so as to be able to communicate information, and receives action support information BSI1-BSI4 for the subject OP from the action support apparatus 10. The communication device 40 transmits, for example, the action support information BSI1-BSI4 received from the action support apparatus 10 to the UI 20 via the wireless communication line.

[0022] The operations of the action support apparatus 10 and the action support system 100 of this embodiment will be described below.

[0023] Fig. 3 is a plan view showing movement routes R1-R4 of the autonomous machines M1, M2 and the subject OP, which are moving bodies in the environment E in Fig. 2. The overall action planning unit 16 shown in Fig. 1 optimizes the actions of the subjects OP, such as multiple workers, and the autonomous machines M1, M2, such as multiple forklifts, for example, calculates movement routes R1, R2, R4 that maximize overall efficiency, and assigns tasks to the subjects OP and the autonomous machines M1, M2.

[0024] More specifically, the overall behavior planning unit 16 generates movement routes R1, R2 for each of the autonomous machines M1, M2 traveling in an environment E such as a logistics warehouse, for example, based on an order, and transmits the movement routes R1, R2 to each of the autonomous machines M1, M2 via the communication device 40. Each of the autonomous machines M1, M2 autonomously travels along the respective movement routes R1, R2 received via a wireless communication line. In addition, the overall behavior planning unit 16 generates an optimal behavior OB including an optimal movement route R4 for the subject OP, for example, based on environmental information EI regarding the environment E surrounding the subject OP.

[0025] Furthermore, the overall action planning unit 16, for example, assigns a task to each subject OP, and transmits information on the assigned task to the UI 20 or a mobile information terminal (not shown) of the subject OP via the communication device 40. The subject OP performs work according to the task displayed on the UI 20 or the mobile information terminal, for example. Here, if the task assigned to the subject OP includes, for example, movement to a point P1 in the environment E, the subject OP starts moving to the point P1 according to the task displayed on the UI 20 or the mobile information terminal.

[0026] Here, the shortest route from the current position of the target person OP to the point P1 shown in Fig. 3 is the movement route R3. However, if the target person OP moves along the movement route R3, the target person OP may obstruct the path of the autonomous machines M1 and M2 moving along the movement routes R1 and R2, and the collision prevention functions of the autonomous machines M1 and M2 may be activated, causing the autonomous machines M1 and M2 to make an emergency stop, which may result in a decrease in the overall efficiency of the work in the environment E.

[0027] In such a case, a subject OP who is familiar with the environment E, that is, a subject OP who has gained experience in the environment E and is highly familiar with it, can predict, for example, the movement routes R1, R2 of the autonomous machines M1, M2 from information about the environment E around the subject OP. Here, examples of the information about the environment E around the subject OP include the movement direction of the autonomous machine M1 while transporting the container box CB, an intersection that is a branching point of the passage along which the autonomous machine M1 travels, a point P2 where the container box CB has not been placed, and the movement direction of the autonomous machine M2 after the container box CB has been placed.

[0028] In an environment E as shown in Fig. 3, when a subject OP familiar with the environment E moves to a point P1, the subject OP moves along a movement route R4 that does not obstruct the paths of the autonomous machines M1 and M2, rather than along the shortest movement route R3, in order to prevent a decrease in the overall efficiency in the environment E. As a result, an emergency stop of the autonomous machines M1 and M2 can be prevented, and a decrease in the overall efficiency in the environment E can be prevented. However, since not all subjects OP are familiar with the environment E, action support for the subject OP is necessary.

[0029] Fig. 4 is a flow diagram showing an embodiment of a behavior support method according to the present disclosure. The behavior support method BSM of this embodiment can be implemented, for example, by the behavior support system 100 shown in Fig. 1. The behavior support method BSM of this embodiment includes, for example, a step S1 of acquiring environmental information EI related to the surrounding environment E of the subject OP, and a step S2 of acquiring behavior information BI related to the behavior of the subject OP with respect to the surrounding environment E.

[0030] The behavior support method BSM of this embodiment also includes, for example, a step S3 of learning the environmental information EI and the behavior information BI by machine learning to generate a behavior model BM of the subject OP, and a step S4 of generating a predicted behavior PB of the subject OP based on the behavior model BM and the environmental information EI. The behavior support method BSM of this embodiment also includes, for example, a step S5 of generating behavior support information BSI1-BSI4 for the subject OP according to the degree of agreement DC between the optimal behavior OB of the subject OP based on the environmental information EI and the predicted behavior PB of the subject OP. The behavior support method BSM of this embodiment also includes, for example, a step S6 of providing the behavior support information BSI1-BSI4 to the subject OP.

[0031] The action support device 10 of this embodiment repeatedly executes, for example, steps S1 to S6 shown in Fig. 4 at a predetermined cycle. Details of each step S1-S6 shown in Fig. 4 and the operation of the action support device 10 and the action support system 100 in each step S1-S6 will be described below with reference to Figs. 5 to 14.

[0032] Fig. 5 is a flow diagram showing details of step S1 of acquiring environmental information in Fig. 4. When the action support device 10 of the action support system 100 shown in Fig. 1 starts step S1 shown in Fig. 5, for example, it performs steps S11 and S12 of acquiring camera images CI and LiDAR data LD from the camera 31 and the LiDAR 32, respectively. More specifically, the environmental information acquisition unit 15 of the action support device 10 acquires camera images CI including images of the environment E around the subject OP from the multiple cameras 31 in step S11, and acquires LiDAR data LD including point cloud data of the environment E around the subject OP from the multiple LiDARs 32 in step S12.

[0033] Next, the environmental information acquisition unit 15 executes step S13 of acquiring environmental information EI regarding the environment E around the subject OP from, for example, the camera image CI and the LiDAR data LD. In this step S13, the environmental information acquisition unit 15 extracts point cloud data of the autonomous machines M1, M2, ..., which are moving bodies, from the LiDAR data LD by background subtraction, and estimates the positions of the autonomous machines M1, M2, .... Also, in this step S13, the environmental information acquisition unit 15 applies semantic segmentation to, for example, the camera image CI of the application area A of the environment E, and classifies objects in the application area A into categories such as passages, container boxes CB, and autonomous machines M1, M2, ....

[0034] As a result, the environmental information acquisition unit 15 can acquire environmental information EI including, for example, the estimated positions of the autonomous machines M1, M2, ... and the categories of objects in the application area A. After that, the environmental information acquisition unit 15 performs step S14 of outputting the acquired environmental information EI to the learning unit 11 and the prediction unit 12, and ends step S1 shown in FIG.

[0035] Fig. 6 is a flow diagram showing details of step S2 of acquiring behavior information BI of the subject OP in Fig. 4. When the behavior support device 10 constituting the behavior support system 100 starts step S2 shown in Fig. 6, for example, it performs steps S11 and S12 of acquiring camera images CI and LiDAR data LD from the cameras 31 and LiDAR 32 shown in Fig. 1 and Fig. 2, respectively. More specifically, the behavior information acquisition unit 14 of the behavior support device 10 acquires camera images CI including images of the subject OP from the multiple cameras 31 in step S21, and acquires LiDAR data LD including point cloud data of the subject OP from the multiple LiDARs 32 in step S22.

[0036] Next, the behavior information acquisition unit 14 executes step S23 of acquiring behavior information BI related to the behavior of the subject OP from, for example, the camera image CI and the LiDAR data LD. In this step S23, the behavior information acquisition unit 14 performs, for example, tracking to track the moving route of the subject OP. More specifically, the behavior information acquisition unit 14 identifies the subject OP from, for example, the camera image CI including the image of the subject OP, and grasps the general position of the subject OP.

[0037] In addition, the behavioral information acquisition unit 14, for example, extracts point cloud data of a moving object including the subject OP from the LiDAR data LD by background subtraction, and extracts point cloud data of the subject OP from the rough position of the subject OP based on the camera image CI, thereby estimating the detailed position of the subject OP. In this way, the behavioral information acquisition unit 14 can acquire behavioral information BI including, for example, the movement route of the subject OP. Thereafter, the behavioral information acquisition unit 14 performs step S24 of outputting the acquired behavioral information BI to the learning unit 11 and the prediction unit 12, and ends step S2 shown in FIG. 6.

[0038] Fig. 7 is a flow diagram showing details of step S3 of generating a behavior model BM of the subject OP in Fig. 4. In this step S3, the action support device 10 of the action support system 100 learns, by machine learning, environmental information EI related to the environment E and behavior information BI related to the behavior of the subject OP with respect to the environment E, to generate a behavior model BM of the subject OP. When starting step S3 shown in Fig. 7, the action support device 10 performs, for example, a learning data collection step S31 and a learning step S32 in parallel.

[0039] In the learning data collection step S31, the learning unit 11 of the action support device 10 performs, for example, steps S311 to S316. The learning unit 11 first performs a step S311 of acquiring environmental information EI around the subject OP from the environmental information acquisition unit 15, a step S312 of acquiring behavioral information BI of the subject OP from the behavioral information acquisition unit 14, and a step S313 of acquiring destination information DI of the subject OP from the overall behavior planning unit 16.

[0040] Furthermore, the learning unit 11 executes step S314 in which it compares the location information of the subject OP included in the behavior information BI of the subject OP acquired in step S312 with the destination information DI of the subject OP acquired in step S313 to determine whether the subject OP has arrived at the destination. In this step S314, for example, if the difference between the location information of the subject OP and the destination information DI of the subject OP, i.e., the distance between the location of the subject OP and the destination, is greater than a predetermined threshold, the learning unit 11 determines that the subject OP has not arrived at the destination (No). In this case, the learning unit 11 repeats steps S311 to S314 again.

[0041] On the other hand, in step S314, if the difference between the location information of the subject OP and the destination information DI of the subject OP, i.e., the distance between the location of the subject OP and the destination, is equal to or less than a predetermined threshold, the learning unit 11 determines that the subject OP has arrived at the destination (Yes). In this case, the learning unit 11 performs step S315 to acquire information on the time when the subject OP arrived at the destination. Furthermore, the learning unit 11 performs step S316 to store the time series of the environmental information EI and the behavioral information BI acquired in steps S311 and S312 in a database, for example, by dividing the time of arrival of the subject OP at the destination acquired in step S315.

[0042] By this step S316, the learning unit 11 can associate the environmental information EI and the behavioral information BI while the subject OP moves from the previous destination to the current destination, for example, and store them in the database as one episode log. As a result, the behavioral information BI including the movement route of the subject OP from the previous destination to the current destination, and the environmental information EI around the subject OP while moving along that movement route become available.

[0043] In parallel with the above-described learning data collection step S31, the learning unit 11 performs a learning step S32. In the learning step S32, the learning unit 11 first performs a step S321 of determining whether or not to start machine learning. In this step S321, the learning unit 11 determines not to start learning (No) when, for example, the amount of episode log data accumulated in the database does not reach a predetermined level, or when a predetermined time (for example, about one hour) has not elapsed.

[0044] In this case, the learning unit 11 performs step S325 of acquiring the behavioral model BM without performing step S322 of acquiring learning data, step S323 of learning the behavioral model BM, and step S324 of storing parameters of the behavioral model BM in the database.

[0045] On the other hand, the learning unit 11 determines to start learning (Yes) when, for example, the amount of episode log data accumulated in the database exceeds a predetermined level, or when a preset predetermined time (for example, about one hour) has elapsed, etc. This allows the learning unit 11 to perform step S322 of acquiring learning data in a state in which episode logs with a data amount equal to or greater than the predetermined level are accumulated in the database, and machine learning can be performed more efficiently.

[0046] In step S322, the learning unit 11 acquires, as learning data, episode logs accumulated in the database by the above-mentioned learning data collection step S31 from the database. As described above, the episode log is information that associates environmental information EI around the subject OP with behavioral information BI of the subject OP while the subject OP moves from the previous destination to the current destination. Furthermore, the learning unit 11 executes step S323 of generating, by machine learning using the acquired episode log, a behavior model BM for predicting the behavior of the subject OP with respect to the environmental information EI around the subject OP.

[0047] Here, the behavior model BM generated by the learning unit 11 is a model that inputs, for example, information on the departure point included in the behavior information BI of the target person OP, destination information DI of the target person OP, and environmental information EI around the target person OP, and outputs a predicted movement route of the target person OP. Such a behavior model BM of the target person OP can be constructed, for example, by a method such as deep learning.

[0048] That is, in step S323, the learning unit 11 extracts the departure point information, destination information DI, and environmental information EI of the subject OP from the episode log. Furthermore, the learning unit 11 performs machine learning of the behavior model BM by providing the movement route of the subject OP in the episode log to the behavior model BM as teacher data. After that, the learning unit 11 performs step S324 of storing new parameters of the behavior model BM for which machine learning has been completed in a database, or updating old parameters of the behavior model BM stored in the database with the new parameters.

[0049] As described above, in the learning step S32, the learning unit 11 learns the environmental information EI related to the environment E and the behavioral information BI related to the behavior of the subject OP with respect to the environment E, and generates the behavioral model BM of the subject OP. After that, the learning unit 11 performs a step S325 of acquiring the behavioral model BM from the database and a step S326 of outputting the behavioral model BM to the prediction unit 12, and ends the step S3 shown in FIG.

[0050] Fig. 8 is a flow diagram showing details of step S4 of generating a predicted behavior PB of the subject OP in Fig. 4. In this step S4, the prediction unit 12 of the action support apparatus 10 shown in Fig. 1 generates a predicted behavior PB of the subject OP based on the action model BM and environmental information EI.

[0051] 8, the prediction unit 12 performs, for example, a step S41 of acquiring destination information DI related to the destination of the subject OP from the overall behavior planning unit 16, and a step S42 of acquiring behavior information BI related to the behavior of the subject OP from the behavior information acquiring unit 14. Furthermore, in parallel with these steps, the prediction unit 12 performs a step S43 of acquiring environmental information EI related to the surrounding environment E of the subject OP from the environmental information acquiring unit 15, and a step S44 of acquiring a behavior model BM of the subject OP from the learning unit 11.

[0052] Next, the prediction unit 12 performs step S45 of inputting the position information of the subject OP included in the behavior information BI of the subject OP, the destination information DI of the subject OP, and the surrounding environment information EI of the subject OP into the behavior model BM, and generating a predicted behavior PB of the subject OP as an output of the behavior model BM. The predicted behavior PB of the subject OP includes, for example, a movement path of the subject OP predicted based on the previous behavior of the subject OP with respect to the surrounding environment E of the subject OP. After that, the prediction unit 12 performs step S46 of outputting the generated predicted behavior PB to the behavior support unit 13, and ends step S4 shown in FIG. 8.

[0053] Fig. 9 is a flow diagram showing details of step S5 of generating action support information BSI1-BSI4 for the subject OP in Fig. 4. In this step S5, the action support unit 13 of the action support device 10 shown in Fig. 1 generates action support information BSI1-BSI4 for the subject OP according to, for example, the degree of coincidence DC between the optimal action OB of the subject OP based on the environmental information EI and the predicted action PB of the subject OP.

[0054] 9, the action support unit 13 performs a step S501 of acquiring a predicted action PB of the subject OP from the prediction unit 12, and a step S502 of acquiring an optimal action OB based on the environmental information EI around the subject OP from the overall action planning unit 16. Next, the action support unit 13 performs, for example, a step S503 of calculating a degree of coincidence DC between the predicted action PB of the subject OP and the optimal action OB of the subject OP.

[0055] The degree of agreement DC can be calculated based on the degree of agreement between multiple indicators, such as, for example, proceeding in the same direction along the same passage from the starting point, or branching in the same direction at a branch point such as an intersection of the passage. The degree of agreement DC may be set to 100% when the predicted action PB and the optimal action OB completely match, and may be decreased as the difference between the predicted action PB and the optimal action OB increases. The degree of agreement DC may be increased, for example, as the difference between the moving speed of the subject OP included in the predicted action PB and the optimal action OB decreases. The degree of agreement DC may be decreased as the difference between the overall efficiency of the environment E for the optimal action OB of the subject OP and the overall efficiency of the environment E based on the predicted action PB of the subject OP increases.

[0056] Next, the action support unit 13 performs step S504 to determine whether the degree of coincidence DC calculated in step S503 is equal to or greater than a first threshold value Th1. In step S504, for example, if the first threshold value Th1 is set to 100%, it is determined whether the movement path included in the predicted action PB of the subject OP and the movement path included in the optimal action OB of the subject OP completely coincide with each other. Note that the first threshold value Th1 may be set to a value less than 100%.

[0057] In step S504, when the action support unit 13 determines that the degree of coincidence DC is equal to or greater than the first threshold value Th1 (Yes), for example, that the movement path included in the predicted action PB of the subject OP and the movement path included in the optimal action OB of the subject OP completely match, it executes step S505 of generating first action support information BSI1. The first action support information BSI1 generated in this step S505 is information provided to the subject OP who is completely familiar with the environment E and whose predicted action PB and optimal action OB match or nearly match.

[0058] Therefore, the first action support information BSI1 generated in step S505 is the simplest information among the action support information BSI1-BSI4 generated by the action support unit 13. Note that in step S504, for example, when it is determined that the predicted action PB of the subject OP and the optimal action OB of the subject OP match (Yes) as described above, step S5 shown in Fig. 9 may be terminated without generating the action support information BSI1-BSI4. That is, the action support unit 13 may not generate the action support information BSI1-BSI4 when the degree of match DC between the optimal action OB of the subject OP and the predicted action PB exceeds a predetermined threshold value.

[0059] On the other hand, in the above-mentioned step S504, if the action support unit 13 determines that the degree of agreement DC is less than the first threshold value Th1 (No), for example, that the predicted action PB of the subject OP does not match the optimal action OB, the action support unit 13 performs the next step S506. In this step S506, for example, the action support unit 13 determines whether the degree of agreement DC is equal to or greater than the second threshold value Th2. Here, the second threshold value Th2 of the degree of agreement DC is set to a value lower than the first threshold value Th1 of the degree of agreement DC, for example.

[0060] In step S506, when the action support unit 13 determines that the degree of coincidence DC is equal to or greater than the second threshold value Th2 (Yes), it executes step S507 of generating second action support information BSI2. The second action support information BSI2 generated in step S507 is information provided to a subject OP who has some experience with the environment E and is relatively familiar with the environment E.

[0061] Therefore, the second action support information BSI2 generated in step S507 is the next simplest information after the first action support information BSI1, among the action support information BSI1-BSI4 generated by the action support unit 13. In other words, the second action support information BSI2 generated in step S507 is simpler than the third action support information BSI3 or the fourth action support information BSI4 described below, but is more detailed than the first action support information BSI1, which is the simplest.

[0062] On the other hand, if the action support unit 13 determines in the above-mentioned step S506 that the degree of agreement DC is less than the second threshold value Th2 (No), the action support unit 13 executes the next step S508. In this step S508, the action support unit 13 determines, for example, whether the degree of agreement DC is equal to or greater than a third threshold value Th3. Here, the third threshold value Th3 of the degree of agreement DC is set to, for example, a value lower than the second threshold value Th2.

[0063] In step S508, when the action support unit 13 determines that the degree of coincidence DC is equal to or greater than the third threshold value Th3 (Yes), it executes step S509 of generating third action support information BSI3. The third action support information BSI3 generated in step S509 is information provided to a subject OP who lacks experience with the environment E and is not very familiar with the environment E.

[0064] Therefore, the third action support information BSI3 generated in step S509 is, for example, more detailed information than the second action support information BSI2. However, the subject OP to whom the third action support information BSI3 will be provided has, for example, some experience with the environment E. Therefore, the third action support information BSI3 is, for example, more simplified information than the most detailed fourth action support information BSI4 described later.

[0065] On the other hand, in the above-mentioned step S508, when the action support unit 13 determines that the degree of coincidence DC is less than the third threshold value Th3 (No), it executes step S510 of generating fourth action support information BSI4. The fourth action support information BSI4 generated in this step S509 is, for example, information provided to a subject OP who has almost no experience with the environment E and is completely unfamiliar with the environment E. Therefore, the fourth action support information BSI4 generated in this step S510 is, for example, the most detailed information among the action support information BSI1-BSI4.

[0066] Thereafter, the action support unit 13 performs step S511 of outputting any one of the first to fourth action support information BSI1-BSI4 generated in the above-mentioned step S505, S507, S509, or S510 to the communication device 40 shown in Fig. 1, and ends step S5 shown in Fig. 9. The communication device 40 transmits the action support information BSI1-BSI4 input from the action support unit 13 to the UI 20, for example, via a wireless communication line.

[0067] Fig. 10 is a flow diagram showing details of step S6 of providing action support information BSI1-BSI4 to the subject OP in Fig. 4. When starting step S6 shown in Fig. 10, the UI20 of the action support system 100 shown in Fig. 1 first performs step S61 of receiving action support information BSI1-BSI4 transmitted from the action support device 10 via the communication device 40. Next, the UI20 performs step S62 of determining whether or not reception of action support information BSI1-BSI4 was successful.

[0068] In this step S62, if the UI20 determines that the reception of the action support information BSI1-BSI4 has failed (No), for example, it ends step S6 shown in Fig. 6 without performing step S63 of providing the action support information BSI1-BSI4 to the subject OP. After that, the action support system 100 ends the action support method BSM shown in Fig. 4, and repeatedly performs each step S1-S6 of the action support method BSM shown in Fig. 4 at a predetermined cycle.

[0069] In this case, the UI20 may notify the subject OP that reception of the action support information BSI1-BSI4 has failed. Furthermore, the UI20 may repeat step S61 of receiving the action support information BSI1-BSI4 until reception of any of the action support information BSI1-BSI4 is successful. On the other hand, in the above-mentioned step S62, when the UI20 determines that reception of any of the action support information BSI1-BSI4 has been successful (Yes), for example, it performs step S63 of providing any of the received action support information BSI1-BSI4 to the subject OP.

[0070] Fig. 11 is an image diagram showing an example of the first action support information BSI1 provided to the subject OP by the UI 20. When the UI 20 receives the first action support information BSI1 in the above-mentioned step S61, in step S63, for example, the most simplified action support information BSI1 as shown in Fig. 11 is displayed on the display screen of the AR glasses. As described above, the first action support information BSI1 is, for example, information provided to the subject OP who is completely familiar with the environment E.

[0071] 11, the action support information BSI1 displayed on the display screen of the UI20 indicates only that action support is not required for the subject OP. By displaying in this manner by the UI20, the subject OP with a high level of proficiency, to whom the simplest action support information BSI1 is provided, can recognize that the action support system 100 is operating normally.

[0072] FIG. 12 is an image diagram showing an example of the second action support information BSI2 provided to the subject OP by the UI20. When the UI20 receives the second action support information BSI2 in the above-mentioned step S61, in step S63, for example, as shown in FIG. 12, the UI20 displays the simplified action support information BSI2 on the display screen of the AR glasses. As described above, the second action support information BSI2 is information provided to the subject OP who has some experience with the environment E and is relatively familiar with the environment E. Therefore, in the example shown in FIG. 12, the second action support information BSI2 displayed on the display screen of the UI20 is more detailed than the above-mentioned first action support information BSI1, but is simplified information showing only an arrow corresponding to the traveling direction of the autonomous machine M1.

[0073] FIG. 13 is an image diagram showing an example of the third action support information BSI3 provided to the subject OP by the UI 20. When the UI 20 receives the third action support information BSI3 in the above-mentioned step S61, in step S63, for example, the action support information BSI3 as shown in FIG. 13 is displayed on the display screen of the AR glasses. As described above, the third action support information BSI3 is information provided to the subject OP who lacks experience with the environment E and is not very familiar with the environment E. Therefore, in the example shown in FIG. 13, the third action support information BSI3 displayed on the display screen of the UI 20 is more detailed than the above-mentioned second action support information BSI2, but is simplified information that only shows dashed lines corresponding to the movement routes R1, R2 of the autonomous machines M1, M2.

[0074] Fig. 14 is an image diagram showing an example of the fourth action support information BSI4 provided to the subject OP by the UI20. When the UI20 receives the fourth action support information BSI4 in the above-mentioned step S61, in step S63, for example, the most detailed action support information BSI4 as shown in Fig. 14 is displayed on the display screen of the AR glasses. As described above, the fourth action support information BSI4 is information provided to the subject OP who has little experience with the environment E and is completely unfamiliar with the environment E. Therefore, in the example shown in Fig. 14, the fourth action support information BSI4 displayed on the display screen of the UI20 is information that directly displays a movement route R4 corresponding to the optimal action OB of the subject OP.

[0075] After completing step S63 of providing any one of the above-mentioned action support information BSI1-BSI4, the UI20 ends step S6 shown in Fig. 10. Thereafter, the action support system 100 ends the action support method BSM shown in Fig. 4, and again repeatedly performs each step S1-S6 of the action support method BSM at a predetermined cycle.

[0076] As described above, the behavior support device 10 of this embodiment is a device that supports the behavior of the subject OP according to the environment E surrounding the subject OP, and includes a learning unit 11, a prediction unit 12, and a behavior support unit 13. The learning unit 11 learns environmental information EI related to the environment E and behavior information BI related to the behavior of the subject OP with respect to the environment E to generate a behavior model BM of the subject OP. The prediction unit 12 generates a predicted behavior PB of the subject OP based on the behavior model BM and the environmental information EI. The behavior support unit 13 generates behavior support information BSI1-BSI4 for the subject OP according to the degree of agreement DC between the optimal behavior OB of the subject OP based on the environmental information EI and the predicted behavior PB of the subject OP.

[0077] With such a configuration, the behavior support device 10 of this embodiment can learn the behavior of each subject OP with respect to the surrounding environment E by the learning unit 11, and generate a predicted behavior PB of the subject OP with respect to the surrounding environment E by the prediction unit 12. Furthermore, by generating behavior support information BSI1-BSI4 according to the degree of coincidence DC between the predicted behavior PB of the subject OP and the optimal behavior OB of the subject OP by the behavior support unit 13, behavior support according to the familiarity of each subject OP with respect to the environment E becomes possible.

[0078] More specifically, in the action support apparatus 10 of this embodiment, the action support unit 13 refines the action support information BSI1-BSI4 as the degree of coincidence DC between the optimal action OB and the predicted action PB of the subject OP decreases.

[0079] With this configuration, the action support device 10 of this embodiment can provide the most detailed action support information BSI4 as shown in Fig. 14 to a subject OP whose degree of matching DC between the predicted action PB and the optimal action OB is at the lowest level and whose familiarity with the environment E is at the lowest level. This makes it possible for even a subject OP whose familiarity with the environment E is low to take the optimal action OB according to the environment E around the subject OP, and prevents the overall efficiency of the environment E from decreasing.

[0080] Furthermore, in the action support apparatus 10 of this embodiment, the action support unit 13 simplifies the action support information BSI1-BSI4 as the degree of coincidence DC between the optimal action OB and the predicted action PB of the subject OP increases.

[0081] With this configuration, the action support device 10 of this embodiment can provide a subject OP whose familiarity with the environment E has improved and whose degree of match DC between the predicted action PB and the optimal action OB has increased with more simplified action support information BSI3 as shown in Fig. 13. Also, for a subject OP whose familiarity with the environment E has further improved and whose degree of match DC between the predicted action PB and the optimal action OB has further increased, the action support device 10 of this embodiment can provide a further simplified action support information BSI2 as shown in Fig. 12.

[0082] Furthermore, for a subject OP who is sufficiently familiar with the environment E, such as when the predicted action PB and the optimal action OB are almost identical, the action support device 10 of this embodiment can provide the most simplified action support information BSI1, such as a display of no need for target action support, as shown in FIG. 11. This prevents the subject OP from excessively depending on the action support information BSI1-BSI4, and can efficiently improve the autonomy and proficiency of the subject OP. As a result, when a malfunction occurs in the action support device 10, the subject OP can take appropriate action according to the surrounding environment E, and it is possible to prevent the overall efficiency of the environment E from decreasing.

[0083] Furthermore, in the action support device 10 of this embodiment, the action support unit 13 may not generate the action support information BSI1-BSI4 when the degree of coincidence DC between the optimal action OB and the predicted action PB of the subject OP exceeds a predetermined threshold value, as described above. Even with such a configuration, the action support device 10 of this embodiment can prevent the subject OP from excessively depending on the action support information BSI1-BSI4, and can efficiently improve the autonomy and proficiency of the subject OP. As a result, when a malfunction occurs in the action support device 10, the subject OP can take appropriate action according to the surrounding environment E, and the overall efficiency of the environment E can be prevented from decreasing.

[0084] In the action support device 10 of this embodiment, the action support unit 13 may simplify the action support information BSI1-BSI4 as the degree of coincidence DC between the optimal action OB and the predicted action PB of the subject OP decreases, as opposed to the above example. Even with such a configuration, the action support device 10 of this embodiment can prevent the subject OP from excessively depending on the action support information BSI1-BSI4, and can efficiently improve the autonomy and proficiency of the subject OP. As a result, when a malfunction occurs in the action support device 10, the subject OP can take appropriate action according to the surrounding environment E, and the overall efficiency of the environment E can be prevented from decreasing.

[0085] Moreover, the action support system 100 of this embodiment includes the above-mentioned action support device 10 and a user interface (UI20) that provides the subject OP with the action support information BSI1-BSI4 generated by the action support device 10. Therefore, according to the action support system 100 of this embodiment, not only can the same effects as those of the above-mentioned action support device 10 be achieved, but the action support information BSI1-BSI4 can also be efficiently provided to the subject OP by the UI20.

[0086] Moreover, the behavior support method BSM of this embodiment is a method for supporting the behavior of the subject OP according to the environment E around the subject OP, as described above. The behavior support method BSM has a step S3 of learning environmental information EI related to the environment E and behavior information BI related to the behavior of the subject OP with respect to the environment E by machine learning to generate a behavior model BM of the subject OP. The behavior support method BSM also has a step S4 of generating a predicted behavior PB of the subject OP based on the behavior model BM and the environmental information EI, and a step S5 of generating behavior support information BSI1-BSI4 for the subject OP according to the degree of agreement DC between the optimal behavior OB of the subject OP based on the environmental information EI and the predicted behavior PB of the subject OP. Furthermore, the behavior support method BSM has a step S6 of providing the behavior support information BSI1-BSI4 to the subject OP. With such a configuration, the behavior support method BSM of this embodiment can achieve the same effects as the behavior support device 10 and the behavior support system 100 described above.

[0087] The above describes in detail the embodiments of the action support device, action support system, and action support method according to the present disclosure using the drawings, but the specific configurations are not limited to these embodiments, and even if there are design changes, etc., that do not deviate from the gist of the present disclosure, they are included in the present disclosure. [Explanation of symbols]

[0088] 10 Behavior support device 11 Learning Department 12 Prediction Department 13 Behavior Support Department 20 UI (User Interface) 100 Behavioral Support System BI Behavioral Information BM Behavioral Model BSI1 Behavioral Support Information BSI2 Behavioral Support Information BSI3 Behavioral Support Information BSI4 Behavioral Support Information BSM behavior support method DC matching E. Environment EI environmental information OB Optimal Action OP Target PB Predictive Behavior

Claims

1. An action support device that supports an action of a subject in accordance with a surrounding environment of the subject, a learning unit that learns environmental information related to the environment and behavioral information related to the behavior of the subject with respect to the environment to generate a behavioral model of the subject; a prediction unit that generates a predicted behavior of the subject based on the behavior model and the environmental information; a behavior support unit that generates behavior support information for the subject according to a degree of agreement between an optimal behavior of the subject based on the environmental information and the predicted behavior of the subject; An action support device comprising:

2. The action support device according to claim 1 , wherein the action support unit simplifies the action support information as the degree of agreement between the optimal action and the predicted action of the subject increases.

3. The action support device according to claim 1 , wherein the action support unit is configured to make the action support information more detailed as the degree of agreement between the optimal action and the predicted action of the subject decreases.

4. 2. The action support device according to claim 1, wherein the action support unit does not generate the action support information when the degree of coincidence between the optimal action and the predicted action of the subject exceeds a predetermined threshold value.

5. The action support device according to any one of claims 1 to 4, a user interface for providing the subject with the action support information generated by the action support device; and An action support system comprising:

6. A behavior support method in which a computer supports a behavior of a subject in accordance with a surrounding environment of the subject, comprising: The computer learns environmental information related to the environment and behavioral information related to the behavior of the subject with respect to the environment through machine learning to generate a behavioral model of the subject; The computer generates a predicted behavior of the subject based on the behavior model and the environmental information; the computer generates behavioral support information for the subject according to a degree of agreement between an optimal behavior of the subject based on the environmental information and the predicted behavior of the subject; A behavior support method, comprising the steps of: providing the behavior support information to the subject.

Images