Operation support device, operation support system, operation support method, and operation support program

Abstract

To provide a technique capable determining a display position of information to support an operator operating an operation object, by taking an operator's psychological state into consideration when causing a display device to display the information.SOLUTION: An operation support device 300 is provided with a controller 70 making a display device 40 display specific information 80 to support an operator OP operating an operation object 200. The controller 70 determines a display position of the specific information 80 on the display device 40 in accordance with operator's real sensitivity degree of the operator OP to make the display device 40 display the specific information 80 at the determined display position.SELECTED DRAWING: Figure 16

Description

[Technical Field]

[0001] The present disclosure relates to a technique for assisting an operator who operates an operation object such as a work machine. [Background technology]

[0002] Patent Documents 1 to 3 disclose techniques for assisting an operator who operates a work machine.

[0003] The construction support system of Patent Document 1 comprises a computing device that generates a surrounding image based on an image captured by an imaging device, an information acquisition device that is attached to the shovel or external to the shovel, an operating device that is external to the shovel, and a display device that is external to the shovel and displays the surrounding image, and alerts the operator based on information acquired by the information acquisition device.

[0004] In the work machine of Patent Document 2, the display control device estimates the operator's line of sight based on the attitude of the work device detected by an attitude sensor, sets multiple display areas on the screen of the display device based on the estimated operator's line of sight, calculates the display position of the display image so that it is located in a display area among the multiple display areas that corresponds to the type of display image, and if the position of the multiple display areas on the screen of the display device changes due to a change in the operator's line of sight, the display position of the display image is maintained if the display image is still located within the display area corresponding to the type of display image after the change, and calculates a new display position of the display image so that the display image fits within the display area corresponding to the type of display image after the change.

[0005] The system of Patent Document 3 comprises a first imaging function control element that controls the zoom magnification to a specified magnification or less using the zoom function as an imaging function of the actual imaging device in a first state in which the work machine can be operated or the work machine is working, and a second imaging function control element that controls the zoom magnification using the zoom function as an imaging function of the actual imaging device in accordance with zoom operation via an input interface in a second state in which the work machine cannot be operated or the work machine is stopped working.

[0006] Non-Patent Document 1 describes the sense of agency, which it explains as a subjective feeling that, even though a human and some system are physically working together autonomously to some extent toward some goal, the human is ultimately in control of the goal and the control of the system.

[0007] Patent Document 4 discloses an evaluation device that quantifies the sense of ownership of an operator. Patent Document 4 explains that the sense of ownership is the sensation that an operator of a controlled device is operating the device himself. This evaluation device includes a calculation unit that calculates internal parameters of a mathematical model using sensor information, which is the measurement results of the input and output values, based on an input value representing a motor stimulus given to the body in conjunction with the movement of the controlled device, and outputs an output value related to an involuntary movement that occurs in response to the motor stimulus. The calculation unit then calculates an index value of the sense of ownership based on the calculated internal parameters. The input value representing the motor stimulus is a value representing the amount of head movement. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Takako Yoshida and two others, "Psychological and neuroscientific measurement of 'how comfortable one's body is to use'", [online], March 2016, Measurement and Control, Vol. 55, No. 3, [Retrieved May 2024], Internet<URL:https: / / www.jstage.jst.go.jp / article / sicejl / 55 / 3 / 55_252 / _pdf> [Patent documents]

[0009] [Patent Document 1] Japanese Patent Publication No. 2021-155936 [Patent Document 2] Japanese Patent Publication No. 2020-148005 [Patent Document 3] Japanese Patent Application Publication No. 2022-144495 [Patent Document 4] Japanese Patent Publication No. 2020-42641 Summary of the Invention [Problem to be solved by the invention]

[0010] When displaying information on a display device to assist an operator who operates an operating object such as a work machine, the display position of the information is an important factor in supporting the operator, but in the technologies of Patent Documents 1-3, the display position of the information is not determined taking into consideration the psychological state of the operator.

[0011] The present disclosure aims to provide a technology that, when displaying information on a display device to assist an operator operating an operation object, can determine the display position of the information taking into account the psychological state of the operator. [Means for solving the problem]

[0012] In the process of examining means for achieving the above-mentioned object, the inventors analyzed the relationship between the operator's sense of agency and the operator's effective visual field under various conditions in which the responsiveness of the operation object to the operator's operation was changed, i.e., various conditions in which the sense of agency was artificially changed.As a result, they found that when the operator's sense of agency is low, the effective visual field is narrower than when the sense of agency is high, making it more difficult for the operator to correctly recognize visual stimuli (the rate of correct answers to tasks decreases).Therefore, when specific information for supporting the operator who operates the operation object is displayed on a display device, it is desirable that the display position of the specific information be determined using the degree of the operator's sense of agency as an index.

[0013] The operation assistance device according to the first aspect has been made from the above viewpoints, and includes a controller that causes a display device to display specific information for assisting an operator who operates an operation target, and the controller determines a display position of the specific information on the display device according to a degree of a sense of ownership of the operation by the operator, and causes the display device to display the specific information at the determined display position.

[0014] In this first aspect, the controller can determine the display position of the specific information on the display device in consideration of the degree of the sense of ownership of the operation by the operator, i.e., the psychological state of the operator, thereby making it possible to display the specific information at the display position on the display device determined in consideration of the psychological state of the operator.

[0015] It is preferable that the operation assistance device according to the second aspect further comprises the following configuration in addition to the operation assistance device according to the first aspect. That is, in the operation assistance device according to the second aspect, it is preferable that the controller determines the display position to be closer to a reference position as the degree of the sense of ownership of the operation by the operator is lower. In this second aspect, when the degree of the sense of ownership of the operation by the operator is low, that is, when the effective field of view of the operator is narrowed and it becomes difficult for the operator to correctly recognize the specific information, the display position is displayed at a position close to the reference position, so that the operator can easily recognize the specific information.

[0016] In this second aspect, the controller may determine the display position of the specific information based on a map representing the relationship between the distance from the reference position to the display position of the specific information and the degree of sense of ownership of the operation.

[0017] It is preferable that the operation assistance device according to the third aspect further comprises the following configuration in addition to the operation assistance device according to the second aspect. That is, in the operation assistance device according to the third aspect, the reference position may be the position of the operator's point of gaze with respect to the display device. In this third aspect, the lower the degree of the operator's sense of ownership of the operation, the closer the specific information is displayed on the display device to the position of the operator's point of gaze, making it easier for the operator to recognize the specific information.

[0018] In this third aspect, it is preferable that, after displaying the specific information at a display position corresponding to the degree of the sense of ownership, the controller maintains the specific information displayed at the display position until a predetermined condition is met. This prevents the specific information from moving in accordance with the movement of the operator's point of gaze until the predetermined condition is met, making it easier for the operator to recognize the specific information. The predetermined condition may be, for example, the passage of a predetermined time. In this case, after displaying the specific information at a display position corresponding to the degree of the sense of ownership, the controller maintains the specific information displayed at the display position until the predetermined time has passed. The predetermined condition may also be, for example, the movement of the operator's point of gaze from within a reference range to outside the reference range. In this case, after displaying the specific information at a display position corresponding to the degree of the sense of ownership, the controller maintains the specific information displayed at the display position until the operator's point of gaze from within the reference range to outside the reference range. The reference range may be, for example, the operator's visual field range, which is set based on the operator's degree of the sense of ownership. In this case, the controller may set the field of view range, which was set at the time when the specific information was displayed at the display position, as the reference range.

[0019] Preferably, the operation assistance device according to the fourth aspect further includes the following configuration in the operation assistance device according to the second aspect. That is, in the operation assistance device according to the fourth aspect, the reference position may be either the position of a portion of the operation target or the position of a portion of an object present at a work site where the operation target is located, or the reference position may be either the position of a portion of the operation target included in an image displayed on the display device or the position of a portion of an object included in an image of the work site displayed on the display device. The operator operating the operation target may operate the operation target while viewing the portion of the operation target through the display area of ​​the display device or while viewing the portion of the operation target included in the image displayed in the display area of ​​the display device. Furthermore, if an object present at a work site is the target of work, the operator operating the operation target may operate the operation target while viewing the portion of the object through the display area of ​​the display device or while viewing the portion of the object included in the image displayed in the display area of ​​the display device. Therefore, when the operator has a low degree of sense of ownership of the operation, the specific information is displayed at a position closer to the apparent position of a part of the object to be operated in the display area, or at a position closer to the apparent position of a part of the object in the display area, compared to when the operator has a high degree of ownership of the operation, making it easier for the operator to recognize the specific information.

[0020] Specifically, if the operation target is a shovel, which is an example of a work machine, a portion of the operation target may be, for example, the bucket of the shovel or the arm of the shovel. If the object present at the work site is, for example, a truck, a portion of the object may be the bed of the truck (or a part of the bed).

[0021] An operation assistance device according to a fifth aspect may be the operation assistance device according to any one of the first to fourth aspects, further including the following configuration. That is, in the operation assistance device according to the fifth aspect, the controller may set a field of view of the operator on the display device based on the degree of the sense of ownership of the operation of the operator, and cause the display device to display the specific information so that the specific information is located within the field of view or on the outer edge of the field of view range. In this fifth aspect, since the specific information is displayed within the field of view or on the outer edge of the field of view range, the operator can easily recognize the specific information.

[0022] In this fifth aspect, the controller may set a visual field range of the operator centered on the gaze point based on the degree of the sense of ownership of the operation of the operator. The visual field range can also be expressed as a visual field angle of the operator.

[0023] In this fifth aspect, the controller may set the field of view range so that the field of view range is smaller when the degree of the sense of ownership is low than when the degree of the sense of ownership is high. More specifically, the controller may set the field of view range so that the field of view range becomes smaller as the degree of the sense of ownership becomes lower.

[0024] In this fifth aspect, the controller may set the field of view based on a map that indicates a relationship between the size of the field of view and the degree of the sense of ownership of the operation.

[0025] An operation assistance device according to a sixth aspect preferably includes the following configuration in addition to the operation assistance device according to any one of the first to fifth aspects. That is, the operation assistance device according to the sixth aspect preferably further includes a gaze point acquirer configured to acquire gaze point information related to the gaze point of the operator, and the controller determines the degree of the sense of ownership of the operator based on the time it takes the operator to react to a visual target other than the operation target. In this sixth aspect, even while the operator is operating the operation target, the controller can determine the degree of the sense of ownership based on a gaze reaction time, which is the time it takes the operator to react to the visual target. Furthermore, in this sixth aspect, even if the degree of the sense of ownership changes while the operator is operating the operation target, the controller can recognize the degree of the sense of ownership based on the gaze reaction time. Furthermore, in this sixth aspect, there is no need to measure the amount of head movement as in the technology of Patent Document 4, so even if the head movement of the operator operating the operation object does not occur sufficiently, the controller can determine the degree of the sense of ownership of the operation based on the gaze reaction time.

[0026] The visual target may be, for example, a visual stimulus that stimulates the vision of the operator, and the visual stimulus may be an image that is displayed on the display device so that the controller can determine the degree of the sense of ownership of the operation.

[0027] It is preferable that an operation assistance device according to a seventh aspect further includes the following configuration in addition to the operation assistance device according to the sixth aspect. That is, in the operation assistance device according to the seventh aspect, the visual target may be the specific information displayed on the display device. In this seventh aspect, the controller can determine the degree of the sense of ownership by utilizing the specific information displayed on the display device to assist the operator in operating the operation object. That is, the controller does not need to cause the display device to display information other than the specific information as the visual target for determining the degree of the sense of ownership.

[0028] An operation assistance device according to an eighth aspect is preferably the operation assistance device according to the sixth aspect further including the following configuration. That is, in the operation assistance device according to the eighth aspect, the visual target may be information displayed on the display device by the controller and displayed separately from the specific information, or the visual target may be either an object present at a work site or an object included in a video of the work site displayed on the display device. In this eighth aspect, the controller can determine the degree of sense of ownership by using the information displayed separately from the specific information or the object as the visual target even in a time period other than a time period in which the specific information is displayed.

[0029] An operation assistance device according to a ninth aspect is preferably the operation assistance device according to the sixth aspect further including the following configuration. That is, in the operation assistance device according to the ninth aspect, the visual target may be information acquired through a network. In this ninth aspect, the controller may acquire information such as information about a weather forecast, information about news, or information about the progress of a task through a network such as the Internet, and display the acquired information on the display device. In this way, the controller can not only provide useful information to the operator, but also determine the degree of the sense of ownership of the operation by using the information.

[0030] An operation assistance device according to a tenth aspect is preferably the operation assistance device according to any one of the first to ninth aspects, further comprising the following configuration. That is, in the operation assistance device according to the tenth aspect, the controller may acquire identification information for identifying the operator, and determine the degree of the operator's sense of agency based on information related to the operator's sense of agency stored in a storage device in association with the identification information. In this tenth aspect, the controller can determine the degree of the operator's sense of agency based on the identification information and the information related to the operator's sense of agency stored in association with the identification information, without measuring the time it takes the operator to react to the visual target.

[0031] The information regarding the operator's sense of ownership may be, for example, a previously determined degree of the operator's sense of ownership.

[0032] An operation assistance device according to an eleventh aspect is preferably the operation assistance device according to any one of the first to tenth aspects, further including the following configuration. That is, the operation assistance device according to the eleventh aspect may further include the display device and a controller for the operator to remotely operate the operation object. In this case, the operation assistance device is a remote operation device for remotely operating the operation object. When the operator remotely operates the operation object, the operator cannot directly see the actual work site and needs to understand the situation of the work site by relying on an image of the work site displayed on the display device. For this reason, remote operation is more difficult to operate the operation object than when the operator is on board the operation object and operates the operation object. In this eleventh aspect, during remote operation, the controller determines the display position of the specific information in consideration of the operator's degree of sense of ownership of the operation, i.e., the operator's psychological state, thereby easing the difficulty of the remote operation.

[0033] An operation assistance device according to a twelfth aspect may be the operation assistance device according to any one of the first to eleventh aspects, further including the following configuration. That is, the operation assistance device according to the twelfth aspect may further include a gaze point acquirer configured to acquire gaze point information related to the gaze point of the operator, and the controller may cause the display device to display the specific information so that the specific information is located at a destination of the gaze point or on a path of movement of the gaze point. In this twelfth aspect, since the specific information is displayed at the destination of the gaze point or on the path of movement of the gaze point, the operator can easily recognize the specific information.

[0034] An operation assistance device according to a thirteenth aspect may be the operation assistance device according to any one of the first to twelfth aspects, further including the following configuration. That is, in the operation assistance device according to the thirteenth aspect, the controller may set a display prohibited area in a display area of ​​the display device that prohibits display of the specific information, and cause the display device to display the specific information so that the specific information is located in an area other than the display prohibited area. In this thirteenth aspect, the displayed specific information is less likely to interfere with operation by the operator.

[0035] In this thirteenth aspect, it is more preferable that the controller displays the specific information on the display device so that the specific information is located within the field of view or on the outer edge of the field of view and in an area other than the display prohibited area. In this case, it is possible to maintain a certain degree of ease in recognizing the specific information by the operator and to prevent the displayed specific information from interfering with the operation by the operator.

[0036] An operation assistance device according to a fourteenth aspect may be the operation assistance device according to any one of the first to thirteenth aspects, further including the following configuration. That is, in the operation assistance device according to the fourteenth aspect, when there are multiple candidates to be displayed on the display device as the specific information, the controller may cause the display device to preferentially display, among the multiple candidates, a candidate with a high priority set based on a predetermined condition. If multiple pieces of specific information are displayed on the display device, this may be annoying to the operator, or the operator may overlook some of the multiple pieces of specific information. In this fourteenth aspect, it is possible to reduce the occurrence of such problems while allowing the operator to preferentially recognize specific information with a high priority.

[0037] A manipulation assistance device according to a fifteenth aspect may be the manipulation assistance device according to any one of the first to tenth and twelfth to fourteenth aspects, further including the following configuration. That is, the manipulation assistance device according to the fifteenth aspect may be mounted on the manipulation target. In this fifteenth aspect, the operator riding on the manipulation target can manipulate the manipulation target while viewing the specific information displayed on the display device, the display position of which is determined according to the degree of the operator's sense of ownership of the manipulation.

[0038] An operation assistance system according to a sixteenth aspect includes the operation assistance device according to any one of the first to fifteenth aspects and the operation target.

[0039] An operation assistance system according to a seventeenth aspect may further include the following configuration in addition to the operation assistance system according to the sixteenth aspect. That is, the operation assistance system according to the seventeenth aspect may include a surrounding information detector configured to acquire surrounding information, which is information about the surroundings of the operation target, and the specific information may include at least a part of the surrounding information acquired by the surrounding information detector. In this seventeenth aspect, the operator can easily grasp the surrounding situation (surrounding environment) of the operation target by looking at the specific information displayed on the display device.

[0040] An operation assistance method according to an eighteenth aspect is a method for displaying specific information on a display device to assist an operator in operating an operation object, the operation assistance method including: determining, by a controller, a display position of the specific information on the display device in accordance with a degree of a sense of ownership of the operation by the operator; and displaying, by the controller, the specific information on the display device at the determined display position.

[0041] An operation assistance program according to a nineteenth aspect is a program for displaying, on a display device, specific information for assisting an operator operating an operation object, the operation assistance program causing a computer to execute a process of determining a display position of the specific information on the display device in accordance with a degree of a sense of ownership of the operation by the operator, and a process of displaying the specific information on the display device at the determined display position. [Effects of the Invention]

[0042] According to the present disclosure, when information to assist an operator operating an operation object is displayed on a display device, a technology is provided that can determine the display position of the information taking into account the psychological state of the operator. [Brief explanation of the drawings]

[0043] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an operation assistance system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a side view showing an example of a work machine of the operation support system. [Figure 3] FIG. 2 is a diagram illustrating an example of an operation assistance device of the operation assistance system. [Figure 4] FIG. 2 is a block diagram showing an example of the configuration of the operation support system. [Figure 5] FIG. 2 is a diagram showing an example of a display area (screen) of a display device. [Figure 6] FIG. 10 is a diagram showing another example of the display area (screen) of the display device. [Figure 7]FIG. 10 is a diagram illustrating an example of a map showing the relationship between the degree of a sense of ownership of an operator and the display position of specific information. [Figure 8] FIG. 10 is a diagram showing yet another example of a display area (screen) of a display device. [Figure 9] FIG. 10 is a diagram illustrating an example of a map showing the relationship between the degree of the operator's sense of ownership of the operation and the operator's visual field range. [Figure 10] FIG. 10 is a diagram showing yet another example of a display area (screen) of a display device. [Figure 11] FIG. 10 is a diagram illustrating an example of a map showing the relationship between gaze reaction time and the degree of a sense of ownership of the operation. [Figure 12] FIG. 10 is a diagram showing yet another example of a display area (screen) of a display device. [Figure 13] FIG. 10 is a diagram showing yet another example of a display area (screen) of a display device. [Figure 14] FIG. 10 is a diagram showing yet another example of a display area (screen) of a display device. [Figure 15] FIG. 10 is a block diagram showing the configuration of a modified example of the operation support system. [Figure 16] 4 is a flowchart illustrating an example of a calculation process performed by a controller of the operation assistance device. [Figure 17] 10 is a flowchart showing another example of the calculation process performed by the controller of the operation assistance device. [Figure 18] 10 is a flowchart illustrating yet another example of the arithmetic processing performed by the controller of the operation assistance device. DETAILED DESCRIPTION OF THE INVENTION

[0044] Embodiments of the present disclosure will be described with reference to the drawings.

[0045] [Operation support system] FIG. 1 is a block diagram showing the configuration of an operation assistance system 100 according to an embodiment of the present disclosure. As shown in FIG. 1, the operation assistance system 100 includes a work machine 200 and an operation assistance device 300. The work machine 200 is a machine for performing various tasks at a work site. The operation assistance device 300 is a device for assisting an operator who operates the work machine 200. The operation assistance device 300 according to this embodiment is a remote operation device for remotely operating the work machine 200. This operation assistance device 300 not only has a function for remotely operating the work machine 200, but also has a function for displaying specific information on a display device to assist the operator who remotely operates the work machine 200.

[0046] The operation assistance device 300 is connected to the work machine 200 so that they can communicate with each other via a network 400. The network 400 is an information communication network such as the Internet, a telephone network, a mobile phone network, a satellite communication network, a wide area network (WAN), a local area network (LAN), or a dedicated line. The network 400 may be any one of these communication networks, or may be configured by combining multiple types of communication networks.

[0047] [Work machinery] Fig. 2 is a side view showing a work machine 200. The work machine 200 shown in Fig. 2 is a shovel. The work machine 200 is an example of an operation target in the present disclosure.

[0048] The work machine 200 comprises a lower traveling body 1, an upper rotating body 2 supported on the lower traveling body 1 so as to be rotatable about a vertical rotation axis Z, and a working device 3 supported on the upper rotating body 2. The lower traveling body 1 comprises a lower frame and a traveling device supporting the lower frame. The traveling device includes, for example, a pair of left and right crawlers and a traveling motor that drives the crawlers. The upper rotating body comprises a rotating frame supported on the lower frame, a cabin 2A supported on the rotating frame, and a machine room 2B. The machine room 2B houses an engine, a hydraulic circuit device, etc., and a counterweight is provided at the rear of the machine room 2B. The working device 3 comprises a boom 4 rotatably supported on the upper rotating body 2, an arm 5 rotatably supported at the tip of the boom 4, and a tip attachment 6 rotatably supported at the tip of the arm 5. In the specific example shown in FIG. 2, the end attachment 6 is a bucket, but the end attachment 6 is not limited to a bucket and may be another end attachment such as a crusher or a fork.

[0049] The work machine 200 further includes a plurality of actuators and a drive unit (not shown) that drives the plurality of actuators. The plurality of actuators include a boom cylinder 7 for rotating the boom 4 relative to the upper rotating body 2, an arm cylinder 8 for rotating the arm 5 relative to the boom 4, a tip attachment cylinder 9 for rotating the tip attachment 6 relative to the arm 5, a swing motor 10 for swinging the upper rotating body 2 relative to the lower traveling body 1, and the traveling motor.

[0050] Each of the multiple actuators may be a hydraulic actuator or an electric actuator. When the multiple actuators include hydraulic actuators, the drive device includes a hydraulic circuit including a hydraulic pump that is driven by a drive source such as an engine and discharges hydraulic oil, and a control valve. When the multiple actuators include electric actuators, the drive device includes an electric circuit including a battery.

[0051] The controller 70, which will be described later, may calculate the amount of soil held by the bucket (the amount of excavated soil) from the attitude of the working implement 3 and the moment around the boom foot of the boom cylinder 7. The work machine 200 may be equipped with sensors that measure the rotation amount of each of the boom 4, arm 5, and tip attachment 6, and may be equipped with sensors that measure the extension and contraction amount of each of the boom cylinder 7, arm cylinder 8, and tip attachment cylinder 9. In this case, the controller 70 can calculate the attitude of the working implement 3 based on the rotation amount measured by the sensors and / or the extension and contraction amount measured by the sensors. Furthermore, if the boom cylinder 7 is a hydraulic actuator, the work machine 200 may be equipped with sensors that measure the head pressure and rod pressure of the boom cylinder 7. In this case, the controller 70 can calculate the moment around the boom foot of the boom cylinder 7 using the head pressure and the rod pressure. Furthermore, if the boom cylinder 7 is an electric actuator, the controller 70 can calculate the moment from the output torque of the electric actuator.

[0052] The work machine 200 further includes an operator 35 and a surrounding information detector 30 .

[0053] The controller 35 includes an operating lever, an operating pedal, etc. that are operated by an operator inside the cabin 2A of the work machine 200. The operator can operate the work machine 200 using the operation support device 300 as a remote control device, but can also operate the work machine 200 using the controller 35 inside the cabin 2A of the work machine 200.

[0054] The surrounding information detector 30 is a detector that detects surrounding information, which is information about the surroundings of the work machine 200 at the work site. The surrounding information detector 30 transmits the detected surrounding information via the network 400, and the operation assistance device 300 receives the surrounding information.

[0055] The surrounding information detector 30 includes a camera 31 that captures images of the work site where the work machine 200 is located. The camera 31 is disposed, for example, at a predetermined location on the work machine 200. The predetermined location may be, for example, a location close to the eye height of the operator when he or she is sitting in a seat inside the cabin 2A as shown in Figure 2.

[0056] The surrounding information detector 30 may include an object information detector 32 capable of acquiring information about objects present at the work site. The object information detector 32 may be, for example, a three-dimensional information acquirer capable of acquiring three-dimensional information such as point cloud data of the work site, or may be an object detector such as an infrared sensor capable of detecting objects present at the work site. The three-dimensional information acquirer may be, for example, a LiDAR (Light Detection And Ranging). The object information detector 32 may be arranged at a predetermined location on the work machine 200 so as to acquire information about objects present at the work site around the work machine 200, or may be arranged at the work site where the work machine 200 is located.

[0057] [Operation support device] As shown in FIGS. 1, 3 and 4, an operation assistance device 300 as a remote operation device includes an operation device 50, a display device 40, a gaze point obtaining device 20, a controller 70, and a seat 90.

[0058] The controller 50 includes control levers 51, 52, control pedals, etc. that are operated by an operator OP to operate the work machine 200. The operator OP operates the controller 50 while sitting in the seat 90 to control the work machine 200.

[0059] As shown in Fig. 5, the display device 40 is a display that displays images of the work site captured by the camera 31 of the surrounding information detector 30. The display device 40 is disposed in front of the seat 90. The operator OP sitting in the seat 90 operates the working machine 200 by operating the controller 50 while mainly looking at the display area 41 (screen) of the display device 40. Therefore, the field of view FV that the operator OP can see while operating the working machine 200 is mainly the area in front of the seat 90.

[0060] The gaze point acquirer 20 is configured to acquire gaze point information relating to the gaze point of the operator OP operating the operation device 50. The gaze point acquirer 20 inputs the acquired gaze point information to the controller .

[0061] The gaze point acquirer 20 is configured to detect the line of sight of the operator OP. A commercially available eye tracking device (eye tracker) can be used as the gaze point acquirer 20. In the specific example shown in FIG. 3, the gaze point acquirer 20 is mounted on a head-mounted device worn on the head of the operator OP, but it may also be located at a position away from the head of the operator OP. The gaze point acquirer 20 inputs gaze point information including the line of sight of the operator OP to the controller 70.

[0062] The controller 70 comprises a computer having an arithmetic processing device and a memory. The controller 70 has an operator support function, such as a function to display specific information on the display device 40 to support the operator OP who remotely operates the work machine 200. Specifically, as shown in Fig. 4, for example, the controller 70 comprises a display necessity determination unit 71, an operator subjectivity sense determination unit 72, a display position determination unit 73, and a display command unit 74. The respective functions of the display necessity determination unit 71, the operator subjectivity sense determination unit 72, the display position determination unit 73, and the display command unit 74 are realized by the arithmetic processing device executing a control program stored in the memory.

[0063] The display necessity determining unit 71 determines, based on the peripheral information input from the peripheral information detector 30, whether or not to display specific information 80, which will be described later, on the display device 40.

[0064] The sense of subjectivity of operation determining unit 72 determines the degree of the sense of subjectivity of operation of the operator OP based on the information input from the gaze point obtaining unit 20 and a reference position P0, which will be described later.

[0065] The display position determination unit 73 determines the display position of the specific information 80 in the display area 41 of the display device 40 based on the degree of the operator OP's sense of ownership of the operation.

[0066] The display command unit 74 outputs a command to display the specific information 80 in the display area 41 of the display device 40 at the determined display position. The output command is input to the display device 40, and the display device 40 displays the specific information 80 at the display position according to the command.

[0067] The operation assistance device 300 has the following first feature, and may have at least one of the second to fifteenth features.

[0068] [First feature] The controller 70 of the operation assistance device 300 determines the display position of specific information 80 to be displayed on the display device 40 in order to assist the operator OP in operating the work machine 200, depending on the degree of the operator OP's sense of agency (SoA). The controller 70 then causes the display device 40 to display the specific information 80 at the determined display position. The controller 70 can determine the display position of the specific information 80 on the display device 40, taking into consideration the degree of the operator OP's sense of agency, i.e., the psychological state of the operator OP.

[0069] The sense of ownership is the feeling that the operator OP himself or herself is intentionally causing and controlling the movement of the work machine 200. The degree of ownership represents the level of the sense of ownership.

[0070] 5, the specific information 80 is information relating to safety. This specific information 80 is information relating to an object 85, such as a person, that is present in the vicinity of the work machine 200. The controller 70 can acquire information (information relating to the object 85) that an object 85 is present in the vicinity of the work machine 200 based on the surrounding information detected by the object information detector 32. For example, the controller 70 may cause the display device 40 to display the specific information 80 when an object 85 is detected in a predetermined area based on the work machine 200.

[0071] The display device 40 displays the specific information 80 superimposed at a predetermined position on the image of the work site captured by the camera 31 of the surrounding information detector 30. The predetermined position is a predetermined position in the display area 41. The predetermined position may be a position that avoids the center of the display area 41 (for example, an end such as the left end, right end, top end, or bottom end of the display area 41). The predetermined position may be a position on the display area 41 based on an object reflected in the display area 41, and in this case, if the position of the object reflected in the display area 41 changes, the predetermined position may move in accordance with the position of the object.

[0072] [Second feature] The controller 70 of the operation support device 300 displays the specific information 80 on the display device 40 so that the lower the degree of the sense of ownership of the operation, the closer the specific information 80 is to a reference position P0 on the image displayed on the display device 40. When the degree of the operator OP's sense of ownership of the operation is low, that is, when the effective field of view of the operator OP is narrowed and it becomes difficult for the operator OP to correctly recognize the specific information 80, the display position of the specific information 80 is displayed at a position closer to the reference position P0 compared to when the degree of the sense of ownership of the operation is high, making it easier for the operator OP to recognize the specific information 80.

[0073] For example, if a map such as that shown in FIG. 7 is stored in advance, the controller 70 may determine the display position of the specific information 80 based on the sense of agency of the operator OP and the map shown in FIG. 7. In the graph shown in FIG. 7, the horizontal axis represents the degree of the sense of agency (SoA), and the vertical axis represents the display position of the specific information 80. This display position may be represented by the distance from a reference position P0 of the specific information 80, or may be represented by the angle formed between a line connecting the operator OP's gaze point (e.g., reference position P0) and the operator OP's eyes and a line connecting the display position of the specific information 80 and the operator OP's eyes. The map shown in FIG. 7 represents a relationship in which the display position of the specific information 80 is closer to the reference position P0 when the degree of the sense of agency is low than when the degree of the sense of agency is high.

[0074] 7 shows a relationship in which the display position of specific information 80 becomes a constant value (a relationship in which specific information 80 is displayed at the predetermined position) when the degree of the sense of ownership of operation exceeds a certain value. When the degree of the sense of ownership of operation is equal to or less than a certain value, controller 70 determines the display position of specific information 80 to be closer to reference position P0 as the degree of the sense of ownership of operation becomes lower, thereby causing display device 40 to display specific information 80 at a position closer to reference position P0 from the predetermined position.

[0075] The controller 70 may set the display position of the specific information 80 between the reference position P0 and the object 85 on the image. This makes it easier for the operator OP to see both the specific information 80 and the object 85. The controller 70 may also set the display position of the specific information 80 between the reference position P0 and the estimated position of the object 85 that exists outside the range of the image. This makes it easier for the operator OP to recognize the direction in which the object 85 is located in addition to the specific information 80.

[0076] [Third feature] The reference position P0 may be the position of the operator OP's point of gaze relative to the image displayed on the display device 40. In this case, when the operator OP has a low degree of sense of ownership of the operation, the controller 70 determines the display position of the specific information 80 to be closer to the reference position P0, which is the operator OP's point of gaze, compared to when the operator OP has a high degree of ownership of the operation. This causes the specific information 80 to be displayed at a position closer to the position of the operator OP's point of gaze, making it easier for the operator OP to recognize the specific information 80. The controller 70 can acquire the point of gaze of the operator OP based on information related to the line of sight of the operator OP detected by the point of gaze acquirer 20.

[0077] Furthermore, it is preferable that the controller 70, after displaying the specific information 80 at a display position according to the degree of sense of ownership of the operation, maintains the specific information 80 displayed at that display position until a predetermined condition is satisfied. This prevents the specific information 80 from moving in accordance with the movement of the gaze point of the operator OP until the predetermined condition is satisfied, making it even easier for the operator OP to recognize the specific information 80.

[0078] The predetermined condition may be, for example, the passage of a predetermined time. In this case, the controller 70 displays the specific information 80 at a display position according to the degree of sense of ownership, and then maintains the specific information 80 displayed at that display position until the predetermined time has passed. The start point of measurement of the predetermined time may be, for example, the time when the specific information 80 is displayed on the display device 40, or may be some other timing.

[0079] Furthermore, the predetermined condition may be, for example, the point of gaze of the operator OP moving from within the reference range R0 to outside the reference range R0. In this case, the controller 70 displays the specific information 80 at a display position according to the degree of the sense of ownership of the operation, and then maintains the specific information 80 displayed at the display position until the point of gaze of the operator OP moves from within the reference range R0 to outside the reference range R0.

[0080] The reference range R0 is a range based on the reference position P0 as shown in FIGS. 5 and 6. The reference range R0 may be, for example, a circular range centered on the reference position P0, or an elliptical range centered on the reference position P0. The reference range R0 may be, for example, a field of view FV of the operator OP that is set based on the degree to which the operator OP feels a sense of ownership of the operation. In this case, the controller 70 may set the field of view FV that was set at the time when the specific information 80 was displayed at the display position as the reference range R0.

[0081] [Fourth feature] The reference position P0 may be set to the position of a portion of the work machine 200 included in the image displayed on the display device 40, or the position of a portion of an object included in the image of the work site displayed on the display device 40. The operator OP who operates the work machine 200 that is the operation target may operate the work machine 200 while looking at a portion of the work machine 200 during operation. Furthermore, if the target of the work is an object (for example, the ground) present at the work site, the operator OP who operates the work machine 200 may operate the work machine 200 while looking at a portion of the object during operation. Therefore, when the operator OP has a low degree of sense of ownership of the operation, the specific information 80 is displayed at a position closer to the position of a portion of the work machine 200 or a position closer to the position of a portion of the object, making it easier for the operator OP to recognize the specific information 80, compared to when the degree of the sense of ownership of the operation is high.

[0082] Specifically, the portion of the work machine 200 for which the reference position P0 is set on the image may be, for example, the tip attachment 6 or the arm 5. If the object present at the work site is, for example, a truck, the portion of the object may be the bed of the truck (or a part of the bed). In this embodiment, the reference position P0 is set at a predetermined location (for example, the centroid of the display area of ​​the bucket) of the bucket constituting the tip attachment 6 on the image displayed in the display area 41 of the display device 40. The controller 70 may identify the position of the predetermined location of the bucket from the display area of ​​the bucket on the image, or may identify the position by calculating the position at which the predetermined location is reflected on the image from information related to the attitude of the work implement 3.

[0083] [5th ​​feature] The controller 70 may set the field of view FV of the operator OP based on the degree of the operator OP's sense of ownership of the operation, and may cause the display device 40 to display the specific information 80 so that the specific information 80 is located within the field of view FV as shown in Figures 5 and 6, or may cause the display device 40 to display the specific information 80 so that the specific information 80 is located on the outer edge of the field of view FV as shown in Figure 8. Displaying the specific information 80 within the field of view FV or on the outer edge of the field of view FV makes it easier for the operator OP to recognize the specific information 80.

[0084] The controller 70 may set a field of view FV of the operator OP centered on the gaze point (reference position P0) based on the degree of the operator OP's sense of ownership of the operation. The field of view FV can also be expressed as the field of view angle of the operator OP. In Figures 5, 6, and 8, the field of view FV is a circular or elliptical range centered on the gaze point.

[0085] The controller 70 may set the field of view range FV so that the field of view range FV is smaller when the degree of the sense of ownership of the operation is low than when the degree of the sense of ownership of the operation is high. More specifically, the controller 70 may set the field of view range FV so that the field of view range FV is smaller the lower the degree of the sense of ownership of the operation.

[0086] In this fifth aspect, the controller 70 may set the field of view FV based on a map showing the relationship between the size of the field of view FV and the degree of the sense of ownership of the operator, as shown in FIG. 9, for example. The vertical axis in FIG. 9 represents the size of the field of view FV, which may be, for example, the radius of the field of view FV or the field of view angle. The map shown in FIG. 9 shows the relationship in which the field of view is maximized when the degree of the sense of ownership of the operator exceeds a certain value. When the degree of the sense of ownership is equal to or less than a certain value, the controller 70 sets the field of view FV to be smaller as the degree of the sense of ownership of the operator decreases.

[0087] When the degree of sense of ownership exceeds a certain value, the specific information 80 is likely to be located within the field of view FV. For example, as shown in FIG. 5, the display position of the specific information 80 is included within the field of view FV, making it easier for the operator OP to recognize the specific information 80. When the degree of sense of ownership is equal to or less than a certain value, as shown in FIG. 6, the controller 70 reduces the field of view FV to a value smaller than that shown in FIG. 5 in accordance with the degree of ownership, and further determines the display position of the specific information 80 to be closer to the reference position P0, which is the center of the field of view FV, so that the specific information 80 is included in the field of view FV, and displays the specific information 80 on the display device 40. This makes it easier for the operator OP to recognize the specific information 80. Furthermore, when the controller 70 determines the display position of the specific information 80 to be on the outer edge of the field of view FV as shown in FIG. 8 and displays the specific information 80 on the display device 40, the operator OP can easily recognize the specific information 80 regardless of the size of the field of view FV.

[0088] [6th feature] The controller 70 determines the degree of the sense of ownership of the operator OP based on the time it takes for the operator OP to react to a visual target VT (see FIG. 10 ) in the image displayed on the display device 40, which is separate from the work machine 200. In this case, even if the operator OP is in the midst of operating the work machine 200, the controller 70 can determine the degree of the sense of ownership of the operator OP based on the gaze reaction time, which is the time it takes for the operator OP to react to the visual target VT. Furthermore, in this aspect, even if the degree of the sense of ownership of the operator OP changes while the operator OP is operating the work machine 200, the controller 70 can recognize the degree of the sense of ownership of the operator based on the gaze reaction time. Furthermore, in this aspect, there is no need to measure the amount of head movement as in the technology of Patent Document 4, and therefore the controller 70 can determine the degree of the sense of ownership of the operator OP based on the gaze reaction time, even if the operator OP operating the work machine 200 does not move his or her head sufficiently.

[0089] The gaze target VT may be a visual stimulus that stimulates the vision of the operator OP, as shown in FIG. 10 . The visual stimulus may be a graphic image displayed in the display area 41 of the display device 40 so that the controller 70 can determine the degree of sense of ownership of the operator OP. When measuring the degree of sense of ownership of the operator OP, the controller 70 displays the gaze target VT (visual stimulus) in the display area 41 of the display device 40 as shown in FIG. 10 and starts measuring time from the point in time when the gaze target VT is displayed. Then, based on information about the gaze of the operator OP detected by the gaze point acquirer 20, the controller 70 measures the elapsed time (gaze reaction time) until the gaze point of the operator OP moves within the range of the gaze target VT or within a predetermined range based on the gaze target VT. In this embodiment, the gaze target VT is displayed when the controller 70 has never measured the gaze reaction time or when a predetermined time has passed since the previous measurement of the gaze reaction time. The gaze reaction time is measured by the controller 70 each time the gaze target VT is displayed. The controller 70 evaluates that the shorter the measured gaze reaction time is, the higher the degree of the sense of agency, and the longer the measured gaze reaction time is, the lower the degree of the sense of agency. The controller 70 determines the degree of the sense of agency of the operator OP based on a map showing the relationship between the gaze reaction time and the degree of the sense of agency, as shown in Fig. 11, for example, and the actually measured gaze reaction time. The map shown in Fig. 11 sets the degree of the sense of agency higher when the gaze reaction time is short, and sets the degree of the sense of agency lower when the gaze reaction time is longer.

[0090] [7th feature] The visual target VT may be the specific information 80. In this case, the controller 70 can determine the degree of the sense of ownership of the operation by utilizing the specific information 80 displayed on the display device 40 to assist the operator OP in operating the work machine 200. In other words, the controller 70 does not need to cause the display device 40 to display information other than the specific information 80 (for example, the visual target VT shown in FIG. 10 ) as the visual target VT for determining the degree of the sense of ownership of the operation.

[0091] [8th feature] The visual target VT may be information that the controller 70 causes to be displayed in the display area 41 of the display device 40, and that is separate from the specific information 80 (for example, the visual target VT in FIG. 10 described above), or the visual target VT may be an object that exists at the work site (for example, another work machine, a transport vehicle, etc.), and may be an image of that object included in video of the work site captured by the camera 31, and that the controller 70 causes to be displayed on the display device 40. The controller 70 can determine the degree of sense of ownership by using the separate information or the object as the visual target, even during time periods other than the time period in which the specific information 80 is displayed.

[0092] [9th feature] The visual target VT that the controller 70 causes the display device 40 to display may be information acquired through the network 400. In this case, the controller 70 may acquire information such as information about a weather forecast, information about news, and information about the progress of work, as shown in Fig. 12, through the network 400, and display the acquired information on the display device 40. This not only enables the controller 70 to provide useful information to the operator OP, but also enables the controller 70 to determine the degree of a sense of ownership of the operation by using the information.

[0093] [10th feature] The controller 70 may acquire identification information for identifying the operator OP, and determine the degree of the operator OP's sense of agency based on information relating to the operator OP's sense of agency that is stored in association with the identification information in the storage device 91. In this case, the controller 70 can determine the degree of the operator OP's sense of agency based on the identification information and the information relating to the operator OP's sense of agency that is stored in association with the identification information, without measuring the time it takes for the operator OP to react to the visual target VT.

[0094] The storage device 91 may be included in the operation assistance device 300, or may be included in a device separate from the operation assistance device 300 (for example, a server).

[0095] Furthermore, an operator with high operating skill is more likely to have a higher sense of ownership of the operator than an operator with low operating skill, even when performing the same operation. Therefore, information regarding the sense of ownership of the operator OP may be determined depending on the level of the operator's skill. In this way, even if the degree of the sense of ownership of the operator OP has not been determined in the past, the sense of ownership of the operator OP of an operator with high operating skill can be highly evaluated and the sense of ownership of the operator OP of an operator with low operating skill can be lowly evaluated, thereby appropriately determining the degree of the sense of ownership of the operator OP.

[0096] The information relating to the sense of ownership of the operator OP may be, for example, the degree of the sense of ownership of the operator OP that was previously determined.

[0097] The controller 70 may obtain the identification information using, for example, face recognition technology to identify the operator OP from among multiple operators stored in the memory device 91 based on an image captured by an imaging device that captures the face of the operator OP sitting in the seat 90 provided in the operation support device 300, or may obtain the identification information by having the operator OP input his or her own identification information into an input device (not shown) and determining whether it matches the identification information of any of the multiple operators stored in the memory device 91.

[0098] [11th feature] The operation assistance device 300 is a remote operation device that includes a display device 40 and a controller 50 that enables the operator OP to remotely operate the work machine 200. When the operator OP remotely operates the work machine 200, the operator OP cannot see the actual work site directly and must rely on an image of the work site displayed on the display device 40 to understand the situation at the work site. For this reason, remote operation makes it more difficult to operate the work machine 200 than when the operator OP is on board the work machine 200 and operates the work machine 200. In this eleventh aspect, during remote operation, the controller 70 determines the display position of the specific information 80 taking into consideration the degree to which the operator OP feels ownership of the operation, i.e., the psychological state of the operator OP, thereby making it possible to alleviate the difficulty of remote operation.

[0099] [12th feature] 13, the controller 70 may display the specific information 80 on the display device 40 so that the specific information 80 is located at the destination of the point of gaze or on the path of movement of the point of gaze. Since the specific information 80 is displayed in this way at the destination of the point of gaze or on the path of movement of the point of gaze, the operator OP can easily recognize the specific information 80.

[0100] The controller 70 can calculate the movement of the gaze point of the operator OP over time (movement vector V of the gaze point) based on information regarding the line of sight of the operator OP detected sequentially by the gaze point acquirer 20, and can calculate the destination and movement path of the gaze point of the operator OP after a predetermined time has elapsed based on information regarding the calculated movement vector V.

[0101] [13th feature] The controller 70 may set a display prohibited area RX in which the display of the specific information 80 is prohibited in the display area 41 of the display device 40, and may display the specific information 80 on the display device 40 so that the specific information 80 is located in an area other than the display prohibited area RX. In this case, the displayed specific information 80 is less likely to interfere with operations by the operator OP.

[0102] It is more preferable that the controller 70 displays the specific information 80 on the display device 40 so that the specific information 80 is located within the field of view FV or on the outer edge of the field of view FV, and is located in an area other than the display prohibited area RX. In this case, it is possible to maintain a certain degree of ease in recognizing the specific information 80 by the operator OP, while preventing the displayed specific information 80 from interfering with operations by the operator OP.

[0103] In the specific example shown in FIG. 14 , the controller 70 sets the display prohibited area RX to an area corresponding to the transporter 500 that is to be loaded with excavated soil using a bucket as the tip attachment 6. The controller 70 can recognize the position of an object (prohibited object) such as the transporter 500, for example, using image recognition technology based on machine learning. The controller 70 may also recognize the position of the prohibited object using information related to the attitude of the work implement 3 and image recognition technology based on machine learning. The controller 70 may also recognize the position of the prohibited object using the position of the gaze point of the operator OP, information related to the attitude of the work implement 3, and image recognition technology based on machine learning. When position information of the transporter 500 can be acquired, the controller 70 may also recognize the position of the transporter 500 in the video of the work site captured by the camera 31, based on the position information of the transporter 500 and the orientation of the camera 31 determined from the orientation of the work machine 200.

[0104] [14th feature] When there are multiple candidates to be displayed on the display device 40 as the specific information 80, the controller 70 may preferentially display on the display device 40 a candidate with a high priority that is set based on predetermined conditions among the multiple candidates. If multiple pieces of specific information 80 are displayed on the display device 40, the operator OP may feel annoyed or may overlook some of the multiple pieces of specific information 80, which may cause problems. On the other hand, in this embodiment, it is possible to reduce the occurrence of such problems while allowing the operator OP to preferentially recognize the specific information 80 with a high priority.

[0105] The priorities of the multiple candidates may be set, for example, as follows: First, the controller 70 sets the number (upper limit) of specific information 80 to be displayed on the display device 40. This setting may be a value stored in advance in the storage device 91, or may be based on a value input by the operator OP to the input device.

[0106] The controller 70 may then set the priorities of the multiple pieces of specific information 80 according to the following criteria, for example.

[0107] (1st place) "Information related to safety" (such as proximity or contact between the work machine 200 and surrounding objects detected by the object information detector 32) (2nd place) "Information related to work efficiency" (such as the current volume of excavated soil held by the bucket and the cumulative volume of soil loaded onto the loading target during previous work) (3rd place) "Information about the work machine" (such as the remaining amount of fuel used in the engine of the work machine 200 and the remaining battery charge) The controller 70 causes the display device 40 to display the specific information 80 in descending order of priority, and does not cause the display device 40 to display the specific information 80 whose rank exceeds the upper limit.

[0108] [15th feature] 15, the operation assistance device 300 may be mounted on the work machine 200. In this case, the operator OP riding inside the cabin 2A of the work machine 200 can operate the work machine 200 using the operating device 50 mounted inside the cabin 2A while looking at specific information 80 displayed on the display device 40, the display position of which is determined according to the degree to which the operator OP feels a sense of ownership of the operation. Note that in this aspect, a remote operation device is not an essential component of the operation assistance system 100.

[0109] In the specific example shown in FIG. 15 , the work machine 200 includes an operating device 35, a surrounding information detector 30, a display device 45, a gaze point acquirer 20, and an operation assistance device 300. The operation assistance device 300 includes a controller 70 similar to the controller 70 described above. The display device 45 may be configured, for example, with a projector and a windshield. The projector is installed inside the cabin 2A and projects specific information 80 onto or near the front window of the cabin 2A. The windshield may be installed inside the front window of the cabin 2A. The windshield has a projection surface onto which image light from the projector is projected, and the projection surface reflects the image light and transmits light from outside (the outside world) of the cabin 2A into the interior of the cabin 2A. In this case, the projection surface of the windshield becomes the display area 41. The display device 45 may also be configured, for example, with a transparent display. The transparent display is, for example, installed on top of the front window of the cabin 2A, has a high transmittance that allows light from outside the cabin 2A (the outside world) to pass into the interior of the cabin 2A, and has a display area 41 that has the function of displaying specific information 80.

[0110] In this case, the reference position P0 is the position of the gaze point of the operator OP inside the cabin 2A relative to the display area 41 of the display device 40, which transmits light from the outside (outside world). In this case, similar to when the specific information 80 is displayed on the display device 40 of the operation support device 300 as a remote operation device, the controller 70 determines the display position of the specific information 80 to be closer to the reference position P0, which is the gaze point of the operator OP, when the degree of the operator OP's sense of ownership is low, compared to when the degree of the operator's sense of ownership is high. Also, similar to when the specific information 80 is displayed on the display device 40 of the operation support device 300 as a remote operation device, it is preferable that the controller 70 displays the specific information 80 at a display position corresponding to the degree of the operator's sense of ownership, and then maintains the specific information 80 displayed at that display position until the predetermined condition is satisfied. The gaze point acquirer 20 may be installed inside the cabin 2A.

[0111] Furthermore, the reference position P0 may be set to a position where the line of sight of the operator OP inside the cabin 2A when looking at the position of a portion of the work machine 200 through the display area 41 intersects with the display area 41, or a position where the line of sight of the operator OP inside the cabin 2A when looking at the position of a portion of an object present at the work site where the work machine 200 is located intersects with the display area 41. The operator OP who operates the work machine 200 from inside the cabin 2A may operate the work machine 200 while looking at a portion of the work machine 200 through the display area 41 during operation. Furthermore, if the work target is an object (for example, the ground) present at the work site, the operator OP inside the cabin 2A may operate the work machine 200 while looking at a portion of the object through the display area 41 during operation. Therefore, when the operator OP on board the cabin 2A has a low degree of sense of ownership of the operation, the specific information 80 is displayed at a position closer to the apparent position of a portion of the work machine 200 in the display area 41, or at a position closer to the apparent position of a portion of the object in the display area 41, compared to when the operator OP has a high degree of ownership of the operation, making it easier for the operator OP to recognize the specific information 80.

[0112] Specifically, the portion of the work machine 200 may be, for example, the tip attachment 6 or the arm 5. The controller 70 identifies the position coordinates of the portion of the work machine 200 based on information related to the attitude of the work implement 3, and identifies the position where the line of sight of the operator OP looking at the portion of the work machine 200 intersects with the display area 41 based on the position coordinates of the head of the operator OP inside the cabin 2A, and sets the identified position as the reference position P0. The position coordinates of the head of the operator OP may be changed according to the identification information of the operator OP, which will be described later. If the object present at the work site is, for example, a truck, the portion of the object may be the bed of the truck (or a portion of the bed). In this case, the controller 70 identifies the position coordinates of the bed of the truck based on peripheral information detected by the object information detector 32, and identifies the position where the line of sight of the operator OP looking at the portion of the work machine 200 intersects with the display area 41 based on the position coordinates of the head of the operator OP inside the cabin 2A, and sets the identified position as the reference position P0.

[0113] Furthermore, similar to the case where the specific information 80 is displayed on the display device 40 of the operation support device 300 as a remote operation device, the controller 70 may determine the degree of the operator OP's sense of ownership of the operation based on the time it takes for the operator OP to react to a visual target VT displayed in the display area 41 separately from the work machine 200 visible through the display area 41. The visual target VT may be the specific information 80.

[0114] Furthermore, the visual target VT may be information that the controller 70 causes the display device 40 to display and that is separate from the specific information 80, or the visual target VT may be an object (e.g., another work machine, a transport vehicle, etc.) that is present at the work site and is visible through the display area 41.

[0115] [Processing by the controller of the operation support device] FIG. 16 is a flowchart showing an example of calculation processing by the controller 70 of the operation assistance device 300.

[0116] In step S11, the controller 70 acquires from the surrounding information detector 30 surrounding information about the work machine 200 at the work site.

[0117] In step S12, the controller 70 determines whether or not it is necessary to cause the display device 40 to display the specific information 80.

[0118] Specifically, when an object 85 is detected in a display necessity determination area, which is an area set in advance with the work machine 200 as the reference, the controller 70 determines that it is necessary to display the specific information 80 on the display device 40 (YES in step S12). On the other hand, when an object 85 is not detected in the display necessity determination area, the controller 70 determines that it is not necessary to display the specific information 80 on the display device 40 (NO in step S12).

[0119] If it is determined that it is necessary to display the specific information 80 on the display device 40 (YES in step S12), then in step S13 the controller 70 reads information relating to the operator OP's sense of ownership of the operation, which is stored in association with the identification information of the operator OP in the storage device 91. On the other hand, if it is determined that it is not necessary to display the specific information 80 on the display device 40 (NO in step S12), the controller 70 returns to executing the process of step S11.

[0120] In step S14, the controller 70 determines a display position according to the degree of the operator OP's sense of ownership of the operation, based on the acquired degree of the operator OP's sense of ownership of the operation and the map shown in Fig. 7. The display position may be, for example, a position away from the center of the display area 41 of the display device 40 by the distance determined based on Fig. 7.

[0121] In step S15, the controller 70 outputs a command to the display device 40 to display the specific information 80 at the determined display position. As a result, the display device 40 displays the specific information 80 at the display position in the display area 41 according to the command.

[0122] FIG. 17 is a flowchart showing another example of the calculation process by the controller 70 of the operation assistance device 300.

[0123] Steps S21-S23, S25, and S26 in the flowchart of Fig. 17 are the same as steps S11-S15 in the flowchart of Fig. 16. The flowchart of Fig. 17 differs from the flowchart of Fig. 16 in that it further includes step S24.

[0124] In step S24, the controller 70 acquires a reference position P0. This reference position P0 is, for example, the gaze point of the operator OP with respect to the display area 41 at that time.

[0125] In step S25, the controller 70 determines a display position according to the degree of the operator OP's sense of subjectivity, based on the gaze point of the operator OP, the degree of the operator OP's sense of subjectivity acquired in step S23, and the map shown in Fig. 7. The display position may be, for example, a position away from the gaze point of the operator OP by the distance determined based on Fig. 7.

[0126] FIG. 18 is a flowchart showing yet another example of the calculation process by the controller 70 of the operation assistance device 300. In FIG.

[0127] Steps S33-S35 and S38 in the flowchart of FIG. 18 are the same as steps S11-S13 and S15 in the flowchart of FIG.

[0128] In step S31, the controller 70 measures the movement vector V of the gaze point of the operator OP when the gaze point of the operator OP moves as the specific information 80 is displayed, based on the information of the gaze point of the operator OP acquired from the gaze point acquisition device 20.

[0129] In step S32, the controller 70 determines and stores the degree of a sense of agency (SoA) based on the gaze reaction time of the operator OP in response to the specific information 80.

[0130] In step S36, the controller 70 sets a reference range R0 (for example, the size of the visual field range FV of the operator OP) based on the map shown in FIG. 9 and the degree of the sense of ownership of the operation.

[0131] In step S37, the controller 70 determines a display position on the movement vector V and within the reference range R0 according to the degree of the sense of ownership of the operation.

[0132] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments and includes, for example, the following modified examples.

[0133] [Operation support method] The operation assistance method according to the modified example of the embodiment is a method for displaying specific information 80 on the display device 40 to assist the operator OP in operating the work machine 200 as an operation target. The operation assistance method includes the controller 70 determining the display position of the specific information 80 on the display device 40 in accordance with the degree of sense of ownership of the operation by the operator OP, and the controller 70 causing the display device 40 to display the specific information 80 at the determined display position.

[0134] [Operation support program] The operation assistance program according to the modified example of the embodiment is a program for displaying, on the display device 40, specific information 80 for assisting an operator OP in operating a work machine 200 as an operation target. The operation assistance program causes a computer to execute a process for determining the display position of the specific information 80 on the display device 40 in accordance with the degree of sense of ownership of the operation by the operator OP, and a process for displaying the specific information 80 on the display device 40 at the determined display position. The computer may be the controller 70, or may constitute a part of the controller 70. The operation assistance program may be provided by communication means, or may be provided by being stored on various recording media (for example, non-transitory tangible recording media).

[0135] [Operation support device] 1 includes the operation assistance device 300 according to the embodiment, which is shown in FIG. 1 , the operation device 50, the display device 40, the gaze point acquirer 20, the controller 70, and the seat 90. However, the operation assistance device according to the present disclosure only needs to include at least the controller 70. In other words, the operation assistance device 50, the display device 40, the gaze point acquirer 20, and the seat 90 are not essential components of the operation assistance device according to the present disclosure. [Explanation of symbols]

[0136] 3: Work equipment 20: gaze point acquisition device 30: Surrounding information detector 31: Camera 32: Object information detector 40:Display device 45:Display device 50:Operator 70: Controller 80: Specific information 85 :object 91 :Storage device 100: Operation support system 200: Work machines 300: Operation support device 500: Transport vehicle FV: Field of view OP: Operator P0: Reference position R0: Reference range RX: Display prohibited area V: Movement vector VT: Visible Target

Claims

1. An operation assistance device including a controller that displays specific information on a display device to assist an operator who operates an operation object, The controller determines a display position of the specific information on the display device according to the degree of the operator's sense of ownership of the operation, and causes the display device to display the specific information at the determined display position.

2. The operation assistance device according to claim 1 , wherein the controller determines the display position to be closer to a reference position as the degree of the sense of ownership of the operation decreases.

3. The operation assistance device according to claim 2 , wherein the reference position is a position of the operator's gaze point relative to the display device.

4. 3. The operation assistance device according to claim 2, wherein the reference position is either a position of a portion of the operation target or a position of a portion of an object present at a work site where the operation target is located, or the reference position is either a position of a portion of the operation target included in an image displayed on the display device or a position of a portion of an object included in an image of the work site displayed on the display device.

5. 2. The operation assistance device according to claim 1, wherein the controller sets the field of view of the operator on the display device based on the degree of the operator's sense of ownership of the operation, and causes the display device to display the specific information so that the specific information is located within the field of view or on the outer edge of the field of view.

6. a gaze point acquirer configured to acquire gaze point information relating to the gaze point of the operator, The operation assistance device according to claim 1 , wherein the controller determines the degree of the sense of ownership of the operator based on a time it takes the operator to react to a visual target different from the operation target.

7. The operation assistance device according to claim 6 , wherein the visually recognized object is the specific information displayed on the display device.

8. 7. The operation assistance device according to claim 6, wherein the visual target is information displayed on the display device by the controller and displayed separately from the specific information, or the visual target is either an object present at a work site or an object included in an image of the work site displayed on the display device.

9. The operation assistance device according to claim 6 , wherein the visual target is information acquired through a network.

10. 2. The operation assistance device according to claim 1, wherein the controller acquires identification information that identifies the operator, and determines the sense of ownership of the operator based on information about the sense of ownership of the operator that is stored in association with the identification information in a storage device.

11. the display device; The operation assistance device according to claim 1 , further comprising: an operation device for the operator to remotely operate the operation target.

12. a gaze point acquirer configured to acquire gaze point information relating to the gaze point of the operator, The operation assistance device according to claim 1 , wherein the controller causes the display device to display the specific information so that the specific information is located at a destination of the point of gaze or on a path of movement of the point of gaze.

13. 2. The operation assistance device according to claim 1, wherein the controller sets a display prohibition area in the display area of ​​the display device that prohibits the display of the specific information, and causes the display device to display the specific information so that the specific information is located in an area other than the display prohibition area.

14. 2. The operation assistance device according to claim 1, wherein, when there are multiple candidates to be displayed on the display device as the specific information, the controller causes the display device to preferentially display, among the multiple candidates, a candidate with a high priority set based on a predetermined condition.

15. The operation assistance device according to claim 1 , wherein the operation assistance device is mounted on the operation target.

16. An operation assistance device according to any one of claims 1 to 15; and an operation support system comprising the operation target.

17. a surrounding information detector configured to acquire surrounding information that is information about the surroundings of the operation target, The operation assistance system according to claim 16 , wherein the specific information includes at least a part of the surrounding information acquired by the surrounding information detector.

18. An operation assistance method for displaying specific information on a display device to assist an operator who operates an operation object, comprising: a controller determining a display position of the specific information on the display device in accordance with a degree of a sense of ownership of the operation by the operator; and causing the controller to display the specific information on the display device at the determined display position.

19. An operation assistance program for displaying specific information on a display device to assist an operator who operates an operation object, a process of determining a display position of the specific information on the display device in accordance with a degree of a sense of ownership of the operation by the operator; and a process of displaying the specific information on the display device at the determined display position.

Images