Work machine display system and display method

Abstract

To recognize the line of sight of an operator during operation in training for the operation of a work machine.SOLUTION: A training system receives input of operation data from an operator. The training system detects the line of sight of the operator during the input of the operation data. The training system simulates a change in the posture of a work machine according to the operation data. The training system simultaneously displays a posture image showing the simulated change in the posture of the work machine and a line-of-sight image showing a change of the line of sight.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present disclosure relates to a display system and a display method for a work machine. [Background technology]

[0002] Patent Document 1 discloses a technique in which an operator simulates the operation of a work machine while referring to the operation of the work machine by himself or another operator. The technique described in Patent Document 1 plays back video images of past work performed by a remotely operated work machine, and simulates the work in a work environment related to the playback position when the video images were stopped. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-103193 A Summary of the Invention [Problem to be solved by the invention]

[0004] When operating a work machine, where you look is important, so in addition to training on how to operate the machine, you also need to be trained on how to look.

[0005] An object of the present disclosure is to provide a display system and a display method for a work machine that are capable of recognizing the line of sight of an operator during operation, during training in the operation of the work machine. [Means for solving the problem]

[0006] According to one aspect of the present invention, a work machine display system accepts input of operation data by an operator, detects the operator's line of sight when inputting the operation data, simulates a change in the attitude of the work machine in accordance with the operation data, and simultaneously displays a posture image representing the simulated change in attitude of the work machine and a gaze image representing the change in the gaze. Effect of the Invention

[0007] According to the above aspect, in training for operating a work machine, it is possible to recognize the line of sight of the operator during operation. [Brief description of the drawings]

[0008] [Figure 1] 1 is a schematic diagram showing a configuration of a training system according to a first embodiment. [Diagram 2] 1 is a block diagram showing a software configuration of a training simulator according to a first embodiment. [Diagram 3] 4 is a flowchart showing a method for creating model data according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of an input screen for setting data according to the first embodiment. [Diagram 5] 4 is a flowchart (part 1) showing a training implementation method according to the first embodiment. [Figure 6] 4 is a flowchart (part 2) showing a method for implementing training according to the first embodiment. [Figure 7] 4 is a flowchart (part 3) showing a method for implementing training according to the first embodiment. [Figure 8] 4 is an example of image data when rendering the entire ghost machine according to the first embodiment. [Figure 9] 13 is an example of image data when rendering only a bucket of a ghost machine according to the first embodiment. [Figure 10]6 is an example of image data when displaying a trajectory of a blade tip of a bucket of a ghost machine according to the first embodiment. [Figure 11] 13 is an example of image data when displaying a lever operation of a trainer according to the first embodiment. [Figure 12] 13 is an example of image data when displaying the line of sight of a trainer according to the first embodiment. [Figure 13] 4 is an example of image data when displaying a side view according to the first embodiment. [Figure 14] FIG. 4 is a diagram illustrating an example of a trajectory model according to the first embodiment. [Figure 15] 4 is a flowchart showing a method for evaluating work data by a trainer according to the first embodiment. [Figure 16] 5 is a flowchart showing a method for a trainee to view an evaluation result according to the first embodiment. [Figure 17] FIG. 1 is a schematic block diagram illustrating a configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] First Embodiment <<Configuration of Training System 1>> Hereinafter, the embodiments will be described in detail with reference to the drawings. FIG. 1 is a schematic diagram showing the configuration of a training system 1 according to the first embodiment. The training system 1 according to the first embodiment is a system that allows a trainee, who is an operator inexperienced in operating the work machine 100, to simulate the operation of the work machine 100 while referring to the operation of a trainer, who is an operator skilled in operating the work machine 100. The trainee can receive evaluation of his / her own operation from the training system 1 and the trainer. Both the trainer and the trainee are operators of the work machine 100. In the first embodiment, the training system 1 is used for training in the operation of a hydraulic excavator, which is the work machine 100. Note that the type of work machine 100 is not limited to a hydraulic excavator, and may be other work machines such as a bulldozer, a wheel loader, or a forklift.

[0010] The training system 1 includes a data server 10 and one or more training simulators 30.

[0011] The data server 10 stores model data and work data used for training. The model data represents the operation of the work machine 100 by the trainer. The trainee performs training with reference to the model data. The work data represents the operation of the work machine 100 by the trainee. The training system 1 and the trainer evaluate the trainee's operation based on the work data. The model data and work data include a time series of the operation of the operating lever, a time series of the posture of the work machine 100 (joint angles and turning angles of the work machine), and a time series of the line of sight. Hereinafter, data representing a time series of the operation and behavior of the work machine 100, such as the model data and work data, is also referred to as behavior record data.

[0012] Specifically, the data server 10 includes a model data table T1 for storing model data, a work data table T2 for storing work data, and a user table T3 for storing authentication data. The model data table T1 stores model data, a model ID that is an ID (identification information) of the model data, and summary data that indicates the work content reproduced by the model data, in association with each other. The work data table T2 stores, in association with each other, work data, a work ID which is the ID (identification information) of the work data, a model ID of the corresponding model data, evaluation data indicating the evaluation by the training system 1, and comment data indicating the evaluation by the trainer. The user table T3 stores a user ID, a user category, and authentication information in association with each other. The value of the user category is either trainer or trainee. The authentication information may be, for example, a password.

[0013] The training simulator 30 receives operation input from the operator and simulates the behavior of the work machine 100 in response to the operation input. The training simulator 30 generates an image showing the behavior of the work machine 100 and presents it to the operator. The training simulator 30 also reproduces the behavior of the work machine 100 based on the model data stored in the data server 10, generates an image showing the behavior of the work machine 100, and presents it to the operator.

[0014] The training simulator 30 includes a calculation device 31, an operation device 33, and a head mounted display 35. The operation device 33 is an input interface for operating the work machine 100. The operation device 33 according to the first embodiment is two operation levers (a right operation lever and a left operation lever). The operation device 33 may differ depending on the type of the work machine 100 to be simulated. The head mounted display 35 displays an image calculated by the calculation device 31. The head mounted display 35 includes an eye tracker that detects the line of sight of the wearer and an IMU (Inertial Measurement Unit) that detects the attitude of the head mounted display 35. The calculation device 31 performs a simulation of the work machine 100 in the virtual space V based on the input of the operation device 33. The calculation device 31 determines the direction of the line of sight in the virtual space V based on the attitude of the head mounted display 35, and renders the simulation results. As a result, the virtual space V is displayed on the head mounted display 35 in conjunction with the attitude of the head mounted display 35.

[0015] Configuration of the work machine 100 The work machine 100 includes a running body 110, a rotating body 120, and a work implement 130. The running body 110 supports the work machine 100 so that the work machine 100 can run. The rotating body 120 is supported by the running body 110 so as to be rotatable about a rotation center. A cab 121 is provided at the front of the rotating body 120. A rendering camera for rendering the virtual space V by the computing device 31 is provided in the cab 121. The work machine 130 is supported at the front of the revolving body 120 so as to be drivable in the vertical direction. The work machine 130 includes a boom 131, an arm 132, and a bucket 133 as a work tool. For example, the work tool has a cutting edge extending in the width direction. Other examples of the work tool include end attachments such as a clam bucket, tilt bucket, tilt rotate bucket, breaker, and grappler.

[0016] The base end of the boom 131 is rotatably attached to the revolving body 120 via a boom pin. In the work machine 100 shown in FIG. 1, the boom 131 is provided at the center of the front of the revolving body 120, but the present invention is not limited to this and the boom 131 may be attached offset in the left-right direction. In this case, the center of rotation of the revolving body 120 is not located on the operating plane of the work implement 130. The arm 132 connects the boom 131 and the bucket 133. A base end of the arm 132 is rotatably attached to a tip end of the boom 131 via an arm pin. The bucket 133 is rotatably attached via a pin to the tip of the arm 132. The bucket 133 functions as a container for storing excavated soil and sand.

[0017] Configuration of Training Simulator 30 FIG. 2 is a block diagram showing the software configuration of the arithmetic device 31 included in the training simulator 30 according to the first embodiment. The calculation device 31 provided in the training simulator 30 includes an input unit 311, an acquisition unit 312, a reproduction unit 313, a simulator 314, a rendering unit 315, a display control unit 316, a generation unit 317, an evaluation unit 318, a transmission unit 319, a setting memory unit 320, a comment unit 321, and an authentication unit 322.

[0018] The input unit 311 acquires operation data of the operating device 33, posture data measured by the IMU of the head mounted display 35, and gaze data measured by the eye tracker of the head mounted display 35. The gaze data is represented by a direction based on the display surface of the head mounted display 35. It should be noted that the absolute gaze direction can be specified by combining the posture data and the gaze data. Hereinafter, the gaze data measured by the eye tracker is referred to as primary gaze data, and the gaze data indicating the absolute gaze direction specified from the posture data and the gaze data is referred to as secondary gaze data.

[0019] The acquisition unit 312 acquires behavior record data (model data and work data) from the data server 10. The reproducing unit 313 reproduces the behavior of the work machine 100 based on the behavior record data acquired by the acquiring unit 312. Hereinafter, the work machine 100 reproduced in the virtual space V by the reproducing unit 313 is referred to as a ghost machine 100G. The reproducing unit 313 reproduces the behavior of the work machine 100 by arranging the ghost machine 100G according to the time series of the joint angles of the work implement 130 and the rotation angle of the rotating body 120 included in the behavior record data. Note that the reproducing unit 313 according to other embodiments may, for example, simulate the behavior of the work machine 100 based on time series data of the movement of the operation lever included in the behavior record data, and arrange the ghost machine 100G.

[0020] The simulator 314 simulates the behavior of the work machine 100 based on the operation data input to the input unit 311. Hereinafter, the work machine 100 simulated by the simulator 314 is referred to as the avatar machine 100A. The simulator 314 simulates the behavior of the avatar machine 100A by calculating the angular velocities of the revolving body 120 and the work machine 130 according to the operation amount indicated by the operation data.

[0021] The rendering unit 315 renders the ghost machine 100G and the avatar machine 100A, and generates image data. The rendering camera in the virtual space is located in the driver's cab 121 of the avatar machine 100A. The rendering camera is directed toward the front of the head mounted display 35 indicated by the posture data input to the input unit 311, based on the front of the avatar machine 100A. Note that rendering by the rendering unit 315 is performed at a predetermined frame rate. Therefore, the image data generated by the rendering unit 315 is treated as frame images of a moving image.

[0022] The display control unit 316 outputs the image data rendered by the rendering unit 315 to the head mounted display 35.

[0023] The generation unit 317 generates behavior record data based on the operation data, posture data, and primary gaze data input to the input unit 311, and the posture of the avatar machine 100A simulated by the simulator 314. Specifically, the generation unit 317 obtains the positions and postures of the running body 110, the revolving body 120, the boom 131, the arm 132, and the bucket 133 of the avatar machine 100A in the virtual space based on the posture of the avatar machine 100A simulated by the simulator 314. Each position and posture is expressed in a virtual space coordinate system, which is a three-dimensional rectangular coordinate system that defines the virtual space. The generation unit 317 generates secondary gaze data indicating the direction of the gaze in the virtual space coordinate system based on the positions and postures of the running body 110, the revolving body 120, the boom 131, the arm 132, and the bucket 133 in the virtual space, as well as the posture data and the primary gaze data.

[0024] The evaluation unit 318 evaluates the operation by the trainee based on the difference between the model data acquired by the acquisition unit 312 and the work data generated by the generation unit 317. For example, the evaluation unit 318 calculates the position and posture of the revolving unit 120 in the virtual space indicated by the model data and the work data, as well as the distance between the posture data and the secondary gaze data. The evaluation unit 318 calculates the distance between the model data and the work data for each of the time series of the positions and postures of the traveling unit 110, the revolving unit 120, the boom 131, the arm 132, and the bucket 133, the time series of the posture data, and the time series of the secondary gaze data, and produces a weighted sum of the distances as an evaluation value. In this case, the closer the evaluation value is to zero, the higher the evaluation. The distance in the time series may be calculated using the DTW method.

[0025] The transmission unit 319 transmits the behavior record data generated by the generation unit 317 to the data server 10 .

[0026] The setting storage unit 320 stores the setting data of the training simulator 30. The setting data stores the playback speed of the ghost machine 100G, whether or not to display the ghost machine 100G, whether or not to display the trajectory of the blade tip of the ghost machine 100G's tool, whether or not to display a side view, whether or not to display the trainer's lever operation, and whether or not to display the trainer's line of sight. The setting data can be updated by the trainee's operation. The side view is an image that displays the virtual space from the side of the avatar machine 100A. The side view is drawn in the cab.

[0027] The comment section 321 accepts input of comments for the work data. The comment can be associated with a playback timing. This allows the comment section 321 to be set so that the comment is displayed at a predetermined playback timing when the work data is played back.

[0028] The authentication unit 322 authenticates the operator of the training simulator 30 based on the authentication information stored in the data server 10.

[0029] 《Training System 1 Processing》 When the training simulator 30 is started, the authentication unit 322 displays a login screen on the head mounted display 35. The login screen displays, for example, an input form for a user ID and authentication information. When the operator inputs the user ID and the authentication information value on the login screen, the authentication unit 322 obtains the authentication information associated with the input user ID from the user table T3 of the data server 10, and verifies the authentication information.

[0030] If the authentication unit 322 succeeds in the matching, the display control unit 316 causes the head mounted display 35 to display a menu screen. When the user category of the operator is a trainer, a trainer menu is displayed on the menu screen for allowing the operator to select either the creation of model data or the evaluation of work data. If the user category of the operator is a trainee, a trainee menu is displayed on the menu screen for allowing the operator to select from among carrying out training, viewing evaluation results, and changing settings.

[0031] <<Generation of model data>> FIG. 3 is a flowchart showing a method for creating model data according to the first embodiment. When a trainer logs in to the training simulator 30 and selects creation of model data on the menu screen, the simulator 314 places the avatar machine 100A in an initial posture and a field in the virtual space (step S101).

[0032] The input unit 311 acquires operation data of the operation device 33, posture data measured by the IMU of the head mounted display 35, and primary gaze data measured by the eye tracker of the head mounted display 35 (step S102).

[0033] The simulator 314 simulates the behavior of the avatar machine 100A after a predetermined frame time based on the operation data input in step S102 (step S103). At this time, the simulator 314 calculates the positions and attitudes of the traveling body 110, the rotating body 120, the boom 131, the arm 132, and the bucket 133.

[0034] The simulator 314 determines the position and attitude of the rendering camera in the virtual space based on the position and attitude of the revolving body 120 and the attitude data input in step S102 (step S104). The rendering unit 315 renders the virtual space in which the avatar machine 100A is placed from the rendering camera determined in step S104 (step S105). The display control unit 316 outputs the image data rendered in step S105 to the head mounted display 35 (step S106).

[0035] The generation unit 317 generates secondary gaze data indicating the direction of the trainer's gaze in the virtual space based on the position and attitude of the rendering camera calculated in step S104 and the primary gaze data acquired in step S102 (step S107). The generation unit 317 generates one frame of behavior record data based on the operation data acquired in step S102, the attitude of the avatar machine 100A calculated in step S104, and the secondary gaze data generated in step S107 (step S108).

[0036] The generation unit 317 judges whether the operation by the trainer has ended (step S109). When the trainer judges that the operation is to end, the trainer instructs the operation to end by performing a predetermined operation on the training simulator 30, such as moving the trainer's gaze to an end button on the screen. If the operation by the trainer has not ended (step S109: NO), the process returns to step S102, and the simulation of the next frame is performed.

[0037] On the other hand, when the operation by the trainer is finished (step S109: YES), the generation unit 317 compiles the generated behavior record data of the multiple frames as model data (step S110). The generation unit 317 also accepts input of summary data for the model data from the trainer (step S111). The transmission unit 319 transmits the model data and summary data to the data server 10 (step S112). As a result, the data server 10 assigns a model ID to the received model data, and records the model ID, model data, and summary data in association with each other in the model data table T1.

[0038] Change settings When the trainee logs in to the training simulator 30 and selects to change the settings on the menu screen, the display control unit 316 reads out the setting data stored in the setting storage unit 320 and outputs an input screen for the setting data to the head mounted display 35. FIG. 4 is a diagram showing an example of an input screen for the setting data according to the first embodiment. The input screen includes an input form for the playback speed of the ghost machine 100G, whether or not to display the ghost machine 100G, whether or not to display the trajectory of the blade tip of the ghost machine 100G, whether or not to display a side view, whether or not to display the trainer's lever operation, and whether or not to display the trainer's line of sight, and a decision button. The setting data can be updated by the trainee's operation. When the display of the ghost machine 100G is ON in the input form, an input form for whether or not to display only the bucket 133 of the ghost machine 100G is displayed on the input screen. When the decision button is operated by the trainee's operation, the setting storage unit 320 updates the setting data according to the value input to the input screen.

[0039] 《Training Implementation》 Fig. 5 is a flowchart (part 1) showing a method for implementing training according to the first embodiment. Fig. 6 is a flowchart (part 2) showing a method for implementing training according to the first embodiment. Fig. 7 is a flowchart (part 3) showing a method for implementing training according to the first embodiment. When the trainee logs in to the training simulator 30 and selects to carry out training on the menu screen, the acquisition unit 312 accesses the data server 10 and acquires a list of model data recorded in the model data table T1 (step S201). The display control unit 316 outputs a selection screen for selecting one model data from the acquired list to the head mounted display 35 (step S202). At this time, the selection screen displays a list of model data and summary data indicating the contents of the model data. The trainee reads the summary data and selects one model data.

[0040] The acquiring unit 312 accesses the data server 10 and acquires the model data selected by the trainee from the model data table T1 (step S203). The simulator 314 places the avatar machine 100A, the ghost machine 100G, and a field in the virtual space (step S204). The avatar machine 100A and the ghost machine 100G are placed overlappingly at the same position.

[0041] The input unit 311 acquires operation data of the operation device 33, posture data measured by the IMU of the head mounted display 35, and primary gaze data measured by the eye tracker of the head mounted display 35 (step S205).

[0042] The simulator 314 simulates the behavior of the avatar machine 100A after a predetermined frame time based on the operation data input in step S205 (step S206). At this time, the simulator 314 calculates the positions and orientations of the traveling body 110, the revolving body 120, the boom 131, the arm 132, and the bucket 133. The simulator 314 calculates the position and orientation of the rendering camera in the virtual space based on the position and orientation of the revolving body 120 and the orientation data input in step S205 (step S207).

[0043] The reproduction unit 313 reproduces the behavior of the ghost machine 100G by calculating the posture of the ghost machine 100G based on the time series of the joint angles of the work machine 130 and the rotation angle of the rotating body 120 contained in the model data acquired in step S203 and the playback speed of the ghost machine 100G indicated by the setting data (step S208).

[0044] The reproducing unit 313 judges whether or not the setting data indicates that the ghost machine 100G should be displayed (step S209). If the setting data indicates that the ghost machine 100G should be displayed (step S209: YES), the reproducing unit 313 judges whether or not the setting data indicates that only the bucket 133 of the ghost machine 100G should be displayed (step S210). If the setting data indicates that only the bucket 133 should be displayed (step S210: NO), the reproducing unit 313 determines that all the components of the ghost machine 100G are to be rendered (step S211). In this case, the image data is drawn as shown in FIG. 8. FIG. 8 is an example of image data when rendering the entire ghost machine 100G according to the first embodiment. If it is set that only the bucket 133 should be displayed (step S210: YES), the reproducing unit 313 determines the bucket 133 of the ghost machine 100G as the rendering target, and excludes components other than the bucket 133 of the ghost machine 100G from the rendering targets (step S212). In this case, the image data is rendered as shown in Fig. 9. Fig. 9 is an example of image data when rendering only the bucket 133 of the ghost machine 100G according to the first embodiment. If it is not set that the ghost machine 100G should be displayed (step S209: NO), the reproducing unit 313 excludes the ghost machine 100G from the rendering target (step S213).

[0045] Next, the reproducing unit 313 judges whether or not the setting data indicates that the trajectory of the blade tip of the ghost machine 100G should be displayed (step S214). If the setting indicates that the trajectory of the blade tip of the ghost machine 100G should be displayed (step S214: YES), the reproducing unit 313 generates a blade tip object Obj1 indicating the trajectory of the blade tip, and places the blade tip object Obj1 at both ends and the center of the blade tip of the bucket 133 of the ghost machine 100G calculated in step S208 (step S215). The blade tip object Obj1 may be, for example, a small sphere. The blade tip object Obj1 may correspond to one or more points on the blade tip, for example, and each blade tip object Obj1 represents the trajectory of each point. The reproducing unit 313 increases the transparency of the blade tip object Obj1 generated in the past by a predetermined value (step S216). As a result, the multiple cutting edge objects Obj1 represent the trajectory of the cutting edge position of the bucket 133 at the most recent predetermined time, and the more recent the position, the more transparent the cutting edge object is displayed. In another embodiment, the reproduction unit 313 may change the size of the cutting edge object Obj1 instead of the transparency. The cutting edge object Obj1 whose transparency exceeds a predetermined threshold may be removed from the virtual space. For example, the cutting edge object Obj1 whose transparency exceeds a predetermined threshold may not be displayed. In this case, the image data is rendered as shown in FIG. 10. FIG. 10 is an example of image data when displaying the trajectory of the cutting edge of the bucket 133 of the ghost machine 100G according to the first embodiment. The multiple cutting edge objects Obj1 are rendered as curves representing the trajectory of the cutting edge. If it is not set that the trajectory of the blade tip of the ghost machine 100G should be displayed (step S214: NO), the reproduction unit 313 does not place the blade tip object Obj1 in the virtual space. The blade tip object Obj1 may have different colors at the left end, center, and right end of the blade tip. For example, the blade tip object Obj1 may be drawn at either the left end, center, or right end of the blade tip. The blade tip object Obj1 may also be represented as a band-like trajectory along which a line segment runs from the left end to the right end of the blade tip.

[0046] Next, the reproducing unit 313 judges whether or not the setting data indicates that the trainer's lever operation should be displayed (step S217). If the setting indicates that the lever operation should be displayed (step S217: YES), the reproducing unit 313 places the lever object Obj2 at a position a predetermined distance away from the rendering camera position determined in step S207 in the direction in which the rendering camera faces (step S218).

[0047] 11 is an example of image data when displaying the trainer's lever operation according to the first embodiment. The lever object Obj2 includes two reference circles Obj21 representing the left and right operating levers, two operation points Obj22 representing the input positions of the left and right operating levers by the trainee, and a plurality of plots Obj23 showing the input positions of the levers by the trainer. The two reference circles Obj21 are arranged with a predetermined interval in the left-right direction. The lever object Obj2 is arranged so that the reference circles face the rendering camera.

[0048] Based on the operation data included in the model data, the reproducing unit 313 places the plot Obj23 at a position away from the center of the reference circle Obj21 by the operation amount at the time of reproduction in step S208, and moves the operation point Obj22 to a position away from the center of the reference circle Obj21 by the operation amount indicated by the operation data acquired in step S205 (step S219). In other words, the lever object Obj2 represents a change in the position of the operation lever when the operation lever is viewed in plan from above the operation plane of the operation lever. The reproducing unit 313 reduces the size of the plot Obj23 placed in the past by a predetermined value (step S220). As a result, the multiple plots Obj23 represent the trajectory of the lever operation at the most recent predetermined time. Note that in another embodiment, the reproducing unit 313 may change the transparency instead of the size of the plot Obj23. The plot Obj23 whose size is below a predetermined threshold may be removed from the virtual space. If it is not set that the lever operation should be displayed (step S217: NO), the reproducing unit 313 does not place the lever object Obj2.

[0049] Next, the reproduction unit 313 judges whether or not the setting data indicates that the trainer's line of sight should be displayed (step S221). If the setting indicates that the line of sight should be displayed (step S221: YES), the reproduction unit 313 places the line of sight object Obj3 in the virtual space (step S222). The line of sight object Obj3 is placed at a position a predetermined distance away from the position of the rendering camera determined in step S207 in the direction indicated by the secondary line of sight data. The line of sight object Obj3 may be, for example, a small sphere. The reproduction unit 313 increases the transparency of the line of sight object Obj3 placed in the past by a predetermined value (step S223). As a result, the multiple line of sight objects Obj3 represent the trajectory of the trainer's line of sight at the most recent predetermined time. Note that in other embodiments, the reproduction unit 313 may change the size or hue of the plot Obj23 instead of the transparency. The line of sight object Obj3 whose transparency exceeds a predetermined threshold may be removed from the virtual space. For example, when the transparency of the line of sight object Obj3 exceeds a predetermined threshold, the reproduction unit 313 may not display the line of sight object Obj3. Fig. 12 is an example of image data when displaying the line of sight of the trainer according to the first embodiment. If it is not set that the line of sight should be displayed (step S221: NO), the reproducing unit 313 does not place the line of sight object Obj3 in the virtual space.

[0050] The rendering unit 315 determines whether or not the setting data indicates that the side view should be displayed (step S224). If the setting indicates that the side view should be displayed (step S224: YES), the rendering unit 315 places the rendering camera to the side of the avatar machine 100A so as to face the direction of the avatar machine 100A (step S225). The rendering unit 315 generates a side-view image Obj4 by drawing the virtual space in which the avatar machine 100A and the ghost machine 100G are placed (step S226). The simulator 314 places the side-view image Obj4 at a predetermined position inside the cab 121 (step S227). FIG. 13 shows an example of image data when the side view according to the first embodiment is displayed.

[0051] Next, the rendering unit 315 renders the virtual space in which the avatar machine 100A and the ghost machine 100G are placed from the rendering camera with the position and orientation obtained in step S207 (step S228). The display control unit 316 outputs the image data rendered in step S228 to the head mounted display 35 (step S229). Since the rendering unit 315 generates image data for each frame time, the image data is treated as a frame image of a moving image. Moreover, the rendering unit 315 can be said to have generated moving image data that simultaneously displays the first moving image in which the avatar machine 100A and the second moving image in which the ghost machine 100G are placed by rendering the virtual space in which the avatar machine 100A and the ghost machine 100G are placed. Moreover, the rendering unit 315 can be said to have generated moving image data that displays the first moving image in which the avatar machine 100A is placed and the second moving image in which the ghost machine 100G is placed in the same virtual space by overlapping them.

[0052] The generation unit 317 generates secondary gaze data indicating the direction of the trainee's gaze in the virtual space based on the position and posture of the rendering camera calculated in step S207 and the primary gaze data acquired in step S205 (step S230). The generation unit 317 generates one frame of behavior record data based on the operation data acquired in step S205, the posture of the avatar machine 100A calculated in step S206, and the secondary gaze data generated in step S230 (step S231).

[0053] The generating unit 317 determines whether the training has ended (step S232). For example, the generating unit 317 may determine that the training has ended when a certain time has elapsed since the playback position of the model data reached the end of the time series. If the training has not ended (step S232: NO), the processing returns to step S205, and the simulation of the next frame is performed.

[0054] On the other hand, when the training is completed (step S232: YES), the generation unit 317 compiles the generated behavior record data of the multiple frames as work data (step S233). The evaluation unit 318 calculates an evaluation value for evaluating the operation by the trainee based on the difference between the work data generated in step S233 and the model data acquired in step S203 (step S234). The transmission unit 319 transmits the work data generated in step S233, the model ID of the model data acquired in step S203, and the evaluation value calculated in step S234 to the data server 10 (step S235). As a result, the data server 10 assigns a work ID to the received work data, and records the work ID, work data, model ID, and evaluation value in the work data table T2 in association with each other. Furthermore, the display control unit 316 outputs an evaluation screen displaying the evaluation values ​​calculated in step S234 to the head mounted display 35 (step S236). The evaluation screen may display evaluation values ​​for each of the turning operation, the operation of the work machine 130, and the line of sight.

[0055] The evaluation unit 318 generates a trajectory model, which is a three-dimensional model in which the movement trajectory of the cutting edge from the start to the end of the training by the trainee's operation shown in the work data and the movement trajectory of the cutting edge from the start to the end of the training by the trainer's operation shown in the model data are arranged in a three-dimensional space. FIG. 14 is a diagram showing an example of the trajectory model according to the first embodiment. The display control unit 316 may rotatably display the trajectory model generated by the evaluation unit 318 on the head-mounted display 35. This allows the display control unit 316 to simultaneously display the movement trajectory of the cutting edge by the trainee's operation and the movement trajectory of the cutting edge by the trainer's operation. The rotation of the trajectory model may be performed by changing the attitude of the head-mounted display 35 or by operating the operation device 33.

[0056] <Work data evaluation> FIG. 15 is a flowchart showing a method for evaluating work data by a trainer according to the first embodiment. When the trainer logs in to the training simulator 30 and selects to evaluate work data on the menu screen, the acquisition unit 312 accesses the data server 10 and acquires a list of work data recorded in the work data table T2 (step S301). The display control unit 316 outputs a selection screen for selecting one work data from the acquired list to the head mounted display 35 (step S302). The trainer selects one work data.

[0057] The acquisition unit 312 accesses the data server 10 and acquires the work data selected by the trainer from the work data table T2 (step S303). The reproduction unit 313 places the ghost machine 100G and the field in the virtual space (step S304).

[0058] The reproducing unit 313 reproduces the behavior of the ghost machine 100G by calculating the attitude of the ghost machine 100G based on the time series of the joint angles of the work machine 130 and the rotation angle of the revolving body 120 included in the work data acquired in step S303 (step S305). The reproducing unit 313 calculates the position and attitude of the rendering camera in the virtual space based on the position and attitude of the revolving body 120 of the ghost machine 100G and the attitude data input in step S305 (step S306).

[0059] The reproducing unit 313 places the lever object Obj2 at a position a predetermined distance away from the rendering camera position determined in step S306 in the direction in which the rendering camera faces (step S307). This enables the trainer to check the lever operation of the trainee. The reproducing unit 313 places the line-of-sight object Obj3 in the virtual space (step S308). The line-of-sight object Obj3 is placed at a position a predetermined distance away from the position of the rendering camera determined in step S306 in the direction indicated by the secondary line-of-sight data.

[0060] The rendering unit 315 renders the virtual space in which the ghost machine 100G is placed from the rendering camera having the position and orientation determined in step S306 (step S309). The display control unit 316 outputs the image data rendered in step S309 to the head mounted display 35 (step S310).

[0061] The comment unit 321 determines whether or not a comment has been input from the trainer at the current playback timing (step S311). If a comment has been input, comment data is generated that associates the input comment with the playback timing of the comment (step S312).

[0062] The reproducing unit 313 judges whether the reproduction of the work data has ended (step S313). If the reproduction has not ended (step S313: NO), the process returns to step S305, and the simulation of the next frame is performed.

[0063] On the other hand, when the playback has ended (step S313: YES), the transmission unit 319 transmits the work ID of the work data acquired in step S303 and the comment data generated in step S312 to the data server 10 (step S314). As a result, the data server 10 adds the received comment data to the data string associated with the received work ID in the work data table T2.

[0064] Viewing the evaluation results FIG. 16 is a flowchart showing a method for a trainee to view an evaluation result according to the first embodiment. When the trainee logs in to the training simulator 30 and selects to view the evaluation results on the menu screen, the acquisition unit 312 accesses the data server 10 and acquires a list of work data recorded in the work data table T2 (step S401). The list acquired at this time may be a list of work data created by the trainee who has logged in. The display control unit 316 outputs a selection screen for selecting one piece of work data from the acquired list to the head mounted display 35 (step S402). The trainee selects one piece of work data. The acquisition unit 312 accesses the data server 10 and acquires the work data selected by the trainee from the work data table T2 (step S403). The reproduction unit 313 places the ghost machine 100G and a field in the virtual space (step S404).

[0065] The reproducing unit 313 reproduces the behavior of the ghost machine 100G by calculating the attitude of the ghost machine 100G based on the time series of the joint angles of the work machine 130 and the rotation angle of the revolving body 120 included in the work data acquired in step S403 (step S405). The reproducing unit 313 determines the position and attitude of the rendering camera in the virtual space based on the position and attitude of the revolving body 120 of the ghost machine 100G and the attitude data input in step S405 (step S406).

[0066] The reproducing unit 313 places the lever object Obj2 at a position a predetermined distance away from the rendering camera position determined in step S406 in the direction in which the rendering camera faces (step S407). The reproducing unit 313 places the line of sight object Obj3 in the virtual space (step S408).

[0067] The rendering unit 315 renders the virtual space in which the ghost machine 100G is placed from the rendering camera with the position and orientation determined in step S406 (step S409). The comment unit 321 determines whether or not a comment associated with the current playback timing exists in the comment data included in the work data (step S410). If a comment associated with the current playback timing exists (step S410: YES), the comment is superimposed on the rendered image data (step S411). The display control unit 316 outputs the image data rendered in step S409 or step S411 to the head mounted display 35 (step S412).

[0068] The reproducing unit 313 judges whether the reproduction of the work data is completed (step S413). If the reproduction is not completed (step S413: NO), the process returns to step S405 and the simulation of the next frame is performed. On the other hand, if the reproduction is completed (step S413: YES), the training simulator 30 ends the process. Training system 1 can also play back work data to which no evaluation has been given by a trainer, in the procedure shown in Fig. 16. In this case, since there are no comments, the comment superimposition process in step S411 is skipped entirely.

[0069] Actions and Effects In this way, the training system 1 according to the first embodiment executes the following processes. The training system 1 accepts input of first operation data by a trainee who is a first operator. The training system 1 simulates a change in posture of the avatar machine 100A which is a first work machine according to the first operation data. The training system 1 reproduces the behavior of the ghost machine 100G which is a second work machine based on model data generated based on second operation data previously input by a trainer who is a second operator. The training system 1 simultaneously displays a first moving image showing a change in posture of the avatar machine 100A and a second moving image showing the behavior of the ghost machine 100G. In this way, the training system 1 allows the trainee to train while comparing his / her own operation with the operation performed by the trainer. In other words, the first operator and the second operator may be the same operator. By comparing past operation data with current operation data, the same operator can check the degree of improvement in operation or look back on past operations.

[0070] The model data for the training system 1 according to the first embodiment includes a time series of the posture of the work machine 100 simulated according to the second operation data. This eliminates the need for the training system 1 to perform a simulation using the simulator 314 in order to reproduce the behavior of the ghost machine 100G. In other words, the training system 1 can reproduce the behavior of the ghost machine 100G by simply changing the posture of the ghost machine 100G according to the posture indicated by the time series. Note that in other embodiments, the model data may not include a time series of the posture of the work machine 100, and may be represented by a time series of operation data. In this case, the training system 1 can reproduce the behavior of the ghost machine 100G by performing a simulation according to the time series of the operation data.

[0071] The model data of the training system 1 according to the first embodiment includes a time series of the trainer's line of sight when the second operation data is input. This allows the training system 1 to present the trainee with not only the operation method of the work machine 100, but also points that the trainee should pay attention to. For example, the points that the trainee should pay attention to are points of gaze that should be paid attention to when operating the work machine 100.

[0072] The training system 1 according to the first embodiment records work data and comment data from a trainer in association with each other. The training system 1 simultaneously displays a moving image that reproduces a change in the posture of the work machine 100 based on the work data and the comment data associated with the work data. This allows the trainee to receive feedback from others on his or her own operation. In particular, the training system 1 according to the first embodiment displays the comment data when the playback timing of the video image coincides with the playback timing related to the comment data, thereby enabling the trainee to recognize which operation, among a series of operations of the work machine 100, requires attention.

[0073] The training system 1 according to the first embodiment evaluates the trainee's operation based on the difference between the first operation data related to the work data and the second operation data related to the model data, thereby enabling the trainee to objectively recognize the magnitude of the deviation between the operation and the model data.

[0074] The training system 1 according to the first embodiment accepts an input of the playback speed of the second moving image related to the model data from the trainee, and simultaneously displays the first moving image in which the avatar machine 100A appears and the second moving image in which the ghost machine 100G appears and is played at the specified playback speed. This allows the trainee to train according to his / her own skill level. That is, when the trainee is not familiar with the operation, the trainee can train while carefully checking the trainer's operation by moving the ghost machine 100G at a low playback speed. On the other hand, when the trainee becomes familiar with the operation, the trainee can train to approach the trainer's actual operation by moving the ghost machine 100G at a playback speed close to 1x. In addition, the trainee can train under more difficult conditions than the actual machine by moving the ghost machine 100G at a playback speed faster than 1x. In this case, the trainee can operate the actual machine carefully by feeling the time it takes to operate it longer.

[0075] The training system 1 according to the first embodiment executes the following process. The training system 1 accepts input of first operation data by a trainee who is a first operator. The training system 1 simulates a change in posture of an avatar machine 100A which is a first work machine according to the first operation data. The training system 1 reproduces the operation of the trainer according to second operation data previously input by a trainer who is a second operator. The training system 1 simultaneously displays a first moving image which represents a change in posture of the avatar machine 100A and a lever object Obj2 which represents the operation of the trainer. This allows the trainee to perform a simulation of the work machine 100 while visually checking the operation by the trainer. Furthermore, according to the first embodiment, an operation point Obj22 which represents the trainee's lever operation and a plot Obj23 which represents the trainer's lever operation are simultaneously displayed as the lever object Obj2. This allows the trainee to perform a simulation of the work machine 100 while visually checking the operation by the trainer.

[0076] The lever object Obj2 according to the first embodiment includes a reference circle Obj21, which is a reference position image showing the reference position of the operating lever, and a plot Obj23 showing the position of the operating lever after operation. This allows the trainee to easily recognize the input position of the operating lever based on the positional relationship between the reference circle Obj21 and the plot Obj23.

[0077] The lever object Obj2 according to the first embodiment includes a plurality of plots Obj23 indicating past positions of the operating lever. The older the plots Obj23 are, the smaller or more transparent they are. This allows the trainee to recognize the trajectory of the operating lever, and prevents confusion between the current input position of the operating lever and the past input position.

[0078] The lever object Obj2 according to the first embodiment is located forward in the line of sight direction on the head mounted display 35. This allows the trainee to visually recognize the operation by the trainer regardless of the direction he is facing.

[0079] The training system 1 according to the first embodiment executes the following process. The training system 1 accepts input of first operation data by a trainee who is a first operator. The training system 1 simulates a change in posture of an avatar machine 100A which is a first work machine according to the first operation data. The training system 1 reproduces a change in posture of a ghost machine 100G which is a second work machine based on model data generated based on second operation data input by a second operator in the past. The training system 1 simultaneously displays a first moving image showing a change in posture of the avatar machine 100A and a second moving image showing a change in posture of the ghost machine 100G. At this time, the initial positions of the avatar machine 100A and the ghost machine 100G are the same. That is, the training system 1 displays the avatar machine 100A and the ghost machine 100G in a superimposed manner. This allows the trainee to easily recognize the difference between his / her operation and the trainer's operation. For example, it is easier to recognize the difference in behavior compared to when the avatar machine 100A and the ghost machine 100G are displayed side by side.

[0080] The moving image of the avatar machine 100A in the training system 1 according to the first embodiment is an image in which the avatar machine 100A is rendered from a viewpoint located in the driver's cab 121 of the avatar machine 100A. The moving image of the ghost machine 100G in the training system 1 according to the first embodiment is an image in which the ghost machine 100G is rendered from a viewpoint located in the driver's cab 121 of the avatar machine 100A. In this way, the training system 1 allows the trainee to easily recognize the difference between his / her own operation and the trainer's operation from the viewpoint from the driver's cab 121 of the avatar machine 100A.

[0081] The training system 1 according to the first embodiment can render only the bucket 133, which is the implement of the ghost machine 100G. This can prevent the revolving body 120, boom 131, and the like of the ghost machine 100G from interfering with the avatar machine 100A and reducing visibility. On the other hand, the training system 1 renders the entire ghost machine 100G, allowing the trainee to more clearly recognize the difference in posture between the avatar machine 100A and the ghost machine 100G.

[0082] The ghost machine 100G according to the first embodiment has a higher transparency than the avatar machine 100A. This allows the training system 1 to prevent the ghost machine 100G from interfering with the simulation. The ghost machine 100G according to other embodiments may have a higher transparency than the avatar machine 100A and a different hue from the avatar machine 100A. This makes it easier to distinguish the ghost machine 100G from the avatar machine 100A. Furthermore, the ghost machine 100G according to other embodiments may be displayed only by a contour line.

[0083] The training system 1 according to the first embodiment executes the following process. The training system 1 accepts input of first operation data by a trainee who is a first operator. The training system 1 simulates a change in posture of an avatar machine 100A which is a first work machine according to the first operation data. The training system 1 reproduces a change in posture of a ghost machine 100G which is a second work machine based on model data generated based on second operation data previously input by a second operator. The training system 1 simultaneously displays the avatar machine 100A and a cutting edge object Obj1 which represents a trajectory of a bucket 133 which is a work tool of the ghost machine 100G. This allows the trainee to easily recognize the difference between his / her own operation and the trainer's operation.

[0084] The training system 1 according to the first embodiment places cutting edge objects Obj1 corresponding to each of a plurality of points on the cutting edge. Each cutting edge object Obj1 represents the trajectory of each point. By providing cutting edge objects Obj1 for a plurality of points on the cutting edge, the trainee can easily recognize the position of the cutting edge of the bucket 133 in the depth direction.

[0085] The blade tip object Obj1 according to the first embodiment has a different color for each of the multiple points, which reduces the possibility of confusing and recognizing the multiple blade tip objects Obj1.

[0086] The training system 1 according to the first embodiment executes the following processing. The training system 1 accepts input of operation data by an operator. The training system 1 detects the line of sight of the operator when inputting the operation data. The training system 1 simulates a change in the posture of the work machine 100 in accordance with the operation data. The training system 1 simultaneously displays a posture image representing the simulated change in posture of the work machine 100 and a gaze object Obj3 representing the change in gaze. This allows the training system 1 to display the relationship between the operation of the work machine 100 by the operator and the gaze direction. This allows the trainee to recognize points to watch from the trainer's gaze during operation. This also allows the trainer to evaluate the driving based on the trainee's gaze.

[0087] In the first embodiment, the cutting edge object Obj1, the plot Obj23 of the lever object Obj2, and the line of sight object Obj3 are a plurality of points that represent a trajectory. On the other hand, in other embodiments, some or all of these objects may be curves that represent a trajectory.

[0088] Second embodiment In the training system 1 according to the first embodiment, the behavior of the ghost machine 100G is continuously reproduced at a reproduction speed specified in the setting data. On the other hand, when the trainee is a beginner, there is a possibility that the operation cannot keep up with the behavior of the ghost machine 100G. In such a case, if the ghost machine 100G is continuously reproduced and the postures of the avatar machine 100A and the ghost machine 100G differ greatly, the trainee may lose sight of the indicators of the operation. The training system 1 according to the second embodiment enables the implementation of appropriate training even when the trainee's operation is slow.

[0089] The training system 1 according to the first embodiment reproduces the behavior of the ghost machine 100G for each frame time in step S208 as shown in Fig. 5. In contrast to this, in the second embodiment, the following process is executed instead of step S208.

[0090] The reproducing unit 313 calculates the distance between the posture of the avatar machine 100A calculated in step S206 and the posture of the ghost machine 100G at the previous time. The distance between the postures can be calculated, for example, by the square root of the sum of the squares of the difference in the rotation angle, the difference in the boom angle, the difference in the arm angle, and the difference in the bucket angle. If the calculated distance exceeds a threshold value previously set in the setting data, the reproducing unit 313 maintains the posture of the ghost machine 100G. On the other hand, if the calculated distance does not exceed a threshold value previously set in the setting data, the reproducing unit 313 reproduces the behavior of the ghost machine 100G by calculating the posture of the ghost machine 100G based on the time series of the joint angle of the work machine 130 and the rotation angle of the revolving body 120 included in the model data and the playback speed of the ghost machine 100G indicated by the setting data.

[0091] Thus, according to the second embodiment, the training system 1 suspends reproduction of the behavior of the ghost machine 100G when the difference between the posture of the simulated avatar machine 100A and the posture of the ghost machine 100G exceeds a predetermined threshold. For example, when suspending reproduction of the behavior of the ghost machine 100G, the training system 1 may store an image of the moment of suspension. In this way, the training system 1 according to the second embodiment enables appropriate training to be carried out even when the trainee's operation is slow.

[0092] Other Embodiments Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes are possible. That is, in other embodiments, the order of the above-mentioned processes may be changed as appropriate. Also, some of the processes may be executed in parallel.

[0093] The training system 1 according to the embodiment described above includes the data server 10 and the training simulator 30, but is not limited thereto. For example, the training system 1 according to another embodiment may be configured with the training simulator 30 alone. In this case, the training can be performed based on model data recorded in advance in the training simulator 30. In addition, the training system 1 according to another embodiment may perform training while referring to work data operated by the trainee in the past. In addition, in another embodiment, a part of the training simulator 30 may be provided in an external computer. For example, in another embodiment, the training simulator 30 may only be responsible for inputting operation data and displaying the calculation results, and the simulation calculation may be performed by an external device.

[0094] The training system 1 according to the embodiment described above uses the head mounted display 35 as a display device, but is not limited to this. For example, in another embodiment, a large display used for remote operation of the work machine 100 may be used as a display device. At this time, the input unit 311 may receive designation of the position and attitude of the rendering camera from the operator. Examples of designation of the position and attitude of the rendering camera include a subjective viewpoint (driver's viewpoint), an objective viewpoint (diagonally behind the driver), an overhead viewpoint (to the side of the work machine 100), and a free viewpoint. The rendering unit 315 renders the virtual space according to the designated position and attitude of the rendering camera. In addition, the display control unit 316 may receive an instruction to replay (start over) or pause when playing back the work data. In addition, the display control unit 316 may display a seek bar indicating the playback position, and receive an instruction to cue to an arbitrary point by operating the seek bar.

[0095] Computer Configuration FIG. 17 is a schematic block diagram illustrating a configuration of a computer according to at least one embodiment. The computer 90 comprises a processor 91, a main memory 92, a storage 93, and an interface 94. The above-mentioned training simulator 30 is implemented in a computer 90. The operations of the above-mentioned processing units are stored in the form of a program in a storage 93. The processor 91 reads the program from the storage 93, loads it in a main memory 92, and executes the above-mentioned processes in accordance with the program. The processor 91 also secures storage areas in the main memory 92 corresponding to the above-mentioned storage units in accordance with the program. Examples of the processor 91 include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and a microprocessor.

[0096] The program may be for realizing a part of the functions to be performed by the computer 90. For example, the program may be for realizing the functions by combining with other programs already stored in the storage or by combining with other programs implemented in other devices. In another embodiment, the computer 90 may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to the above configuration or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, a part or all of the functions realized by the processor 91 may be realized by the integrated circuit. Such an integrated circuit is also included in an example of a processor. In another embodiment, the computer 90 may be virtualized on one or more computers.

[0097] Examples of the storage 93 include a magnetic disk, a magneto-optical disk, an optical disk, a semiconductor memory, etc. The storage 93 may be an internal medium directly connected to the bus of the computer 90, or an external medium connected to the computer 90 via the interface 94 or a communication line. In addition, when the program is distributed to the computer 90 via a communication line, the computer 90 that receives the program may load the program into the main memory 92 and execute the above-mentioned processing. In at least one embodiment, the storage 93 is a non-transitory tangible storage medium.

[0098] The program may be for realizing part of the above-mentioned functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the above-mentioned functions in combination with another program already stored in the storage 93. [Explanation of symbols]

[0099] 1...Training system 10...Data server 100...Working machine 100A...Avatar machine 100G...Ghost machine 110...Traveling body 120...Rotating body 121...Operator's cab 130...Working machine 131...Boom 132...Arm 133...Bucket 30...Training simulator 31...Calculation device 311...Input unit 312...Acquisition unit 313...Reproduction unit 314...Simulator 315...Rendering unit 316...Display control unit 317...Generation unit 318...Evaluation unit 319...Transmission unit 320...Settings storage unit 321...Comment unit 322...Authentication unit 33...Operation device 35...Head mounted display 90...Computer 91...Processor 92...Main memory 93...Storage 94...Interface Obj1...Blade tip object Obj2...Lever object Obj21...Reference circle Obj22...Operation point Obj23...Plot Obj3...Gaze object Obj4...Side view image T1...Model data table T2...Work data table T3...User table V...Virtual space

Claims

1. A display system for a work machine, comprising: Accepting input of operation data by an operator; Detecting a line of sight of the operator when inputting the operation data; simulating a change in attitude of the work machine in accordance with the operation data; A posture image showing a change in the posture of the simulated working machine and a line of sight image showing a change in the line of sight are simultaneously displayed. Display system.

2. The line of sight image includes a curve or a plurality of points representing a line of sight trajectory. The display system of claim 1 .

3. The curve or the points included in the line of sight image have a higher transparency as they represent an initial line of sight. The display system of claim 2 .

4. A portion of the curve or the plurality of points included in the line of sight image whose transparency exceeds a predetermined threshold is not displayed. The display system of claim 3.

5. accepting an input of first operation data by a first operator who is the operator; simulating a change in attitude of a first work machine in accordance with the first operation data; receiving an input of second operation data by a second operator who is the operator before the input of the first operation data; Detecting a line of sight of the second operator when the second operation data is input; simulating a change in attitude of a second work machine in accordance with the second operation data; A first moving image showing a simulated change in posture of the first work machine and a second moving image including the posture image showing the simulated change in posture of the second work machine and the line of sight image showing the change in the line of sight are simultaneously displayed. A display system according to any one of claims 1 to 4.

6. accepting an input of first operation data by a first operator who is the operator; detecting a first line of sight, which is a line of sight of the first operator when the first operation data is input; simulating a change in attitude of a first work machine in accordance with the first operation data; receiving an input of second operation data by a second operator who is the operator before the input of the first operation data; detecting a second line of sight that is a line of sight of the second operator when the second operation data is input; Evaluating an operation of the first operator based on a difference between the first line of sight and the second line of sight. A display system according to any one of claims 1 to 4.

7. A computer of a training system for conducting training on the operation of a work machine receives input of operation data by an operator, Detecting a line of sight of the operator when inputting the operation data; simulating a change in attitude of the work machine in accordance with the operation data; A posture image showing a change in the posture of the simulated working machine and a line of sight image showing a change in the line of sight are simultaneously displayed. Display method.

Images