Display system and display method for work machine

Abstract

This display system receives an input of first operation data by a first operator. The display system simulates a change in the orientation of a work machine in accordance with the first operation data. The display system reproduces an operation by a second operator on the basis of model data including second operation data input by the second operator in the past. The display system simultaneously displays a first video representing the simulated change in the orientation of the work machine and a second video representing the reproduced operation by the second operator.

Description

Display system and display method for work machine

[0001] This application claims priority to Japanese Patent Application No. 2023-099484, filed on June 16, 2023, the contents of which are incorporated herein by reference.

[0002] Patent Document 1 discloses a technique in which an operator simulates the operation of a work machine while referring to the operation of the work machine by the operator or another operator. The technique described in Patent Document 1 plays back video images of past work performed by a remotely controlled work machine, and simulates the work in a work environment related to the playback position when the video images are stopped.

[0003] Japanese Patent Application Laid-Open No. 2021-103193

[0004] In training simulations, it is necessary to simultaneously check past operation records of a work machine while simulating the operation. For example, a novice trainee can efficiently learn operation by following the operation of an expert trainer. Also, for example, an intermediate trainee can notice small differences in operation by recognizing the differences between his or her own operation and that of the expert trainer in real time.

[0005] An object of the present disclosure is to provide a display system and display method for a work machine that allows an operator to easily compare operations with other operations during training in the operation of the work machine.

[0006] According to one aspect of the present invention, a work machine display system accepts input of first operation data by a first operator, simulates changes in the posture of the work machine in accordance with the first operation data, reproduces the operation by the second operator based on model data including second operation data previously input by a second operator, and simultaneously displays a first moving image representing the simulated change in posture of the work machine and a second moving image representing the reproduced operation by the second operator.

[0007] According to the above aspect, in training for operating a work machine, the operation by the operator can be easily compared with other operations.

[0008] FIG. 1 is a schematic diagram showing the configuration of a training system according to a first embodiment. FIG. 2 is a block diagram showing the software configuration of a training simulator according to the first embodiment. FIG. 3 is a flowchart showing a method of creating model data according to the first embodiment. FIG. 4 is a diagram showing an example of an input screen for setting data according to the first embodiment. FIG. 5 is a flowchart (part 1) showing a method of conducting training according to the first embodiment. FIG. 6 is a flowchart (part 2) showing a method of conducting training according to the first embodiment. FIG. 7 is a flowchart (part 3) showing a method of conducting training according to the first embodiment. FIG. 8 is an example of image data when rendering the entire ghost machine according to the first embodiment. FIG. 9 is an example of image data when rendering only the bucket of the ghost machine according to the first embodiment. FIG. 10 is an example of image data when displaying the trajectory of the cutting edge of the bucket of the ghost machine according to the first embodiment. FIG. 11 is an example of image data when displaying the lever operation of a trainer according to the first embodiment. FIG. 12 is an example of image data when displaying the line of sight of the trainer according to the first embodiment. FIG. 13 is a diagram showing an example of a trajectory model according to the first embodiment. FIG. 14 is a flowchart showing a method of evaluating work data by a trainer according to the first embodiment. FIG. 15 is a flowchart showing a method of viewing evaluation results by a trainee according to the first embodiment. FIG. 16 is a schematic block diagram showing the configuration of a computer according to at least one embodiment.

[0009] <First Embodiment> <Configuration of Training System 1> Hereinafter, embodiments will be described in detail with reference to the drawings. FIG. 1 is a schematic diagram showing the configuration of a training system 1 according to a first embodiment. The training system 1 according to the first embodiment is a system that enables a trainee, who is an operator unfamiliar with operating a work machine 100, to simulate the operation of the work machine 100 while referring to the operations of a trainer, who is an operator skilled in operating the work machine 100. The trainee can have his or her own operation evaluated by the training system 1 and the trainer. Both the trainer and the trainee are operators of the work machine 100. In the first embodiment, the training system 1 is used for training in the operation of a hydraulic excavator, which is the work machine 100. Note that the type of work machine 100 is not limited to a hydraulic excavator, and may be other work machines such as a bulldozer, a wheel loader, or a forklift.

[0010] The training system 1 includes a data server 10 and one or more training simulators 30 .

[0011] The data server 10 stores model data and work data used for training. The model data represents the operation of the work machine 100 by the trainer. The trainee performs training using the model data as a reference. The work data represents the operation of the work machine 100 by the trainee. The training system 1 and the trainer evaluate the trainee's operation based on the work data. The model data and work data include a time series of operation of the control levers, a time series of the attitude of the work machine 100 (joint angles and swing angles of the work machine), and a time series of line of sight. Hereinafter, data that represents a time series of the operation and behavior of the work machine 100, such as the model data and work data, will also be referred to as behavior record data.

[0012] Specifically, the data server 10 includes a model data table T1 for storing model data, a work data table T2 for storing work data, and a user table T3 for storing authentication data. The model data table T1 stores model data, a model ID that is the ID (identification information) of the model data, and summary data that indicates the work content reproduced by the model data, in association with each other. The work data table T2 stores work data, a work ID that is the ID (identification information) of the work data, the model ID of the corresponding model data, evaluation data indicating an evaluation by the training system 1, and comment data indicating an evaluation by the trainer, in association with each other. The user table T3 stores user IDs, user categories, and authentication information in association with each other. The user category value is either trainer or trainee. The authentication information may be, for example, a password.

[0013] The training simulator 30 accepts operation inputs from the operator and simulates the behavior of the work machine 100 in response to the operation inputs. The training simulator 30 generates and presents to the operator an image showing the behavior of the work machine 100. The training simulator 30 also reproduces the behavior of the work machine 100 based on model data stored in the data server 10, generates and presents to the operator an image showing the behavior of the work machine 100.

[0014] The training simulator 30 includes a computing device 31, an operation device 33, and a head-mounted display 35. The operation device 33 is an input interface for operating the work machine 100. The operation device 33 according to the first embodiment is two operation levers (a right operation lever and a left operation lever). The operation device 33 may differ depending on the type of work machine 100 being simulated. The head-mounted display 35 displays images calculated by the computing device 31. The head-mounted display 35 includes an eye tracker that detects the line of sight of the wearer and an IMU (Inertial Measurement Unit) that detects the attitude of the head-mounted display 35. The computing device 31 simulates the work machine 100 in a virtual space V based on input from the operation device 33. The computing device 31 determines the direction of the line of sight in the virtual space V based on the attitude of the head-mounted display 35 and renders the simulation results. As a result, the virtual space V is displayed on the head-mounted display 35 in conjunction with the attitude of the head-mounted display 35.

[0015] <Configuration of Work Machine 100> The work machine 100 comprises a traveling body 110, a revolving body 120, and a work implement 130. The traveling body 110 supports the work machine 100 so that it can travel. The revolving body 120 is supported on the traveling body 110 so that it can rotate about a rotation center. A cab 121 is provided at the front of the revolving body 120. A rendering camera for rendering the virtual space V by the computing device 31 is provided within the cab 121. The work implement 130 is supported at the front of the revolving body 120 so that it can be driven in the vertical direction. The work implement 130 comprises a boom 131, an arm 132, and a bucket 133 as a work implement. For example, the work implement has a cutting edge that extends across the width. Other examples of work implements include end attachments such as a clam bucket, tilt bucket, tilt rotate bucket, breaker, and grappler.

[0016] The base end of the boom 131 is rotatably attached to the revolving unit 120 via a boom pin. Note that, in the work machine 100 shown in FIG. 1 , the boom 131 is provided in the center portion of the front of the revolving unit 120, but this is not limitative and the boom 131 may be attached offset in the left-right direction. In this case, the center of rotation of the revolving unit 120 is not located on the operating plane of the work implement 130. The arm 132 connects the boom 131 and the bucket 133. The base end of the arm 132 is rotatably attached to the tip of the boom 131 via an arm pin. The bucket 133 is rotatably attached to the tip of the arm 132 via a pin. The bucket 133 functions as a container for collecting excavated soil and sand.

[0017] 2 is a block diagram showing the software configuration of the arithmetic device 31 included in the training simulator 30 according to the first embodiment. The arithmetic device 31 included in the training simulator 30 includes an input unit 311, an acquisition unit 312, a reproduction unit 313, a simulator 314, a rendering unit 315, a display control unit 316, a generation unit 317, an evaluation unit 318, a transmission unit 319, a setting storage unit 320, a comment unit 321, and an authentication unit 322.

[0018] The input unit 311 acquires operation data from the operating device 33, posture data measured by the IMU of the head-mounted display 35, and gaze data measured by the eye tracker of the head-mounted display 35. The gaze data is represented by a direction based on the display surface of the head-mounted display 35. Note that an absolute gaze direction can be identified by combining the posture data and the gaze data. Hereinafter, the gaze data measured by the eye tracker will be referred to as primary gaze data, and the gaze data indicating the absolute gaze direction identified from the posture data and gaze data will be referred to as secondary gaze data.

[0019] The acquisition unit 312 acquires behavior record data (model data and work data) from the data server 10. The reproduction unit 313 reproduces the behavior of the work machine 100 based on the behavior record data acquired by the acquisition unit 312. Hereinafter, the work machine 100 reproduced in the virtual space V by the reproduction unit 313 will be referred to as the ghost machine 100G. The reproduction unit 313 reproduces the behavior of the work machine 100 by arranging the ghost machine 100G in accordance with the time series of the joint angles of the work implement 130 and the swing angle of the rotating body 120 included in the behavior record data. Note that the reproduction unit 313 according to other embodiments may, for example, simulate the behavior of the work machine 100 based on time series data of the movement of the operation lever included in the behavior record data, and arrange the ghost machine 100G.

[0020] Simulator 314 simulates the behavior of work machine 100 based on the operation data input to input unit 311. Hereinafter, work machine 100 simulated by simulator 314 will be referred to as avatar machine 100A. Simulator 314 simulates the behavior of avatar machine 100A by calculating the angular velocities of revolving unit 120 and work implement 130 according to the operation amount indicated by the operation data.

[0021] The rendering unit 315 renders the ghost machine 100G and the avatar machine 100A and generates image data. The rendering camera in the virtual space is located in the driver's cab 121 of the avatar machine 100A. The rendering camera is directed toward the front of the head-mounted display 35 indicated by the posture data input to the input unit 311, based on the front of the avatar machine 100A. Note that rendering by the rendering unit 315 is performed at a predetermined frame rate. Therefore, the image data generated by the rendering unit 315 is treated as frame images of a moving image.

[0022] The display control unit 316 outputs the image data rendered by the rendering unit 315 to the head-mounted display 35 .

[0023] The generation unit 317 generates behavior record data based on the operation data, posture data, and primary gaze data input to the input unit 311, and the posture of the avatar machine 100A simulated by the simulator 314. Specifically, the generation unit 317 obtains the positions and postures of the traveling unit 110, the revolving unit 120, the boom 131, the arm 132, and the bucket 133 of the avatar machine 100A in the virtual space based on the posture of the avatar machine 100A simulated by the simulator 314. Each position and posture is represented in a virtual space coordinate system, which is a three-dimensional Cartesian coordinate system that defines the virtual space. The generation unit 317 generates secondary gaze data indicating the direction of the gaze in the virtual space coordinate system based on the positions and postures of the traveling unit 110, the revolving unit 120, the boom 131, the arm 132, and the bucket 133 in the virtual space, as well as the posture data and the primary gaze data.

[0024] The evaluation unit 318 evaluates the trainee's operation based on the difference between the model data acquired by the acquisition unit 312 and the work data generated by the generation unit 317. For example, the evaluation unit 318 calculates the position and attitude of the revolving unit 120 in the virtual space indicated by the model data and the work data, as well as the distance between the attitude data and the secondary line of sight data. The evaluation unit 318 calculates the distance between the model data and the work data for each of the time series of the positions and attitudes of the traveling unit 110, the revolving unit 120, the boom 131, the arm 132, and the bucket 133, the time series of the attitude data, and the time series of the secondary line of sight data, and generates a weighted sum of the distances as an evaluation value. In this case, the closer the evaluation value is to zero, the higher the evaluation. The time series distance may be calculated using the DTW method.

[0025] The transmission unit 319 transmits the behavior record data generated by the generation unit 317 to the data server 10 .

[0026] The setting storage unit 320 stores setting data for the training simulator 30. The setting data stores the playback speed of the ghost machine 100G, whether or not to display the ghost machine 100G, whether or not to display the trajectory of the cutting edge of the ghost machine 100G's implement, whether or not to display a side view, whether or not to display the trainer's lever operation, and whether or not to display the trainer's line of sight. The setting data can be updated by the trainee's operation. The side view is an image that displays the virtual space from the side of the avatar machine 100A. The side view is drawn in the driver's cab.

[0027] The comment section 321 accepts input of comments for work data. Comments can be associated with playback timing. This allows the comment section 321 to be set so that the comment is displayed at a predetermined playback timing when the work data is played back.

[0028] The authentication unit 322 authenticates the operator of the training simulator 30 based on the authentication information stored in the data server 10 .

[0029] <<Processing of Training System 1>> When the training simulator 30 is started, the authentication unit 322 displays a login screen on the head-mounted display 35. The login screen displays, for example, an input form for a user ID and authentication information. When the operator inputs the user ID and authentication information values ​​into the login screen, the authentication unit 322 obtains the authentication information associated with the input user ID from the user table T3 of the data server 10 and verifies the authentication information.

[0030] When the authentication unit 322 succeeds in the matching, the display control unit 316 displays a menu screen on the head-mounted display 35. If the operator's user category is trainer, a trainer menu is displayed on the menu screen, allowing the operator to select either creating model data or evaluating work data. If the operator's user category is trainee, a trainee menu is displayed on the menu screen, allowing the operator to select either conducting training, viewing evaluation results, or changing settings.

[0031] 3 is a flowchart showing a method for creating model data according to the first embodiment. When a trainer logs in to the training simulator 30 and selects to create model data on the menu screen, the simulator 314 places the avatar machine 100A in an initial pose and a field in the virtual space (step S101).

[0032] The input unit 311 acquires operation data of the operation device 33, posture data measured by the IMU of the head-mounted display 35, and primary gaze data measured by the eye tracker of the head-mounted display 35 (step S102).

[0033] The simulator 314 simulates the behavior of the avatar machine 100A after a predetermined frame time based on the operation data input in step S102 (step S103). At this time, the simulator 314 calculates the positions and attitudes of the traveling body 110, the revolving body 120, the boom 131, the arm 132, and the bucket 133.

[0034] The simulator 314 determines the position and orientation of the rendering camera in the virtual space based on the position and orientation of the revolving unit 120 and the orientation data input in step S102 (step S104). The rendering unit 315 renders the virtual space in which the avatar machine 100A is placed from the rendering camera determined in step S104 (step S105). The display control unit 316 outputs the image data rendered in step S105 to the head-mounted display 35 (step S106).

[0035] The generation unit 317 generates secondary gaze data indicating the direction of the trainer's gaze in the virtual space based on the position and orientation of the rendering camera calculated in step S104 and the primary gaze data acquired in step S102 (step S107). The generation unit 317 generates one frame of behavior record data based on the operation data acquired in step S102, the orientation of the avatar machine 100A calculated in step S104, and the secondary gaze data generated in step S107 (step S108).

[0036] The generation unit 317 determines whether the trainer's operation has ended (step S109). If the trainer determines that the operation has ended, the trainer instructs the trainer to end the operation by performing a predetermined operation on the training simulator 30, such as moving his / her gaze to an end button on the screen. If the trainer's operation has not ended (step S109: NO), the process returns to step S102, and the next frame is simulated.

[0037] On the other hand, if the trainer's operation has ended (step S109: YES), the generation unit 317 compiles the generated behavior record data of multiple frames into model data (step S110). The generation unit 317 also accepts input of summary data for the model data from the trainer (step S111). The transmission unit 319 transmits the model data and summary data to the data server 10 (step S112). The data server 10 then assigns a model ID to the received model data and records the model ID, model data, and summary data in association with each other in the model data table T1.

[0038] <<Changing Settings>> When the trainee logs in to the training simulator 30 and selects "Change Settings" on the menu screen, the display control unit 316 reads out the setting data stored in the setting storage unit 320 and outputs an input screen for the setting data to the head-mounted display 35. FIG. 4 is a diagram showing an example of the input screen for setting data according to the first embodiment. The input screen includes input forms for the playback speed of the ghost machine 100G, whether to display the ghost machine 100G, whether to display the trajectory of the cutting edge of the ghost machine 100G, whether to display a side view, whether to display the trainer's lever operation, and whether to display the trainer's line of sight, as well as a confirm button. The setting data can be updated by the trainee's operation. When the display of the ghost machine 100G is ON in the input form, an input form for whether to display only the bucket 133 of the ghost machine 100G is displayed on the input screen. When the trainee operates the confirm button, the setting storage unit 320 updates the setting data according to the values ​​entered on the input screen.

[0039] <<Implementation of Training>> FIG. 5 is a flowchart (Part 1) showing a method for implementing training according to the first embodiment. FIG. 6 is a flowchart (Part 2) showing a method for implementing training according to the first embodiment. FIG. 7 is a flowchart (Part 3) showing a method for implementing training according to the first embodiment. When a trainee logs in to the training simulator 30 and selects implementation of training on the menu screen, the acquisition unit 312 accesses the data server 10 and acquires a list of model data recorded in the model data table T1 (step S201). The display control unit 316 outputs a selection screen to the head-mounted display 35 for selecting one model data item from the acquired list (step S202). At this time, the selection screen displays a list of model data items and summary data indicating the contents of the model data items. The trainee reads the summary data and selects one model data item.

[0040] The acquiring unit 312 accesses the data server 10 and acquires the model data selected by the trainee from the model data table T1 (step S203). The simulator 314 places the avatar machine 100A, the ghost machine 100G, and a field in the virtual space (step S204). The avatar machine 100A and the ghost machine 100G are placed overlappingly in the same position.

[0041] The input unit 311 acquires operation data of the operation device 33, posture data measured by the IMU of the head-mounted display 35, and primary gaze data measured by the eye tracker of the head-mounted display 35 (step S205).

[0042] The simulator 314 simulates the behavior of the avatar machine 100A after a predetermined frame time based on the operation data input in step S205 (step S206). At this time, the simulator 314 calculates the positions and orientations of the traveling body 110, the revolving body 120, the boom 131, the arm 132, and the bucket 133. The simulator 314 calculates the position and orientation of the rendering camera in the virtual space based on the position and orientation of the revolving body 120 and the orientation data input in step S205 (step S207).

[0043] The reproduction unit 313 reproduces the behavior of the ghost machine 100G by calculating the posture of the ghost machine 100G based on the time series of the joint angles of the work machine 130 and the rotation angle of the rotating body 120 contained in the model data acquired in step S203 and the playback speed of the ghost machine 100G indicated by the setting data (step S208).

[0044] The reproduction unit 313 determines whether the setting data specifies that the ghost machine 100G should be displayed (step S209). If the setting data specifies that the ghost machine 100G should be displayed (step S209: YES), the reproduction unit 313 determines whether the setting data specifies that only the bucket 133 of the ghost machine 100G should be displayed (step S210). If the setting data specifies that only the bucket 133 should be displayed (step S210: NO), the reproduction unit 313 determines that all components of the ghost machine 100G should be rendered (step S211). In this case, the image data is drawn as shown in FIG. 8. FIG. 8 shows an example of image data when rendering the entire ghost machine 100G according to the first embodiment. If it is set that only the bucket 133 should be displayed (step S210: YES), the reproduction unit 313 determines the bucket 133 of the ghost machine 100G as the rendering target, and excludes components of the ghost machine 100G other than the bucket 133 from the rendering target (step S212). In this case, the image data is drawn as shown in FIG. 9. FIG. 9 is an example of image data when rendering only the bucket 133 of the ghost machine 100G according to the first embodiment. If it is not set that the ghost machine 100G should be displayed (step S209: NO), the reproduction unit 313 excludes the ghost machine 100G from the rendering target (step S213).

[0045] Next, the reproduction unit 313 determines whether the setting data indicates that the trajectory of the blade tip of the ghost machine 100G should be displayed (step S214). If the setting indicates that the trajectory of the blade tip of the ghost machine 100G should be displayed (step S214: YES), the reproduction unit 313 generates blade tip objects Obj1 representing the trajectory of the blade tip and places them at both ends and the center of the blade tip of the bucket 133 of the ghost machine 100G calculated in step S208 (step S215). The blade tip objects Obj1 may be, for example, small spheres. The blade tip objects Obj1 may correspond to, for example, one or more points on the blade tip, and each blade tip object Obj1 represents the trajectory of the corresponding point. The reproduction unit 313 increases the transparency of the blade tip objects Obj1 generated previously by a predetermined value (step S216). As a result, the multiple cutting edge objects Obj1 represent the trajectory of the cutting edge position of the bucket 133 at the most recent predetermined time, with the more recent positions being displayed with higher transparency. In other embodiments, the reproduction unit 313 may change the size of the cutting edge objects Obj1 instead of the transparency. Cutting edge objects Obj1 whose transparency exceeds a predetermined threshold may be removed from the virtual space. For example, cutting edge objects Obj1 whose transparency exceeds a predetermined threshold may not be displayed. In this case, the image data is rendered as shown in FIG. 10 . FIG. 10 is an example of image data when displaying the trajectory of the cutting edge of the bucket 133 of the ghost machine 100G according to the first embodiment. The multiple cutting edge objects Obj1 are rendered as curves representing the trajectory of the cutting edge. If it is not set that the trajectory of the cutting edge of the ghost machine 100G should be displayed (step S214: NO), the reproduction unit 313 does not place the cutting edge objects Obj1 in the virtual space. The blade tip object Obj1 may have different colors at the left end, center, and right end of the blade tip. For example, the blade tip object Obj1 may be drawn at any one of the left end, center, and right end of the blade tip. The blade tip object Obj1 may also be represented as a band-like locus made up of a line segment that runs from the left end to the right end of the blade tip.

[0046] Next, the reproduction unit 313 determines whether or not the setting data specifies that the trainer's lever operation should be displayed (step S217). If the setting specifies that the trainer's lever operation should be displayed (step S217: YES), the reproduction unit 313 places the lever object Obj2 at a position a predetermined distance away from the rendering camera position determined in step S207 in the direction in which the rendering camera faces (step S218).

[0047] 11 shows an example of image data when displaying the trainer's lever operation according to the first embodiment. The lever object Obj2 includes two reference circles Obj21 representing the left and right operating levers, two operation points Obj22 representing the input positions of the left and right operating levers by the trainee, and multiple plots Obj23 indicating the input positions of the levers by the trainer. The two reference circles Obj21 are arranged with a predetermined gap between them in the left-right direction. The lever object Obj2 is arranged so that the reference circles face the rendering camera.

[0048] Based on the operation data included in the model data, the reproduction unit 313 places the plot Obj23 at a position spaced from the center of the reference circle Obj21 by the operation amount at the time of reproduction in step S208, and moves the operation point Obj22 to a position spaced from the center of the reference circle Obj21 by the operation amount indicated by the operation data acquired in step S205 (step S219). In other words, the lever object Obj2 represents the change in the position of the operating lever when viewed from above the operating plane of the operating lever. The reproduction unit 313 reduces the size of the previously placed plot Obj23 by a predetermined value (step S220). As a result, the multiple plots Obj23 represent the trajectory of the lever operation over the most recent predetermined time period. Note that in other embodiments, the reproduction unit 313 may change the transparency of the plot Obj23 instead of the size. Plots Obj23 whose size falls below a predetermined threshold may be removed from the virtual space. If it is not set that the lever operation should be displayed (step S217: NO), the reproducing unit 313 does not place the lever object Obj2.

[0049] Next, the reproduction unit 313 determines whether the setting data specifies that the trainer's line of sight should be displayed (step S221). If the setting specifies that the line of sight should be displayed (step S221: YES), the reproduction unit 313 places a line of sight object Obj3 in the virtual space (step S222). The line of sight object Obj3 is placed at a predetermined distance from the rendering camera position determined in step S207 in the direction indicated by the secondary line of sight data. The line of sight object Obj3 may be, for example, a small sphere. The reproduction unit 313 increases the transparency of previously placed line of sight objects Obj3 by a predetermined value (step S223). As a result, multiple line of sight objects Obj3 represent the trainer's line of sight trajectory at the most recent predetermined time. Note that in other embodiments, the reproduction unit 313 may change the size or hue of the plot Obj23 instead of its transparency. Line of sight objects Obj3 with a transparency exceeding a predetermined threshold may be removed from the virtual space. For example, if the transparency of the gaze object Obj3 exceeds a predetermined threshold, the reproduction unit 313 may not display the gaze object Obj3. Fig. 12 shows an example of image data when displaying the trainer's gaze according to the first embodiment. If it is not set that the gaze should be displayed (step S221: NO), the reproduction unit 313 does not place the gaze object Obj3 in the virtual space.

[0050] The rendering unit 315 determines whether the setting data indicates that a side view should be displayed (step S224). If the setting indicates that a side view should be displayed (step S224: YES), the rendering unit 315 positions the rendering camera to the side of the avatar machine 100A so that it faces the direction of the avatar machine 100A (step S225). The rendering unit 315 generates a side-view image Obj4 by drawing a virtual space in which the avatar machine 100A and the ghost machine 100G are positioned (step S226). The simulator 314 positions the side-view image Obj4 at a predetermined position inside the driver's cab 121 (step S227). FIG. 13 shows an example of image data for displaying a side view according to the first embodiment.

[0051] Next, the rendering unit 315 renders a virtual space in which the avatar machine 100A and the ghost machine 100G are placed from the rendering camera with the position and orientation determined in step S207 (step S228). The display control unit 316 outputs the image data rendered in step S228 to the head-mounted display 35 (step S229). Because the rendering unit 315 generates image data for each frame time, the image data is treated as frame images of a moving image. Furthermore, by rendering the virtual space in which the avatar machine 100A and the ghost machine 100G are placed, the rendering unit 315 can be said to have generated moving image data that simultaneously displays a first moving image in which the avatar machine 100A appears and a second moving image in which the ghost machine 100G appears. Furthermore, the rendering unit 315 can be said to have generated moving image data that displays the first moving image in which the avatar machine 100A appears and the second moving image in which the ghost machine 100G appears superimposed in the same virtual space.

[0052] The generation unit 317 generates secondary gaze data indicating the direction of the trainee's gaze in the virtual space based on the position and orientation of the rendering camera calculated in step S207 and the primary gaze data acquired in step S205 (step S230). The generation unit 317 generates one frame of behavior record data based on the operation data acquired in step S205, the orientation of the avatar machine 100A calculated in step S206, and the secondary gaze data generated in step S230 (step S231).

[0053] The generation unit 317 determines whether the training has ended (step S232). For example, the generation unit 317 may determine that the training has ended when a certain amount of time has elapsed since the playback position of the model data reached the end of the time series. If the training has not ended (step S232: NO), the process returns to step S205, and the next frame is simulated.

[0054] On the other hand, if the training is completed (step S232: YES), the generation unit 317 compiles the generated behavior record data of multiple frames as work data (step S233). The evaluation unit 318 calculates an evaluation value for evaluating the trainee's operation based on the difference between the work data generated in step S233 and the model data acquired in step S203 (step S234). The transmission unit 319 transmits the work data generated in step S233, the model ID of the model data acquired in step S203, and the evaluation value calculated in step S234 to the data server 10 (step S235). As a result, the data server 10 assigns a work ID to the received work data and records the work ID, work data, model ID, and evaluation value in association with each other in the work data table T2. The display control unit 316 also outputs an evaluation screen displaying the evaluation value calculated in step S234 to the head-mounted display 35 (step S236). The evaluation screen may display evaluation values ​​for each of the turning operation, the operation of the work implement 130, and the line of sight.

[0055] The evaluation unit 318 generates a trajectory model, which is a three-dimensional model in which the movement trajectory of the cutting edge from the start to the end of training by the trainee's operation, as indicated by the work data, and the movement trajectory of the cutting edge from the start to the end of training by the trainer's operation, as indicated by the model data, are arranged in three-dimensional space. FIG. 14 is a diagram showing an example of a trajectory model according to the first embodiment. The display control unit 316 may rotatably display the trajectory model generated by the evaluation unit 318 on the head-mounted display 35. This allows the display control unit 316 to simultaneously display the movement trajectory of the cutting edge by the trainee's operation and the movement trajectory of the cutting edge by the trainer's operation. The rotation of the trajectory model may be performed by changing the posture of the head-mounted display 35 or by operating the operation device 33.

[0056] <Evaluation of Work Data> Figure 15 is a flowchart showing a method for evaluating work data by a trainer according to the first embodiment. When the trainer logs in to the training simulator 30 and selects "evaluate work data" on the menu screen, the acquisition unit 312 accesses the data server 10 and acquires a list of work data recorded in the work data table T2 (step S301). The display control unit 316 outputs a selection screen to the head-mounted display 35, allowing the trainer to select one piece of work data from the acquired list (step S302). The trainer selects one piece of work data.

[0057] The acquisition unit 312 accesses the data server 10 and acquires the work data selected by the trainer from the work data table T2 (step S303). The reproduction unit 313 places the ghost machine 100G and the field in the virtual space (step S304).

[0058] The reproduction unit 313 reproduces the behavior of the ghost machine 100G by calculating the attitude of the ghost machine 100G based on the time series of the joint angles of the work machine 130 and the rotation angle of the revolving unit 120 included in the work data acquired in step S303 (step S305). The reproduction unit 313 calculates the position and attitude of the rendering camera in the virtual space based on the position and attitude of the revolving unit 120 of the ghost machine 100G and the attitude data input in step S305 (step S306).

[0059] The reproduction unit 313 places a lever object Obj2 at a position a predetermined distance away from the rendering camera position determined in step S306 in the direction the rendering camera is facing (step S307). This allows the trainer to check the trainee's lever operation. The reproduction unit 313 places a line-of-sight object Obj3 in the virtual space (step S308). The line-of-sight object Obj3 is placed at a position a predetermined distance away from the rendering camera position determined in step S306 in the direction indicated by the secondary line-of-sight data.

[0060] The rendering unit 315 renders the virtual space in which the ghost machine 100G is placed from the rendering camera whose position and orientation are determined in step S306 (step S309). The display control unit 316 outputs the image data rendered in step S309 to the head-mounted display 35 (step S310).

[0061] The comment unit 321 determines whether or not a comment has been input from the trainer at the current playback timing (step S311). If a comment has been input, comment data is generated that associates the input comment with the playback timing of the comment (step S312).

[0062] The reproducing unit 313 determines whether the reproduction of the work data has finished (step S313). If the reproduction has not finished (step S313: NO), the process returns to step S305, and the simulation of the next frame is performed.

[0063] On the other hand, if the playback has ended (step S313: YES), the transmitting unit 319 transmits the work ID of the work data acquired in step S303 and the comment data generated in step S312 to the data server 10 (step S314). As a result, the data server 10 adds the received comment data to the data string associated with the received work ID in the work data table T2.

[0064] Viewing Evaluation Results FIG. 16 is a flowchart showing a method for a trainee to view evaluation results according to the first embodiment. When the trainee logs in to the training simulator 30 and selects "View Evaluation Results" on the menu screen, the acquisition unit 312 accesses the data server 10 and acquires a list of work data recorded in the work data table T2 (step S401). The list acquired at this time may be a list of work data created by the logged-in trainee. The display control unit 316 outputs a selection screen to the head-mounted display 35 for selecting one piece of work data from the acquired list (step S402). The trainee selects one piece of work data. The acquisition unit 312 accesses the data server 10 and acquires the work data selected by the trainee from the work data table T2 (step S403). The reproduction unit 313 places a ghost machine 100G and a field in the virtual space (step S404).

[0065] The reproduction unit 313 reproduces the behavior of the ghost machine 100G by calculating the attitude of the ghost machine 100G based on the time series of the joint angles of the work machine 130 and the rotation angle of the revolving unit 120 included in the work data acquired in step S403 (step S405). The reproduction unit 313 determines the position and attitude of the rendering camera in the virtual space based on the position and attitude of the revolving unit 120 of the ghost machine 100G and the attitude data input in step S405 (step S406).

[0066] The reproduction unit 313 places the lever object Obj2 at a position a predetermined distance in the direction in which the rendering camera faces from the position of the rendering camera determined in step S406 (step S407). The reproduction unit 313 places the line-of-sight object Obj3 in the virtual space (step S408).

[0067] The rendering unit 315 renders a virtual space in which the ghost machine 100G is placed from the rendering camera with the position and orientation determined in step S406 (step S409). The comment unit 321 determines whether or not a comment associated with the current playback timing exists in the comment data included in the work data (step S410). If a comment associated with the current playback timing exists (step S410: YES), the comment is superimposed on the rendered image data (step S411). The display control unit 316 outputs the image data rendered in step S409 or step S411 to the head-mounted display 35 (step S412).

[0068] The reproducing unit 313 determines whether the playback of the work data has finished (step S413). If the playback has not finished (step S413: NO), the process returns to step S405, and the next frame is simulated. On the other hand, if the playback has finished (step S413: YES), the training simulator 30 ends the process. Note that the training system 1 can also play back work data that has not been evaluated by the trainer, using the procedure shown in FIG. 16. In this case, since there are no comments, the comment superimposition process in step S411 is skipped entirely.

[0069] <<Actions and Effects>> As described above, the training system 1 according to the first embodiment executes the following processes. The training system 1 accepts input of first operation data by a trainee, who is a first operator. The training system 1 simulates changes in posture of an avatar machine 100A, which is a first work machine, in accordance with the first operation data. The training system 1 reproduces the behavior of a ghost machine 100G, which is a second work machine, based on model data generated based on second operation data previously input by a trainer, who is a second operator. The training system 1 simultaneously displays a first moving image representing changes in posture of the avatar machine 100A and a second moving image representing the behavior of the ghost machine 100G. This allows the training system 1 to allow the trainee to train while comparing their own operation with the operation performed by the trainer. In other words, the first operator and the second operator may be the same operator. By comparing past operation data with current operation data, the same operator can check their level of improvement in operation or look back on past operations.

[0070] The model data for the training system 1 according to the first embodiment includes a time series of the posture of the work machine 100 simulated in accordance with the second operation data. This eliminates the need for the training system 1 to perform a simulation using the simulator 314 in order to reproduce the behavior of the ghost machine 100G. In other words, the training system 1 can reproduce the behavior of the ghost machine 100G simply by changing the posture of the ghost machine 100G in accordance with the posture indicated by the time series. Note that in other embodiments, the model data may not include a time series of the posture of the work machine 100, but may be represented by a time series of operation data. In this case, the training system 1 can reproduce the behavior of the ghost machine 100G by performing a simulation in accordance with the time series of the operation data.

[0071] The model data of the training system 1 according to the first embodiment includes a time series of the trainer's line of sight when the second operation data is input. This allows the training system 1 to present the trainee with not only how to operate the work machine 100, but also points to which they should pay attention. For example, points to which they should pay attention are points of gaze that should be paid attention to when operating the work machine 100.

[0072] The training system 1 according to the first embodiment records work data and comment data from the trainer in association with each other. The training system 1 simultaneously displays a moving image that reproduces changes in the posture of the work machine 100 based on the work data, and the comment data associated with the work data. This allows the trainee to receive feedback from others on his or her own operations. In particular, the training system 1 according to the first embodiment displays the comment data when the playback timing of the moving image coincides with the playback timing related to the comment data. This allows the trainee to recognize which operations, among a series of operations of the work machine 100, require attention.

[0073] The training system 1 according to the first embodiment evaluates the trainee's operation based on the difference between the first operation data related to the work data and the second operation data related to the model data, thereby enabling the trainee to objectively recognize the magnitude of the deviation from the model data.

[0074] The training system 1 according to the first embodiment accepts input from the trainee of the playback speed of the second moving image related to the model data, and simultaneously displays a first moving image showing the avatar machine 100A and a second moving image showing the ghost machine 100G, which are played at the specified playback speed. This allows the trainee to train according to his or her own skill level. That is, when the trainee is unfamiliar with the operation, the trainee can operate the ghost machine 100G at a slow playback speed, allowing the trainee to carefully check the trainer's operation while training. On the other hand, when the trainee becomes accustomed to the operation, the trainee can operate the ghost machine 100G at a playback speed close to 1x, allowing the trainee to train in a manner closer to the trainer's actual operation. Furthermore, the trainee can train under more difficult conditions than with the actual machine by operating the ghost machine 100G at a playback speed faster than 1x. In this case, the trainee can operate the actual machine more carefully by feeling that the time spent operating the machine is longer.

[0075] The training system 1 according to the first embodiment executes the following processing. The training system 1 accepts input of first operation data by a trainee who is a first operator. The training system 1 simulates a change in posture of an avatar machine 100A which is a first work machine in accordance with the first operation data. The training system 1 reproduces the operation of the trainer in accordance with second operation data previously input by a trainer who is a second operator. The training system 1 simultaneously displays a first moving image which represents a change in posture of the avatar machine 100A and a lever object Obj2 which represents the trainer's operation. This allows the trainee to simulate the work machine 100 while visually checking the trainer's operation. Furthermore, according to the first embodiment, an operation point Obj22 which represents the trainee's lever operation and a plot Obj23 which represents the trainer's lever operation are simultaneously displayed as the lever object Obj2. This allows the trainee to simulate the work machine 100 while visually checking the trainer's operation.

[0076] The lever object Obj2 according to the first embodiment includes a reference circle Obj21, which is a reference position image indicating the reference position of the operating lever, and a plot Obj23, which indicates the position of the operating lever after operation. This allows the trainee to easily recognize the input position of the operating lever based on the positional relationship between the reference circle Obj21 and the plot Obj23.

[0077] The lever object Obj2 according to the first embodiment includes a plurality of plots Obj23 that indicate past positions of the control lever. The older the plots Obj23, the smaller or more transparent they are. This allows the trainee to recognize the trajectory of the control lever and prevents confusion between the current input position of the control lever and the past input position.

[0078] The lever object Obj2 according to the first embodiment is located in front of the trainee in the line of sight direction on the head-mounted display 35. This allows the trainee to visually recognize the operation by the trainer regardless of the direction in which he or she is facing.

[0079] The training system 1 according to the first embodiment executes the following process. The training system 1 receives input of first operation data from a trainee, who is a first operator. The training system 1 simulates a change in posture of an avatar machine 100A, which is a first work machine, in accordance with the first operation data. The training system 1 reproduces a change in posture of a ghost machine 100G, which is a second work machine, based on model data generated based on second operation data previously input by a second operator. The training system 1 simultaneously displays a first moving image representing a change in posture of the avatar machine 100A and a second moving image representing a change in posture of the ghost machine 100G. At this time, the initial positions of the avatar machine 100A and the ghost machine 100G are aligned. In other words, the training system 1 displays the avatar machine 100A and the ghost machine 100G in a superimposed manner. This allows the trainee to easily recognize the difference between his or her own operation and the trainer's operation. For example, it is easier to recognize the difference in behavior compared to when the avatar machine 100A and the ghost machine 100G are displayed side by side.

[0080] The moving images of avatar machine 100A in training system 1 according to the first embodiment are images in which avatar machine 100A is rendered from a viewpoint located inside driver's cab 121 of avatar machine 100A. The moving images of ghost machine 100G in training system 1 according to the first embodiment are images in which ghost machine 100G is rendered from a viewpoint located inside driver's cab 121 of avatar machine 100A. In this way, training system 1 allows the trainee to easily recognize the difference between his or her own operation and the operation of the trainer from the viewpoint from driver's cab 121 of avatar machine 100A.

[0081] The training system 1 according to the first embodiment can render only the bucket 133, which is the implement of the ghost machine 100G. This prevents the revolving unit 120, boom 131, and the like of the ghost machine 100G from interfering with the avatar machine 100A and reducing visibility. On the other hand, by rendering the entire ghost machine 100G, the training system 1 can make the trainee more clearly recognize the difference in posture between the avatar machine 100A and the ghost machine 100G.

[0082] The ghost machine 100G according to the first embodiment has a higher transparency than the avatar machine 100A. This allows the training system 1 to prevent the ghost machine 100G from interfering with the simulation. The ghost machine 100G according to other embodiments may have a higher transparency than the avatar machine 100A and may have a different hue from the avatar machine 100A. This makes it easier to distinguish the ghost machine 100G from the avatar machine 100A. Furthermore, the ghost machine 100G according to other embodiments may be displayed using only an outline.

[0083] The training system 1 according to the first embodiment executes the following processing. The training system 1 accepts input of first operation data by a trainee, who is a first operator. The training system 1 simulates changes in posture of an avatar machine 100A, which is a first work machine, in accordance with the first operation data. The training system 1 reproduces changes in posture of a ghost machine 100G, which is a second work machine, based on model data generated based on second operation data previously input by a second operator. The training system 1 simultaneously displays the avatar machine 100A and a cutting edge object Obj1 that represents the trajectory of a bucket 133, which is a work tool of the ghost machine 100G. This allows the trainee to easily recognize differences between his or her own operation and the operation of the trainer.

[0084] The training system 1 according to the first embodiment places cutting edge objects Obj1 corresponding to each of a plurality of points on the cutting edge. Each cutting edge object Obj1 represents the trajectory of the corresponding point. By providing cutting edge objects Obj1 for a plurality of points on the cutting edge, the trainee can easily recognize the position of the cutting edge of the bucket 133 in the depth direction.

[0085] The cutting edge object Obj1 according to the first embodiment has a different color for each of the multiple points, which reduces the possibility of confusing and recognizing the multiple cutting edge objects Obj1.

[0086] The training system 1 according to the first embodiment executes the following processing. The training system 1 accepts input of operation data by the operator. The training system 1 detects the operator's line of sight when inputting the operation data. The training system 1 simulates a change in the attitude of the work machine 100 in accordance with the operation data. The training system 1 simultaneously displays an attitude image representing the simulated change in attitude of the work machine 100 and a line of sight object Obj3 representing the change in line of sight. This allows the training system 1 to display the relationship between the operator's operation of the work machine 100 and the line of sight direction. This allows the trainee to recognize what they should focus on during operation from the trainer's line of sight. This also allows the trainer to evaluate the trainee's driving based on the trainee's line of sight.

[0087] In the first embodiment, the cutting edge object Obj1, the plot Obj23 of the lever object Obj2, and the line of sight object Obj3 are a plurality of points representing a trajectory, whereas in other embodiments, some or all of these objects may be curves representing a trajectory.

[0088] Second Embodiment In the training system 1 according to the first embodiment, the behavior of the ghost machine 100G is continuously reproduced at a reproduction speed specified in the setting data. On the other hand, if the trainee is a beginner, there is a possibility that the trainee's operation cannot keep up with the behavior of the ghost machine 100G. In such a case, if the ghost machine 100G is continuously reproduced and the postures of the avatar machine 100A and the ghost machine 100G differ significantly, the trainee may lose track of the indicators for operation. The training system 1 according to the second embodiment enables appropriate training to be carried out even when the trainee's operation is slow.

[0089] The training system 1 according to the first embodiment reproduces the behavior of the ghost machine 100G for each frame time in step S208, as shown in Fig. 5. In contrast to this, in the second embodiment, the following process is executed instead of step S208.

[0090] The reproduction unit 313 calculates the distance between the posture of the avatar machine 100A calculated in step S206 and the previous posture of the ghost machine 100G. The distance between the postures can be found, for example, by taking the square root of the sum of the squares of the difference in the swing angle, the difference in the boom angle, the difference in the arm angle, and the difference in the bucket angle. If the calculated distance exceeds a threshold value set in advance in the setting data, the reproduction unit 313 maintains the posture of the ghost machine 100G. On the other hand, if the calculated distance does not exceed a threshold value set in advance in the setting data, the reproduction unit 313 calculates the posture of the ghost machine 100G based on the time series of the joint angles of the work implement 130 and the swing angle of the revolving unit 120 included in the model data, and the playback speed of the ghost machine 100G indicated in the setting data, thereby reproducing the behavior of the ghost machine 100G.

[0091] Thus, according to the second embodiment, the training system 1 suspends the reproduction of the behavior of the ghost machine 100G when the difference between the posture of the simulated avatar machine 100A and the posture of the ghost machine 100G exceeds a predetermined threshold. For example, when suspending the reproduction of the behavior of the ghost machine 100G, the training system 1 may store an image of the moment of suspension. This allows the training system 1 according to the second embodiment to provide appropriate training even when the trainee's operations are slow.

[0092] Other Embodiments Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design modifications are possible. That is, in other embodiments, the order of the above-described processes may be changed as appropriate. Furthermore, some processes may be executed in parallel.

[0093] The training system 1 according to the embodiment described above includes a data server 10 and a training simulator 30, but is not limited to this. For example, the training system 1 according to another embodiment may be configured with the training simulator 30 alone. In this case, training can be carried out based on model data recorded in advance in the training simulator 30. Furthermore, the training system 1 according to another embodiment may carry out training while referring to work data that the trainee has previously operated. Furthermore, in another embodiment, part of the training simulator 30 may be provided in an external computer. For example, in another embodiment, the training simulator 30 may only be responsible for inputting operation data and displaying calculation results, and the simulation calculations may be performed by an external device.

[0094] Although the training system 1 according to the embodiment described above uses a head-mounted display 35 as a display device, this is not limiting. For example, in another embodiment, a large display used for remotely operating the work machine 100 may be used as a display device. In this case, the input unit 311 may accept specification of the position and orientation of the rendering camera from the operator. Examples of specified positions and orientations of the rendering camera include a subjective viewpoint (the driver's viewpoint), an objective viewpoint (diagonally behind the driver), a bird's-eye viewpoint (to the side of the work machine 100), and a free viewpoint. The rendering unit 315 renders the virtual space according to the specified position and orientation of the rendering camera. Furthermore, the display control unit 316 may accept instructions to replay (start over) or pause when playing back work data. Furthermore, the display control unit 316 may display a seek bar indicating the playback position and accept an instruction to cue to any point by operating the seek bar.

[0095] <Computer Configuration> Fig. 17 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. The computer 90 includes a processor 91, a main memory 92, a storage 93, and an interface 94. The training simulator 30 described above is implemented on the computer 90. The operations of the above-described processing units are stored in the storage 93 in the form of a program. The processor 91 reads the program from the storage 93, loads it into the main memory 92, and executes the above-described processing in accordance with the program. The processor 91 also allocates storage areas in the main memory 92 corresponding to the above-described storage units in accordance with the program. Examples of the processor 91 include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and a microprocessor.

[0096] The program may be for implementing some of the functions to be performed by the computer 90. For example, the program may be implemented in combination with other programs already stored in storage or in combination with other programs implemented in other devices. In another embodiment, the computer 90 may include a custom large-scale integrated circuit (LSI) such as a programmable logic device (PLD) in addition to or instead of the above configuration. Examples of PLDs include programmable array logic (PAL), generic array logic (GAL), complex programmable logic device (CPLD), and field programmable gate array (FPGA). In this case, some or all of the functions implemented by the processor 91 may be implemented by the integrated circuit. Such an integrated circuit is also an example of a processor. In another embodiment, the computer 90 may be virtualized on one or more computers.

[0097] Examples of storage 93 include a magnetic disk, a magneto-optical disk, an optical disk, and a semiconductor memory. Storage 93 may be an internal medium directly connected to the bus of computer 90, or an external medium connected to computer 90 via interface 94 or a communication line. Furthermore, when this program is distributed to computer 90 via a communication line, computer 90 that receives the program may load the program into main memory 92 and execute the above-described processing. In at least one embodiment, storage 93 is a non-transitory tangible storage medium.

[0098] The program may also be a program for realizing part of the above-described functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the above-described functions in combination with another program already stored in storage 93.

[0099] According to the above aspect, in training for operating a work machine, the operation by the operator can be easily compared with other operations.

[0100] DESCRIPTION OF SYMBOLS 1...Training system 10...Data server 100...Work machine 100A...Avatar machine 100G...Ghost machine 110...Traveling body 120...Swivel body 121...Driver's cab 130...Work machine 131...Boom 132...Arm 133...Bucket 30...Training simulator 31...Calculation device 311...Input unit 312...Acquisition unit 313...Reproduction unit 314...Simulator 315...Rendering unit 316...Display control unit 317...Generation unit 318...Evaluation unit 319...Transmission unit 320...Settings storage unit 321...Comment unit 322...Authentication unit 33...Operation device 35...Head-mounted display 90...Computer 91...Processor 92...Main memory 93...Storage 94...Interface Obj1...Cutting edge object Obj2...Lever object Obj21...Reference circle Obj22...Operation point Obj23...Plot Obj3...Gaze object Obj4...Side view image T1...Model data table T2...Work data table T3...User table V...Virtual space

Claims

1. A display system for a work machine, which receives input of first operation data by a first operator, simulates changes in the posture of the work machine in accordance with the first operation data, reproduces the operation by the second operator based on model data including second operation data previously input by a second operator, and simultaneously displays a first moving image showing the simulated changes in the posture of the work machine and a second moving image showing the reproduced operation by the second operator.

2. A display system as described in claim 1, wherein the behavior of the work machine is reproduced based on the model data, the second video image includes the reproduced operation by the second operator and the reproduced behavior of the work machine, and the first video image and the second video image are displayed in an overlapping manner.

3. The display system according to claim 2, wherein the second moving image includes a reference position image showing a reference position of the operating lever, and an input position image showing a position of the operating lever after operation.

4. The display system according to claim 3, wherein the first moving image includes an input position image showing a position of an operating lever corresponding to the first operation data by the first operator.

5. A display system according to claim 3, wherein the second moving image includes a plurality of past position images showing past positions of the operating lever, and the older the past position images are, the smaller or more transparent the images are.

6. A display system according to any one of claims 1 to 5, wherein the second moving image represents a change in position of the operating lever when the operating lever is viewed in a plan view from above the operating plane of the operating lever.

7. A display system according to any one of claims 1 to 5, wherein the first moving image and the second moving image are displayed on a head mounted display, and the second moving image is positioned forward in the line of sight of the head mounted display.

8. A display method in which a computer receives input of first operation data by a first operator, simulates a change in posture of a work machine in accordance with the first operation data, reproduces an operation by the second operator based on model data including second operation data previously input by a second operator, and simultaneously displays a first moving image showing the simulated change in posture of the work machine and a second moving image showing the reproduced operation by the second operator.

Images