Training system, and generation method of stage data

Abstract

To realize training in various situations when loading onto transport vehicles.SOLUTION: A stage generation unit receives a designation of the type of transport vehicle onto which a work machine is to load a load. The stage generation unit receives a designation of a placement position of the transport vehicle. The stage generation unit generates stage data for placing a specified type of transport vehicle at a placement position in a virtual space.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a training system and a method for generating stage data. [Background technology]

[0002] Patent Document 1 discloses a technology that allows an operator to simulate the operation of a work machine while referring to the operation of the work machine by the operator or another operator. The technology described in Patent Document 1 plays back video footage of past work performed by a remotely controlled work machine, and simulates the work in a work environment related to the playback position when the video footage is stopped. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-103193 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to familiarize operators with the operation of a work machine, it is preferable to have them perform work in a variety of situations. In particular, when training for loading cargo onto a transport vehicle, it is preferable to conduct training in a variety of situations in which the relationship between the transport vehicle and the work machine is different. An object of the present disclosure is to provide a training system and a method for generating stage data that can realize training in various situations when loading a transport vehicle. [Means for solving the problem]

[0005] According to one aspect of the present invention, a training system is a training system for operating a work machine placed in a virtual space, which accepts a specification of the type of transport vehicle for the work machine to load cargo, accepts a specification of a location for the transport vehicle, and generates stage data for placing the specified type of transport vehicle at the location in the virtual space. [Effects of the Invention]

[0006] According to the above aspect, the training system can provide training in various situations in loading a transport vehicle. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic diagram showing the configuration of a training system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing the software configuration of a calculation device included in the training simulator according to the first embodiment. [Figure 3] 4 is a flowchart showing a method for creating model data according to the first embodiment. [Figure 4] FIG. 3 is a diagram showing an example of a work machine layout screen according to the first embodiment. [Figure 5] FIG. 10 is a diagram showing an example of a screen for arranging a transport vehicle according to the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating a method for determining a default placement orientation of a transport vehicle according to the first embodiment. [Figure 7] 4 is a flowchart showing a method for creating model data according to the first embodiment. [Figure 8] 1 is a flowchart (part 1) showing a training method according to the first embodiment. [Figure 9] 10 is a flowchart (part 2) showing a training method according to the first embodiment. [Figure 10] 1 is an example of image data showing a virtual space in which an avatar machine and a ghost machine according to the first embodiment are placed. [Figure 11] FIG. 1 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] First Embodiment <<Configuration of Training System 1>> Hereinafter, the embodiments will be described in detail with reference to the drawings. FIG. 1 is a schematic diagram showing the configuration of a training system 1 according to the first embodiment. The training system 1 according to the first embodiment is a system in which a trainee, who is an operator unfamiliar with operating the work machine 100, simulates the operation of the work machine 100 while referring to the operations of a trainer, who is an operator skilled in operating the work machine 100. The trainee can have his or her own operation evaluated by the training system 1 and the trainer. The trainer and the trainee are both operators of the work machine 100. In other words, the training system 1 is a training system for operating the work machine 100 arranged in a virtual space.

[0009] The training system 1 includes a data server 10 and one or more training simulators 30.

[0010] The data server 10 stores field data, stage data, model data, and work data used for training. The field data represents the terrain on which training will be carried out. The stage data represents the initial positions of the work machine 100 and the transport vehicle 200 during training. The model data represents the operation of the work machine 100 by the trainer. The trainee performs training using the model data as a reference. The work data represents the operation of the work machine 100 by the trainee. The training system 1 and the trainer evaluate the trainee's operation based on the work data. The model data and work data include a time series of operation of the control lever, a time series of the attitude of the work machine 100 (joint angles and swing angles of the work machine), and a time series of line of sight. Hereinafter, data representing a time series of the operation and behavior of the work machine 100, such as the model data and work data, will also be referred to as behavior record data.

[0011] Specifically, the data server 10 includes a field data table T1 for storing field data, a stage data table T2 for storing stage data, a model data table T3 for storing model data, a work data table T4 for storing work data, and a user table T5 for storing authentication data. The field data table T1 stores field data in association with a field ID, which is the ID (identification information) of the field data. The field data is a three-dimensional object representing the terrain placed in virtual space. The field data includes an excavation target G11 to be excavated by the work machine 100. The three-dimensional object constituting the field data has a label indicating whether or not it is an excavation target G11. For example, the excavation target G11 is displayed in a different manner (for example, a different color) from parts that are not excavation targets. The stage data table T2 stores stage data indicating the situation of the work machine 100 and the haulage vehicle 200 during training, a field ID, and a stage ID, which is the ID (identification information) of the stage data, in association with each other. The stage data is data related to the placement of the work machine 100 and the haulage vehicle 200 relative to the terrain represented by the field data indicated by the field ID. The stage data includes the type, position, and orientation of the work machine 100, and the type, position, and orientation of the haulage vehicle 200. The model data table T3 stores model data, a stage ID indicating the stage in which the model data is reproduced, a model ID which is the ID (identification information) of the model data, and summary data indicating the work content reproduced by the model data, in association with each other. The work data table T4 stores work data, a work ID which is the ID (identification information) of the work data, a model ID of the corresponding model data, evaluation data indicating the evaluation by the training system 1, and comment data indicating the evaluation by the trainer, in association with each other. The user table T5 stores a user ID, a user category, and authentication information in association with each other. The value of the user category is either trainer or trainee. The authentication information may be, for example, a password.

[0012] The training simulator 30 receives operation input from the operator and simulates the behavior of the work machine 100 in response to that operation input. The training simulator 30 generates an image showing the behavior of the work machine 100 and presents it to the operator. The training simulator 30 also reproduces the behavior of the work machine 100 based on model data stored in the data server 10, generates an image showing the behavior of the work machine 100, and presents it to the operator.

[0013] The training simulator 30 comprises a computing device 31, an operation device 33, and a head-mounted display 35. The operation device 33 is an input interface for operating the work machine 100. The operation device 33 according to the first embodiment is two operation levers (a right operation lever and a left operation lever). The operation device 33 may differ depending on the type of work machine 100 being simulated. The head-mounted display 35 displays images calculated by the computing device 31. The head-mounted display 35 is equipped with an eye tracker that detects the line of sight of the wearer and an IMU (Inertial Measurement Unit) that detects the attitude of the head-mounted display 35. The computing device 31 simulates the work machine 100 in a virtual space V based on input from the operation device 33. The computing device 31 determines the direction of the line of sight in the virtual space V based on the attitude of the head-mounted display 35 and renders the simulation results. As a result, the virtual space V is displayed on the head-mounted display 35 in conjunction with the attitude of the head-mounted display 35.

[0014] <Configuration of work machine 100> The work machine 100 is an object to be operated in the training simulator 30. Here, a hydraulic excavator will be described as an example of the work machine 100. The work machine 100 includes a traveling body 110, a revolving body 120, and a work implement . The running body 110 supports the work machine 100 so that the work machine 100 can travel. The rotating body 120 is supported on the running body 110 so as to be rotatable about a rotation center. A cab 121 is provided at the front of the rotating body 120. A rendering camera for rendering the virtual space V by the computing device 31 is provided inside the cab 121. The work machine 130 is supported at the front of the revolving body 120 so as to be drivable in the vertical direction. The work machine 130 includes a boom 131, an arm 132, and a bucket 133 as a work implement. For example, the work implement has a cutting edge that extends across the width. Other examples of work implements include tip attachments such as a clam bucket, tilt bucket, and tilt rotate bucket.

[0015] The base end of the boom 131 is rotatably attached to the revolving unit 120 via a boom pin. In the work machine 100 shown in FIG. 1, the boom 131 is provided in the center of the front of the revolving unit 120, but this is not limitative and the boom 131 may be attached offset in the left-right direction. In this case, the center of rotation of the revolving unit 120 is not located on the operating plane of the work implement 130. The arm 132 connects the boom 131 and the bucket 133. The base end of the arm 132 is rotatably attached to the tip of the boom 131 via an arm pin. The bucket 133 is rotatably attached via a pin to the tip of the arm 132. The bucket 133 functions as a container for storing excavated earth and sand.

[0016] Configuration of transport vehicle 200 The transport vehicle 200 is the target of loading by the work machine 100 in the training simulator. The transport vehicle 200 includes a cab 210 and a dump body 220. The dump body 220 is a member on which cargo is loaded. The cab 210 is provided forward of the dump body 220 on the transport vehicle 200.

[0017] <<Configuration of Training Simulator 30>> FIG. 2 is a block diagram showing the software configuration of the arithmetic device 31 included in the training simulator 30 according to the first embodiment. The calculation device 31 provided in the training simulator 30 includes an input unit 311, an acquisition unit 312, a reproduction unit 313, a simulator 314, a rendering unit 315, a display control unit 316, a stage generation unit 317, a behavior recording unit 318, an evaluation unit 319, a transmission unit 320, and an authentication unit 321.

[0018] The input unit 311 acquires operation data from the operating device 33, posture data measured by the IMU of the head-mounted display 35, and gaze data measured by the eye tracker of the head-mounted display 35. The gaze data is expressed as a direction based on the display surface of the head-mounted display 35. Note that an absolute gaze direction can be identified by combining the posture data and gaze data. Hereinafter, the gaze data measured by the eye tracker will be referred to as primary gaze data, and the gaze data indicating the absolute gaze direction identified from the posture data and gaze data will be referred to as secondary gaze data.

[0019] The acquisition unit 312 acquires field data, stage data, and behavior record data (model data and work data) from the data server 10. The reproduction unit 313 reproduces the behavior of the work machine 100 based on the behavior record data acquired by the acquisition unit 312. Hereinafter, the work machine 100 reproduced in the virtual space V by the reproduction unit 313 will be referred to as the ghost machine 100G. The reproduction unit 313 reproduces the behavior of the work machine 100 by arranging the ghost machine 100G in accordance with the time series of the joint angles of the work implement 130 and the swing angle of the rotating body 120 included in the behavior record data. Note that the reproduction unit 313 according to other embodiments may, for example, simulate the behavior of the work machine 100 based on time series data of the movement of the operation lever included in the behavior record data, and arrange the ghost machine 100G.

[0020] Simulator 314 arranges terrain in virtual space V based on the field data acquired by acquisition unit 312, and arranges work machine 100 and transport vehicle 200 in virtual space V based on the stage data. Simulator 314 simulates the behavior of work machine 100 based on operation data input to input unit 311. Hereinafter, work machine 100 simulated by simulator 314 will be referred to as avatar machine 100A. Simulator 314 simulates the behavior of avatar machine 100A by calculating the angular velocities of revolving unit 120 and work machine 130 in accordance with the operation amounts indicated by the operation data.

[0021] The rendering unit 315 renders the ghost machine 100G, the avatar machine 100A, and the transporter vehicle 200, and generates image data. The rendering camera in the virtual space is located inside the driver's cab 121 of the avatar machine 100A. The rendering camera is directed in the direction of the front of the head-mounted display 35 indicated by the posture data input to the input unit 311, based on the front of the avatar machine 100A. Note that rendering by the rendering unit 315 is performed at a predetermined frame rate. Therefore, the image data generated by the rendering unit 315 is treated as frame images of a moving image.

[0022] The display control unit 316 outputs the image data rendered by the rendering unit 315 to the head-mounted display 35.

[0023] The stage generation unit 317 generates stage data in accordance with operational input from the trainer to the input unit 311. Specifically, the stage generation unit 317 receives input from the trainer of the field, the type, position, and orientation of the work machine 100, and the type, position, and orientation of the haulage vehicle 200 onto which the work machine 100 will load cargo, and generates stage data in accordance with this input. For example, the stage generation unit 317 receives designation of the placement positions of the work machine 100 and the haulage vehicle 200 in virtual space, places the work machine 100 and the haulage vehicle 200 at the designated placement positions, and generates stage data. Note that the stage generation unit 317 may also generate stage data in accordance with operational input from the trainee, for example.

[0024] The behavior recording unit 318 generates behavior record data based on the operation data, posture data, and primary gaze data input to the input unit 311, and the posture of the avatar machine 100A simulated by the simulator 314. Specifically, the behavior recording unit 318 obtains the positions and postures of the traveling unit 110, the revolving unit 120, the boom 131, the arm 132, and the bucket 133 of the avatar machine 100A in the virtual space based on the posture of the avatar machine 100A simulated by the simulator 314. Each position and posture is expressed in a virtual space coordinate system, which is a three-dimensional Cartesian coordinate system that defines the virtual space. The behavior recording unit 318 generates secondary gaze data indicating the direction of the gaze in the virtual space coordinate system based on the positions and postures of the traveling unit 110, the revolving unit 120, the boom 131, the arm 132, and the bucket 133 in the virtual space, as well as the posture data and the primary gaze data.

[0025] The evaluation unit 319 evaluates the trainee's operation based on the difference between the model data acquired by the acquisition unit 312 and the work data generated by the behavior recording unit 318. For example, the evaluation unit 319 calculates the position and posture of the revolving unit 120 in the virtual space indicated by the model data and the work data, as well as the distance between the posture data and the secondary gaze data. The evaluation unit 319 calculates the distance between the model data and the work data for each of the time series of the positions and postures of the traveling unit 110, revolving unit 120, boom 131, arm 132, and bucket 133, the time series of the posture data, and the time series of the secondary gaze data, and generates a weighted sum of the distances as an evaluation value. In this case, the closer the evaluation value is to zero, the higher the evaluation. The distance in the time series may be calculated using the DTW method.

[0026] The transmission unit 320 transmits the behavior record data generated by the behavior recording unit 318 to the data server 10.

[0027] The authentication unit 321 authenticates the operator of the training simulator 30 based on the authentication information stored in the data server 10.

[0028] <<Training System 1 Processing>> When the training simulator 30 is started, the authentication unit 321 displays a login screen on the head-mounted display 35. The login screen displays, for example, an input form for a user ID and authentication information. When the operator inputs the user ID and authentication information values ​​into the login screen, the authentication unit 321 obtains the authentication information associated with the input user ID from the user table T5 of the data server 10 and verifies the authentication information.

[0029] If the authentication unit 321 succeeds in the verification, the display control unit 316 causes the head-mounted display 35 to display a menu screen. If the operator's user category is a trainer, a trainer menu is displayed on the menu screen to allow the operator to select from among creation of model data, creation of stage data, and evaluation of work data. If the user category of the operator is a trainee, a trainee menu is displayed on the menu screen, allowing the operator to select one of the following: conducting training, viewing evaluation results, and changing settings.

[0030] <<Generating Stage Data>> FIG. 3 is a flowchart showing a method for creating model data according to the first embodiment. When a trainer logs in to the training simulator 30 and selects to create stage data on the menu screen, the stage generation unit 317 displays a field selection screen that accepts the selection of a field on the head-mounted display 35. Specifically, the stage generation unit 317 generates the field selection screen that includes a list of images overlooking the terrain represented by each field data from the field data table T1. The stage generation unit 317 accepts the selection of a field by the trainer via the input unit 311 (step S1).

[0031] The stage generation unit 317 displays a selection screen on the head-mounted display 35 for selecting the type of work machine 100 to be operated. The type of work machine 100 includes the category of the work machine 100 (for example, backhoe, front shovel, wheel loader, etc.) and the vehicle size (bucket capacity, overall length, etc.). The input unit 311 accepts the trainer's selection of the type of work machine 100 (step S2).

[0032] The stage generation unit 317 displays a layout screen for the work machine 100 on the head-mounted display 35. The stage generation unit 317 accepts the trainer's specification of the layout position for the work machine 100 (step S3). FIG. 4 is a diagram showing an example of the layout screen for the work machine 100 according to the first embodiment. The layout screen for the work machine 100 includes a planar map G1 showing, from above, the terrain represented by the field data selected in step S1, and a work machine icon G2 representing the work machine 100. In the planar map G1, the excavation target G11 is displayed in a different manner (for example, in a different color) from parts that are not the excavation target. The stage generation unit 317 displays the work machine icon G2 at the position of the line of sight measured by the eye tracker. When the trainer gazes at the position on the planar map G1 where he or she wants to place the work machine 100 for a certain period of time, the stage generation unit 317 determines that position as the layout position for the work machine 100. The stage generation unit 317 is not limited to an eye tracker, and may, for example, move a cursor by operating a lever of a work machine, and determine the placement position as the position indicated by the cursor when a switch or the like is pressed.

[0033] Next, the stage generation unit 317 accepts the trainer's designation of the placement orientation of the work machine 100 (step S4). The stage generation unit 317 fixes the work machine icon G2 to the determined placement position and rotates the work machine icon G2 according to the position of the line of sight. When the trainer gazes at the work machine 100 for a certain period of time so that it faces the desired orientation, the stage generation unit 317 determines that orientation as the placement orientation of the work machine 100. The stage generation unit 317 is not limited to an eye tracker, and may also accept a selection of orientation by operating a work machine lever and determine the orientation selected by pressing a switch, etc. as the placement orientation.

[0034] The stage generation unit 317 determines multiple candidates and a default candidate for the haulage vehicle 200 according to the type of work machine 100 selected in step S2 (step S5). For example, multiple candidates and a default candidate for the haulage vehicle 200 may be associated and recorded in advance for each type of work machine 100, and the stage generation unit 317 may identify multiple candidates and a default candidate that correspond to the selection result in step S2. Furthermore, for example, the stage generation unit 317 may determine a range of appropriate maximum load capacities from the bucket capacity of the work machine 100 selected in step S2, and identify multiple candidates and a default candidate for the haulage vehicle 200 based on the maximum load capacities within that range. The default candidate may be, for example, the haulage vehicle 200 that is closest to the median of the range of maximum load capacities.

[0035] The stage generation unit 317 displays on the head-mounted display 35 a selection screen that allows the trainee to select the type of the haulage vehicle 200 to be loaded from among the multiple candidates identified in step S5. At this time, the haulage vehicle 200 identified as the default candidate is in a selected state. In other words, the stage generation unit 317 may present one type corresponding to the type of work machine 100 as the default type from among the multiple types of haulage vehicle. The input unit 311 accepts the trainer's selection of the type of haulage vehicle 200 (step S6). In other words, the trainer can specify one or more types from among the multiple types of haulage vehicle 200. If the trainer determines that the default candidate is acceptable, he or she does not need to perform any special operation.

[0036] The stage generation unit 317 displays a layout screen for the haulage vehicle 200 on the head-mounted display 35. The input unit 311 accepts the trainer's designation of the placement position of the haulage vehicle 200 (step S7). When the trainer gazes at the position on the planar map G1 where the haulage vehicle 200 is to be placed for a certain period of time, the stage generation unit 317 determines the gazed position as the placement position of the haulage vehicle 200. The stage generation unit 317 may determine the placement position based on the operation of a switch on a work equipment lever or the like, without being limited to an eye tracker. FIG. 5 is a diagram showing an example of a layout screen for the haulage vehicle 200 according to the first embodiment. The layout screen for the haulage vehicle 200 includes a planar map G1 showing a top view of the terrain represented by the field data selected in step S1, a work machine icon G2 representing the work machine 100 placed in step S3, and a haulage vehicle icon G3 representing the haulage vehicle 200. A layout area G12 and a movable range G13 are displayed on the planar map G1.

[0037] The movable range G13 represents the movable range of the bucket 133 of the work machine 100. If the movable range G13 of the work machine 100 overlaps with the excavation target G11, the work machine 100 can excavate the excavation target G11 without moving. In this case, the trainer can generate a stage for loading swing training by placing the transport vehicle 200 inside the movable range G13. Note that the work machine 100 is not necessarily placed in a position where it can excavate the excavation target G11. For example, if the trainer is assuming training in which the work machine 100 will travel and then excavate, the initial position of the work machine 100 will be a position away from the excavation target G11. Therefore, the stage generation unit 317 does not prevent the transport vehicle 200 from being placed outside the movable range G13.

[0038] The allocable area G12 is an area of ​​the terrain where the transport vehicle 200 can be placed. The allocable area G12 is a range represented by a trajectory obtained by moving the movable range G13 of the work machine 100 selected in step S2 in parallel along the outer edge of the excavation target G11. In other words, the allocable area G12 is an area where loading is possible when the work machine 100 is located in a position where it can excavate the excavation target G11. If the placement position of the transport vehicle 200 specified by the trainer is outside the allocable area G12, the stage generation unit 317 does not determine that position as the placement position. Note that because the excavation target G11 is outside the allocable area G12, the stage generation unit 317 does not allow the transport vehicle 200 to be placed on the excavation target G11.

[0039] The stage generation unit 317 displays the transport vehicle icon G3 at the position of the line of sight measured by the eye tracker. At this time, the stage generation unit 317 faces the transport vehicle icon G3 in the turning direction during the loading turn of the work machine 100. In other words, the stage generation unit 317 sets the default placement orientation of the transport vehicle icon G3 to an orientation facing away from the excavation target G11. This is to position the transport vehicle 200 so that the bucket 133 does not pass the driver's cab 220 of the transport vehicle 200 during the loading turn.

[0040] 6 is a diagram showing a method for determining the default placement orientation of the haulage vehicle 200 according to the first embodiment. Specifically, the stage generation unit 317 first calculates a circle R that passes through the center C of the dump body 210 of the haulage vehicle icon G3 and the excavation target G11 and is congruent with the standard trajectory of the bucket 133 during the loading swing of the work machine 100. The stage generation unit 317 determines, as the default placement orientation of the haulage vehicle 200, an orientation that faces away from the excavation target G11 along a tangent to the calculated circle R that passes through the center C of the dump body 210.

[0041] Next, the stage generation unit 317 receives the trainer's specification of the placement orientation of the haulage vehicle 200 (step S8). The stage generation unit 317 fixes the haulage vehicle icon G3 at the determined placement position and rotates the haulage vehicle icon G3 according to the position of the line of sight. When the trainer gazes at the haulage vehicle 200 for a certain period of time so that it faces the desired orientation, the stage generation unit 317 determines that orientation as the orientation of the work machine 100. If the trainer determines that the default placement orientation is acceptable, he or she does not need to perform any special operation. For example, the stage generation unit 317 may determine the orientation of the work machine 100 based on the operation of the work machine lever, rather than using an eye tracker.

[0042] The stage generation unit 317 then generates stage data based on the field selected in step S1, the type of work machine 100 specified in step S2, the location of the work machine 100 specified in step S3, the location orientation of the work machine 100 specified in step S4, the type of haulage vehicle 200 specified in step S6, the location position of the haulage vehicle 200 specified in step S7, and the location orientation of the haulage vehicle 200 specified in step S8 (step S9). The transmission unit 320 transmits the stage data to the data server 10 (step S10). As a result, the data server 10 assigns a stage ID to the received stage data, and records the stage ID, stage data, and field data in association with each other in the stage data table T2.

[0043] <<Generation of model data>> FIG. 7 is a flowchart showing a method for creating model data according to the first embodiment. When the trainer logs in to the training simulator 30 and selects to create model data on the menu screen, the behavior recording unit 318 receives a selection of stage data from the trainer (step S100). When the trainer selects stage data, the simulator 314 obtains the selected stage data from the data server 10 and places the avatar machine 100A, the transport vehicle 200, and the field in the virtual space in accordance with the stage data (step S101).

[0044] The input unit 311 acquires operation data of the operation device 33, posture data measured by the IMU of the head-mounted display 35, and primary gaze data measured by the eye tracker of the head-mounted display 35 (step S102).

[0045] Based on the operation data input in step S102, the simulator 314 simulates the behavior of the avatar machine 100A after a predetermined frame time (step S103). At this time, the simulator 314 calculates the positions and attitudes of the traveling body 110, the revolving body 120, the boom 131, the arm 132, and the bucket 133.

[0046] The simulator 314 determines the position and attitude of the rendering camera in the virtual space based on the position and attitude of the revolving unit 120 and the attitude data input in step S102 (step S104). The rendering unit 315 renders the virtual space in which the avatar machine 100A is placed from the rendering camera determined in step S104 (step S105). The display control unit 316 outputs the image data rendered in step S105 to the head-mounted display 35 (step S106).

[0047] The behavior recording unit 318 generates secondary gaze data indicating the direction of the trainer's gaze in the virtual space based on the position and posture of the rendering camera calculated in step S104 and the primary gaze data acquired in step S102 (step S107).The behavior recording unit 318 generates one frame of behavior record data based on the operation data acquired in step S102, the posture of the avatar machine 100A calculated in step S103, and the secondary gaze data generated in step S107 (step S108).

[0048] The behavior recording unit 318 determines whether the trainer's operation has ended (step S109). When the trainer determines that the operation has ended, the trainer instructs the trainer to end the operation by performing a predetermined operation on the training simulator 30, such as moving his / her gaze to an end button on the screen. If the trainer's operation has not ended (step S109: NO), the process returns to step S102, and the next frame is simulated.

[0049] On the other hand, when the trainer's operation has ended (step S109: YES), the behavior recording unit 318 compiles the generated multiple frames of behavior record data into model data (step S110). The behavior recording unit 318 also accepts input of summary data for the model data from the trainer (step S111). The transmission unit 320 transmits the model data and summary data to the data server 10 (step S112). In response, the data server 10 assigns a model ID to the received model data, and records the model ID, model data, and summary data in association with each other in the model data table T3.

[0050] 《Training Implementation》 Fig. 8 is a flowchart (part 1) showing a method for implementing training according to the first embodiment. Fig. 9 is a flowchart (part 2) showing a method for implementing training according to the first embodiment. When the trainee logs in to the training simulator 30 and selects to conduct training on the menu screen, the acquisition unit 312 accesses the data server 10 and acquires a list of model data recorded in the model data table T3 (step S201). The display control unit 316 outputs a selection screen to the head-mounted display 35 for selecting one model data from the acquired list (step S202). At this time, the selection screen displays a list of model data and summary data indicating the contents of the model data. The trainee reads the summary data and selects one model data.

[0051] The acquisition unit 312 accesses the data server 10 and acquires the model data selected by the trainee, the stage data associated with the model data, and the field data associated with the stage data (step S203). The simulator 314 places the avatar machine 100A, the ghost machine 100G, the delivery vehicle 200, and the field in the virtual space in accordance with the stage data (step S204). The avatar machine 100A and the ghost machine 100G are placed overlappingly at the same position.

[0052] The input unit 311 acquires operation data of the operation device 33, posture data measured by the IMU of the head-mounted display 35, and primary gaze data measured by the eye tracker of the head-mounted display 35 (step S205).

[0053] Based on the operation data input in step S205, simulator 314 simulates the behavior of avatar machine 100A after a predetermined frame time (step S206). At this time, simulator 314 calculates the positions and attitudes of traveling body 110, revolving body 120, boom 131, arm 132, and bucket 133. When avatar machine 100A comes into contact with transporter vehicle 200, simulator 314 stops the simulation of avatar machine 100A, and display control unit 316 displays a warning on head-mounted display 35 that avatar machine 100A has come into contact with transporter vehicle 200.

[0054] The simulator 314 calculates the position and attitude of the rendering camera in the virtual space based on the position and attitude of the revolving unit 120 and the attitude data input in step S205 (step S207).

[0055] The reproduction unit 313 reproduces the behavior of the ghost machine 100G by calculating the posture of the ghost machine 100G based on the time series of the joint angles of the work machine 130 and the rotation angle of the rotating body 120 included in the model data acquired in step S203 and the playback speed of the ghost machine 100G indicated by the setting data (step S208).

[0056] Next, rendering unit 315 renders the virtual space in which avatar machine 100A and ghost machine 100G are placed from the rendering camera with the position and orientation determined in step S207 (step S209). FIG. 10 shows an example of image data showing the virtual space in which avatar machine 100A and ghost machine 100G are placed according to the first embodiment. Display control unit 316 outputs the image data rendered in step S209 to head-mounted display 35 (step S210). Because rendering unit 315 generates image data for each frame time, the image data is treated as frame images of a moving image. Furthermore, by rendering the virtual space in which avatar machine 100A and ghost machine 100G are placed, rendering unit 315 can be said to have generated moving image data that simultaneously displays a first moving image in which avatar machine 100A is placed and a second moving image in which ghost machine 100G is placed. Furthermore, it can be said that rendering unit 315 generates moving image data that displays the first moving image in which avatar machine 100A appears and the second moving image in which ghost machine 100G appears superimposed in the same virtual space.

[0057] The behavior recording unit 318 generates secondary gaze data indicating the direction of the trainee's gaze in the virtual space based on the position and posture of the rendering camera calculated in step S207 and the primary gaze data acquired in step S205 (step S211). The behavior recording unit 318 generates one frame of behavior record data based on the operation data acquired in step S205, the posture of the avatar machine 100A calculated in step S206, and the secondary gaze data generated in step S211 (step S212).

[0058] The behavior recording unit 318 determines whether the training has ended (step S213). For example, the behavior recording unit 318 may determine that the training has ended when a certain time has elapsed since the playback position of the model data reached the end of the time series. If the training has not ended (step S213: NO), the process returns to step S205, and the simulation of the next frame is performed.

[0059] On the other hand, if the training has ended (step S213: YES), the behavior recording unit 318 compiles the generated behavior record data of multiple frames as work data (step S214). The evaluation unit 319 calculates an evaluation value for evaluating the trainee's operation based on the difference between the work data generated in step S214 and the model data acquired in step S203 (step S215). The transmission unit 320 transmits the work data generated in step S212, the model ID of the model data and the stage data ID of the stage data acquired in step S203, and the evaluation value calculated in step S215 to the data server 10 (step S216). As a result, the data server 10 assigns a work ID to the received work data and records the work ID, work data, model ID, stage ID, and evaluation value in association with each other in the work data table T4. Furthermore, the display control unit 316 outputs an evaluation screen displaying the evaluation values ​​calculated in step S215 to the head-mounted display 35 (step S217). The evaluation screen may display evaluation values ​​for each of the turning operation, the operation of the work machine 130, and the line of sight.

[0060] Actions and Effects In this way, the training system 1 according to the first embodiment accepts a designation of the type of transport vehicle 200 for the work machine 100 to load cargo, accepts a designation of the placement position of the transport vehicle 200, and generates stage data for placing the designated type of transport vehicle 200 at the placement position in the virtual space V. This allows the trainer to create stage data for any situation for training in loading onto a transport vehicle.

[0061] Other Embodiments Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design modifications are possible. That is, in other embodiments, the order of the above-described processes may be changed as appropriate. Furthermore, some processes may be executed in parallel.

[0062] The training system 1 according to the embodiment described above includes a data server 10 and a training simulator 30, but is not limited to this. For example, the training system 1 according to another embodiment may be configured with the training simulator 30 alone. In this case, training can be carried out based on model data recorded in advance in the training simulator 30. Furthermore, the training system 1 according to another embodiment may carry out training while referring to work data that the trainee has previously operated. Furthermore, in another embodiment, part of the training simulator 30 may be provided in an external computer. For example, in another embodiment, the training simulator 30 may only be responsible for inputting operation data and displaying calculation results, and simulation calculations may be performed by an external device.

[0063] Although the training system 1 according to the embodiment described above uses the head-mounted display 35 as the display device, this is not limiting. For example, in another embodiment, a large display used for remotely operating the work machine 100 may be used as the display device. In this case, the input unit 311 may receive designation of the position and orientation of the rendering camera from the operator. Examples of designated positions and orientations of the rendering camera include a subjective viewpoint (the driver's viewpoint), an objective viewpoint (diagonally behind the driver), an overhead viewpoint (to the side of the work machine 100), and a free viewpoint. The rendering unit 315 renders the virtual space according to the designated position and orientation of the rendering camera.

[0064] The training system 1 according to the embodiment described above limits the placement position of the transport vehicle 200 to the possible placement area G12 when generating stage data, but this is not limited to this. For example, the training system 1 according to another embodiment may set up the transport vehicle 200 in any location. At this time, the training system 1 may display the possible placement area G12 as reference information. Furthermore, for example, the training system 1 according to another embodiment may further limit the placement position of the transport vehicle 200 to within the possible placement area G13 when the movement range G13 of the work machine 100 overlaps with the excavation target G11.

[0065] Furthermore, the training system 1 according to the embodiment described above limits the candidates for the haulage vehicle 200 that can be deployed based on the type of work machine 100, but this is not limited to this. For example, the training system 1 according to another embodiment may be able to deploy any type of haulage vehicle 200. In this case, the training system 1 may display information about commonly used haulage vehicles 200 as reference information.

[0066] <Computer Configuration> FIG. 11 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. The computer 90 includes a processor 91 , a main memory 92 , a storage 93 , and an interface 94 . The training simulator 30 described above is implemented in a computer 90. The operations of the above-described processing units are stored in the form of a program in a storage 93. A processor 91 reads the program from the storage 93, loads it into a main memory 92, and executes the above-described processing in accordance with the program. The processor 91 also allocates storage areas in the main memory 92 corresponding to the above-described storage units in accordance with the program. Examples of the processor 91 include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and a microprocessor.

[0067] The program may be for realizing some of the functions to be performed by the computer 90. For example, the program may be combined with other programs already stored in storage or implemented in other devices to perform the functions. In another embodiment, the computer 90 may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions realized by the processor 91 may be realized by the integrated circuit. Such an integrated circuit is also an example of a processor. In another embodiment, the computer 90 may be virtualized on one or more computers.

[0068] Examples of storage 93 include a magnetic disk, a magneto-optical disk, an optical disk, and a semiconductor memory. Storage 93 may be an internal medium directly connected to the bus of computer 90, or an external medium connected to computer 90 via interface 94 or a communication line. Furthermore, when this program is distributed to computer 90 via a communication line, computer 90 that receives the program may load the program into main memory 92 and execute the above-described processing. In at least one embodiment, storage 93 is a non-transitory tangible storage medium.

[0069] The program may also be a program for realizing part of the above-described functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the above-described functions in combination with another program already stored in storage 93. [Explanation of symbols]

[0070] 1...Training system 10...Data server 100...Work machine 100A...Avatar machine 100G...Ghost machine 110...Traveling body 120...Swing body 121...Driver's cab 130...Work machine 131...Boom 132...Arm 133...Bucket 200...Transport vehicle 210...Dump body 220...Driver's cab 30...Training simulator 31...Calculation device 311...Input unit 312...Acquisition unit 313...Reproduction unit 314...Simulator 315...Rendering unit 316...Display control unit 317...Stage generation unit 318...Behavior recording unit 319...Evaluation unit 320...Transmission unit 321...Authentication unit 33...Operation device 35...Head-mounted display 90...Computer 91...Processor 92...Main memory 93...Storage 94...Interface G1...Plane map G11...Excavation target G12...Possible placement area G13...Movement range G2...Work machine icon G3...Transport vehicle icon T1...Field data table T2...Stage data table T3...Model data table T4...Work data table T5...User table V...Virtual space

Claims

1. A training system for operating a work machine arranged in a virtual space, comprising: Accepting a designation of a type of transport vehicle onto which the work machine is to load a load; Accepting a designation of a placement position of the transport vehicle; generating stage data for placing the specified type of transport vehicle at the placement position in the virtual space; A training system having a stage generation unit.

2. the stage generation unit is capable of specifying one or more types of transport vehicle from among a plurality of types of transport vehicle according to the type of the work machine; The training system of claim 1 .

3. the stage generation unit presents one type corresponding to the type of the work machine as a default type from among a plurality of types of transport vehicles; The training system of claim 1 .

4. the stage generation unit defines an area of ​​the virtual space that is not an excavation target as an area in which the transport vehicle can be placed. The training system of claim 1 .

5. The stage generation unit determines an area in which the transport vehicle can be placed based on the excavation target and a range of movement of a work implement provided in the work machine. The training system of claim 4.

6. The stage generation unit Accepting a designation of the direction in which the transport vehicle is facing; generating the stage data for placing the specified type of transport vehicle at the placement position in the virtual space, facing the specified orientation; The training system of claim 1 .

7. the stage generation unit presents an orientation in which the transport vehicle faces away from the excavation target at the specified placement position as a default orientation of the transport vehicle; The training system of claim 6.

8. a simulator that places the work machine and the transport vehicle in the virtual space in accordance with the stage data and simulates the behavior of the work machine; The training system according to any one of claims 1 to 7, comprising:

9. A method for generating stage data for a training system for operating a work machine arranged in a virtual space, comprising: a step in which a computer receives a designation of a type of transport vehicle onto which the work machine is to load a load; a step of receiving, by the computer, a designation of a placement position of the transport vehicle; generating stage data for placing the specified type of transport vehicle at the placement position in the virtual space by the computer; A stage data generation method comprising:

Images