Training system and method for controlling the training system

Abstract

In training for operating industrial machinery, it is easy to compare the operator's actions with those of other operators. [Solution] The control device drives actuators that act on the operator when the operating device is operated, based on model data including previously input operation data. The control device simulates changes in the posture of the work machine according to the amount of operation detected from the sensors of the operating device. The control device displays a first moving image representing the simulated change in the posture of the work machine on a display device.

Description

Technical Field

[0001] The present disclosure relates to a training system and a method for controlling the training system.

Background Art

[0002] Patent Document 1 discloses a technique for simulating the operation of a working machine while an operator refers to the operation of the working machine by himself or another operator. The technique described in Patent Document 1 reproduces a moving image of the past operation of a working machine by remote control and performs a simulation of the operation in the working environment related to the reproduction position when the moving image is stopped.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the simulation for training, it is required to simultaneously check the operation of a past working machine while performing the simulation of the operation. For example, a trainee who is a beginner can efficiently acquire the operation by following the operation of a trainer who is a skilled person. Also, for example, a trainee who is an intermediate level can notice minute differences in the operation by recognizing the differences from the operation of a trainer who is a skilled person in real time.

[0005] An object of the present disclosure is to provide a training system and a method for controlling the training system that can easily compare the operation by an operator with other operations in the training of the operation of a working machine.

Means for Solving the Problems

[0006] According to one aspect of the present disclosure, a training system is a training system for operating a work machine located in a virtual space, comprising: an operating device having an operating part movable with respect to an operating axis and a sensor for detecting the amount of operation of the operating part with respect to the operating axis; an actuator that acts on the operator when the operator operates the operating device; a display device; and a control device, which drives the actuator based on model data including previously input operation data, simulates changes in the posture of the work machine according to the amount of operation detected from the sensor, and displays a first moving image representing the simulated changes in the posture of the work machine on the display device. [Effects of the Invention]

[0007] According to the above embodiment, the training system allows for easy comparison between the operation performed by one operator and the operation performed by another operator during training on the operation of a work machine. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram showing the configuration of the training system according to the first embodiment. [Figure 2] This is a schematic perspective view showing the external configuration of the operating device according to the first embodiment. [Figure 3] This is a schematic perspective view showing the internal configuration of the operating device according to the first embodiment. [Figure 4] This is a block diagram showing the software configuration of the computing device included in the training simulator according to the first embodiment. [Figure 5] This is a flowchart showing a method for creating exemplary data according to the first embodiment. [Figure 6] This is a flowchart (Part 1) showing the method for conducting training according to the first embodiment. [Figure 7] This is a flowchart (Part 2) showing the method for conducting training according to the first embodiment. [Figure 8]This is an example of image data showing a virtual space in which the avatar machine and ghost machine according to the first embodiment are placed. [Figure 9] This is a schematic perspective view showing the internal configuration of the operating device according to the second embodiment. [Figure 10] This is a schematic block diagram showing the configuration of a computer according to at least one embodiment. [Modes for carrying out the invention]

[0009] <First Embodiment> 《Structure of Training System 1》 The embodiments will be described in detail below with reference to the drawings. Figure 1 is a schematic diagram showing the configuration of the training system 1 according to the first embodiment. The training system 1 according to the first embodiment is a system for trainees, who are operators unfamiliar with operating the work machine 100, to simulate the operation of the work machine 100 while referring to the operation of a trainer, who is an operator skilled in operating the work machine 100. The trainee can receive evaluation of their operation from the training system 1 and the trainer. Both the trainer and the trainee are operators of the work machine 100. In the first embodiment, the training system 1 is used for training in the operation of a hydraulic excavator, which is the work machine 100. Note that the type of work machine 100 is not limited to a hydraulic excavator, but may be other work machines such as a bulldozer, wheel loader, or forklift.

[0010] The training system 1 comprises a data server 10 and one or more training simulators 30.

[0011] The data server 10 stores model data and work data used for training. Model data represents the operation of the work machine 100 by the trainer. Trainees perform training by referring to the model data. Work data represents the operation of the work machine 100 by the trainee. The training system 1 and the trainer evaluate the trainee's operation based on the work data. The model data and work data include time series of lever device operation and time series of the posture of the work machine 100 (joint angles and swivel angles of the work machine 130). Hereinafter, data representing time series of operation and behavior of the work machine 100, such as model data and work data, will also be referred to as behavior record data.

[0012] Specifically, the data server 10 includes a model data table T1 for storing model data, a work data table T2 for storing work data, and a user table T3 for storing authentication data. The model data table T1 stores model data, the model ID (identification information) of the model data, and summary data indicating the work content that can be reproduced using the model data, in association with each other. Work data table T2 stores work data, the work ID (identification information) of the work data, the model ID of the corresponding model data, evaluation data indicating the evaluation by training system 1, and comment data indicating the evaluation by the trainer, all linked together. The user table T3 stores the user ID, user category, and authentication information in association with each other. The user category value is either trainer or trainee. The authentication information may be, for example, a password.

[0013] The training simulator 30 receives operation input from the operator and simulates the behavior of the work machine 100 in response to that operation input. The training simulator 30 generates a video representing the behavior of the work machine 100 and presents it to the operator. The training simulator 30 also reproduces the behavior of the work machine 100 based on model data stored in the data server 10, generates a video representing the behavior of the work machine 100, and presents it to the operator.

[0014] The training simulator 30 includes an arithmetic unit 31, an operating device 33, and a head-mounted display 35. The operating device 33 is an input interface for operating the working machine 100 arranged in the virtual space V. The operating device 33 according to the first embodiment includes two lever devices (a right lever device and a left lever device). Note that the operating device 33 may differ depending on the type of the working machine 100 for which the simulation is performed. The head-mounted display 35 displays the video calculated by the arithmetic unit 31. The head-mounted display 35 includes an IMU (Inertial Measurement Unit) that detects the posture of the head-mounted display 35. For example, the posture of the head-mounted display 35 refers to the relative orientation and position of the head-mounted display 35 with respect to a reference position determined by initial settings or the like. The arithmetic unit 31 performs a simulation of the working machine 100 in the virtual space V based on the input of the operating device 33. The arithmetic unit 31 determines the direction of the line of sight in the virtual space V based on the posture of the head-mounted display 35, and renders the simulation result. As a result, the virtual space V is displayed on the head-mounted display 35 in conjunction with the posture of the head-mounted display 35.

[0015] 《Configuration of the operating device 33》 The training simulator 30 according to the first embodiment feeds back the operation of the working machine 100 by the trainer represented by the reference data to the operating device 33. That is, the training simulator 30 operates the operating device 33 according to the reference data. The operating device 33 includes left and right lever devices and left and right travel pedals. The lever device can input an operation amount with respect to a first operation axis in the left-right direction and a second operation axis in the front-back direction. The travel pedal can input an operation amount with respect to an operation axis in the front-back direction. The lever device and the travel pedal can receive operations in both the positive and negative directions with respect to the operation axis. The positive and negative directions are relative input directions with respect to the origin (center position) of the operation axis. For example, when operating the lever device in the front-back direction, the movement of operating in the forward direction is a positive-direction operation, and the movement of operating in the backward direction is a negative-direction operation. For example, when operating the lever device in the left-right direction, the movement of operating in the right direction is a positive-direction operation, and the movement of operating in the left direction is a negative-direction operation. Here, the configuration of the lever device will be described.

[0016] FIG. 2 is a perspective view schematically showing the external configuration of the operating device 33 according to the first embodiment. FIG. 3 is a perspective view schematically showing the internal configuration of the operating device 33 according to the first embodiment. As shown in FIG. 2, the operating device 33 includes a lever 331, a first support frame 332, a second support frame 333, a third support frame 334, a first motor 335, a second motor 336, a first rotation angle sensor 337, and a second rotation angle sensor 338. The lever 331 is an operation part that is gripped by an operator and is movable with respect to a first operation axis in the left-right direction and a second operation axis in the front-back direction. The first support frame 332 is fixed to the training simulator 30 and supports the lever 331 so as to be tiltable in the front-back, left, and right directions via the second support frame 333 and the third support frame 334. As shown in FIG. 2, the first support frame 332 has a box shape with an opening at the top. Through holes are formed in each side surface of the first support frame 332.

[0017] The second support frame 333 rotates in accordance with the tilt of the lever 331 in the front-rear direction. The second support frame 333 does not rotate with respect to the tilt of the lever 331 in the left-right direction, but supports the left-right rotation of the lever 331. The second support frame 333 is positioned inside the first support frame 332 and is rotatable relative to the first support frame 332. As shown in Figure 3, the second support frame 333 is formed in an inverted U shape when viewed along the front-rear direction. That is, the second support frame 333 has a top surface and a pair of side surfaces. The top surface is provided with a through hole formed along the left-right direction. The width of the through hole in the front-rear direction is set to be approximately the same length as the diameter of the lever 331. The lever 331 tilts left-right along the through hole. A pair of shafts 333a are provided on the sides of the second support frame 333, extending outward in the left-right direction. The shafts 333a are rotatably inserted into through holes in the sides of the first support frame 332.

[0018] The third support frame 334 rotates in accordance with the left-right tilt of the lever 331. The third support frame 334 does not rotate with respect to the front-rear tilt of the lever 331, but supports the front-rear rotation of the lever 331. The third support frame 334 is positioned inside the first support frame 332 and rotatably relative to the first support frame 332. The third support frame 334 is positioned inside the second support frame 333. As shown in Figure 3, the third support frame 334 has an opening in the vertical direction. The third support frame 334 is a rectangular shape formed to be elongated in the front-rear direction when viewed along the vertical direction. The lever 331 is provided so as to pass through the third support frame 334. The lever 331 tilts along the front-rear direction of the third support frame 334. The third support frame 334 has a shaft 334a on each of its pair of front and rear sides that protrude outward along the front-rear direction. The shaft 334a is rotatably inserted into a through hole on the side of the first support frame 332.

[0019] The lever 331 has shafts 331a protruding to the left and right from its base end. The shafts 331a are rotatably inserted into through holes on the left and right sides of the third support frame 334. The shafts 331a and the shafts 333a of the second support frame 333 described above are arranged coaxially. The pair of shafts 334a of the third support frame 334 are arranged coaxially.

[0020] The first motor 335 is connected to the shaft 334a of the third support frame 334. The first motor 335 rotates the shaft 334a according to a specified rotation angle. This allows the first motor 335 to tilt the lever 331 in the left-right direction. The second motor 336 is connected to the shaft 333a of the second support frame 333. The second motor 336 rotates the shaft 333a according to a specified rotation angle. This allows the second motor 336 to tilt the lever 331 in the forward and backward directions. The first motor 335 and the second motor 336 are electric motors capable of position control, such as servo motors and stepping motors. The first motor 335 and the second motor 336 may also be hydraulic motors. In other embodiments, the operating device 33 may include actuators other than motors, such as power cylinders.

[0021] The first rotation angle sensor 337 is connected to the shaft 334a of the third support frame 334. The first rotation angle sensor 337 detects the tilt position, i.e., the amount of operation, of the lever 331 in the left-right direction by detecting the rotation angle of the shaft 334a. In other words, the first rotation angle sensor 337 detects the amount of operation of the lever 331 with respect to the left-right operation axis. The second rotation angle sensor 338 is connected to the shaft 333a of the second support frame 333. The second rotation angle sensor 338 detects the tilt position of the lever 331 in the front-rear direction, i.e., the amount of operation, by detecting the rotation angle of the shaft 333a. In other words, the second rotation angle sensor 338 detects the amount of operation of the lever 331 with respect to the operating axis in the front-rear direction. The first rotation angle sensor 337 and the second rotation angle sensor 338 may be, for example, potentiometers.

[0022] 《Configuration of the 100th work machine》 The work machine 100 comprises a traveling body 110, a rotating body 120, and a work implement 130. The traveling body 110 supports the work machine 100 so that it can move. The rotating body 120 is supported by the traveling body 110 so as to be able to rotate around the pivot point. A driver's cab 121 is provided at the front of the rotating body 120. A rendering camera for drawing the virtual space V by the computing device 31 is provided inside the driver's cab 121. The work implement 130 is supported on the front of the rotating body 120 so as to be movable in the vertical direction. The work machine 130 comprises a boom 131, an arm 132, and a bucket 133 as a work tool. Other examples of work tools include end attachments such as clam buckets, tilt buckets, tilt-rotate buckets, breakers, and grapplers.

[0023] The base end of the boom 131 is rotatably attached to the slewing body 120 via a boom pin. In the work machine 100 shown in Figure 1, the boom 131 is located in the central front portion of the slewing body 120, but it is not limited to this, and the boom 131 may be mounted offset in the left-right direction. In this case, the pivot center of the slewing body 120 is not located on the operating plane of the work machine 130. The arm 132 connects the boom 131 and the bucket 133. The base end of the arm 132 is rotatably attached to the tip of the boom 131 via an arm pin. The bucket 133 is rotatably attached to the tip of the arm 132 via a pin. The bucket 133 functions as a container for holding excavated soil.

[0024] 《Configuration of Training Simulator 30》 Figure 4 is a block diagram showing the software configuration of the computing device 31 included in the training simulator 30 according to the first embodiment. The arithmetic unit 31 of the training simulator 30 includes an input unit 311, an acquisition unit 312, a reproduction unit 313, a simulator 314, a rendering unit 315, a display control unit 316, a generation unit 317, an evaluation unit 318, a transmission unit 319, a setting storage unit 320, a comment unit 321, and an authentication unit 322.

[0025] The input unit 311 acquires operation data obtained from the first rotation angle sensor 337 and the second rotation angle sensor 338 of the operating device 33, as well as attitude data measured by the IMU of the head-mounted display 35.

[0026] The acquisition unit 312 acquires behavioral recording data (model data and work data) from the data server 10. The reproduction unit 313 reproduces the behavior of the work machine 100 based on the behavior recording data acquired by the acquisition unit 312. Hereinafter, the work machine 100 reproduced in the virtual space V by the reproduction unit 313 will be referred to as the ghost machine 100G. The reproduction unit 313 reproduces the behavior of the work machine 100 by arranging the ghost machine 100G according to the time series of joint angles of the work machine 130 and rotation angles of the swivel body 120 included in the behavior recording data. In other embodiments, the reproduction unit 313 may, for example, simulate the behavior of the work machine 100 based on time series data of the movement of the lever device included in the behavior recording data and arrange the ghost machine 100G. The reproduction unit 313 reproduces the operation of the operating device 33 based on the behavior recording data acquired by the acquisition unit 312. The reproduction unit 313 controls the position of the lever 331 by outputting angle commands to the first motor 335 and the second motor 336 of the operating device 33 according to the time-series data of the movement of the lever device included in the behavior recording data. In other words, the reproduction unit 313 controls the tilt (inclination) of the lever 331 by outputting angle commands to the first motor 335 and the second motor 336 of the operating device 33 according to the time-series data of the movement of the lever device included in the behavior recording data.

[0027] Based on the operation data input to the input unit 311, the simulator 314 places the field and the work machine 100 in the virtual space V and simulates the behavior of the work machine 100. Hereinafter, the work machine 100 simulated by the simulator 314 will be referred to as the avatar machine 100A. The simulator 314 simulates the behavior of the avatar machine 100A by calculating the angular velocity of the rotating body 120 and the work machine 130 according to the operation amount indicated by the operation data.

[0028] The rendering unit 315 renders the ghost machine 100G and the avatar machine 100A and generates image data. The rendering camera in the virtual space V is located inside the driver's cab 121 of the avatar machine 100A. The rendering camera is directed in the direction of the front of the head-mounted display 35, which is indicated by the posture data input to the input unit 311, with respect to the front of the avatar machine 100A. The rendering by the rendering unit 315 is performed at predetermined frame rates. Therefore, the image data generated by the rendering unit 315 is treated as frame images of a moving image.

[0029] The display control unit 316 outputs the image data drawn by the rendering unit 315 to the head-mounted display 35.

[0030] The generation unit 317 generates behavioral record data based on the operation data and attitude data input to the input unit 311, as well as the attitude of the avatar machine 100A simulated by the simulator 314. Specifically, the generation unit 317 obtains the positions and attitudes of the avatar machine 100A's running body 110, slewing body 120, boom 131, arm 132, and bucket 133 in the virtual space V based on the attitude of the avatar machine 100A simulated by the simulator 314. Each position and attitude is represented in the virtual space coordinate system, which is a three-dimensional orthogonal coordinate system that defines the virtual space V.

[0031] The evaluation unit 318 evaluates the trainee's operation based on the difference between the model data acquired by the acquisition unit 312 and the work data generated by the generation unit 317. For example, the evaluation unit 318 calculates the distance between the position and posture of the work machine 130 in the virtual space V shown by the model data and the work data. For example, the evaluation unit 318 determines the distance between the model data and the work data for each of the time series of the position and posture of the traveling body 110, the slewing body 120, the boom 131, the arm 132, and the bucket 133, as well as the time series of posture data, and calculates a weighted sum of the distances as the evaluation value. In this case, the closer the evaluation value is to zero, the higher the evaluation. The distances in the time series may be calculated using the DTW method.

[0032] The transmitting unit 319 transmits the behavior recording data generated by the generating unit 317 to the data server 10.

[0033] The setting memory unit 320 stores the setting data for the training simulator 30. The setting data stores the playback speed of the ghost machine 100G, whether or not the ghost machine 100G is displayed, whether or not the trajectory of the blade tip of the ghost machine 100G's tool is displayed, and whether or not the trainer's lever operation is displayed. The setting data can be updated by the trainee's operation.

[0034] The comment unit 321 accepts comments for the work data. A playback timing can be associated with the comments. This allows the comment unit 321 to be set so that when the work data is played back, the comments are displayed at a predetermined playback timing.

[0035] The authentication unit 322 authenticates the operator of the training simulator 30 based on the authentication information stored in the data server 10.

[0036] 《Processing of Training System 1》 When the training simulator 30 starts up, the authentication unit 322 displays a login screen on the head-mounted display 35. The login screen displays input forms for, for example, a user ID and authentication information. When the operator enters the user ID and authentication information values ​​on the login screen, the authentication unit 322 retrieves the authentication information associated with the entered user ID from the user table T3 of the data server 10 and verifies the authentication information.

[0037] If the authentication unit 322 successfully verifies the information, the display control unit 316 displays a menu screen on the head-mounted display 35. If the operator's user category is "Trainer," the menu screen displays a Trainer menu that allows them to choose between creating model data or evaluating work data. If the user category of the operator is "Trainee," the menu screen will display a Trainee menu that allows the user to select one of the following options: conduct training, view evaluation results, or change settings.

[0038] Generating exemplary data Figure 5 is a flowchart showing the method for creating exemplary data according to the first embodiment. When the trainer logs into the training simulator 30 and selects "Create Model Data" from the menu screen, the simulator 314 places the avatar machine 100A in its initial pose and the field in the virtual space V (step S101). For example, the field represents the terrain on which the training will be conducted.

[0039] The input unit 311 acquires operation data obtained from the first rotation angle sensor 337 and the second rotation angle sensor 338 of the operating device 33, and attitude data measured by the IMU of the head-mounted display 35 (step S102).

[0040] Based on the operation data entered in step S102, the simulator 314 simulates the behavior of the avatar machine 100A after a predetermined frame time (step S103). At this time, the simulator 314 calculates the position and attitude of the vehicle 110, the slewing body 120, the boom 131, the arm 132, and the bucket 133.

[0041] The simulator 314 determines the position and orientation of the rendering camera in the virtual space V based on the position and orientation of the work machine 100 and the orientation data input in step S102 (step S104). The rendering unit 315 draws the virtual space V in which the avatar machine 100A is placed from the rendering camera determined in step S104 (step S105). The display control unit 316 outputs the image data drawn in step S105 to the head-mounted display 35 (step S106).

[0042] The generation unit 317 generates behavioral recording data for one frame based on the operation data acquired in step S102 and the pose of the avatar machine 100A calculated in step S103 (step S107).

[0043] The generation unit 317 determines whether the trainer has finished their operation (step S108). If the trainer determines that they have finished their operation, they instruct the training simulator 30 to end the operation by performing a predetermined action, such as shifting their gaze to the end button on the screen. If the trainer has not finished their operation (step S108: NO), the process returns to step S102 and the simulation of the next frame is performed.

[0044] On the other hand, when the trainer's operation is completed (step S108: YES), the generation unit 317 compiles the generated behavioral recording data of multiple frames as model data (step S109). The generation unit 317 also accepts summary data about the model data from the trainer (step S110). The transmission unit 319 sends the model data and summary data to the data server 10 (step S111). As a result, the data server 10 assigns a model ID to the received model data and records the model ID, model data, and summary data in the model data table T1, associating them with each other.

[0045] 《Implementation of training》 Figure 6 is a flowchart (Part 1) showing the method of conducting training according to the first embodiment. Figure 7 is a flowchart (Part 2) showing the method of conducting training according to the first embodiment. When a trainee logs into the training simulator 30 and selects to perform training from the menu screen, the acquisition unit 312 accesses the data server 10 and retrieves a list of model data recorded in the model data table T1 (step S201). The display control unit 316 outputs a selection screen to the head-mounted display 35 for selecting one model data from the acquired list (step S202). At this time, the selection screen displays a list of model data and summary data indicating the content of the model data. The trainee reads the summary data and selects one model data.

[0046] The acquisition unit 312 accesses the data server 10 and acquires the model data selected by the trainee from the model data table T1 (step S203). The simulator 314 places the avatar machine 100A, the ghost machine 100G, and the field in the virtual space V (step S204). The avatar machine 100A and the ghost machine 100G are placed in the same location, overlapping each other.

[0047] The reproduction unit 313 generates angle commands for the first motor 335 and the second motor 336 of the operating device 33 based on the operation data included in the model data acquired in step S203, and controls the first motor 335 and the second motor 336 (step S205). This allows the reproduction unit 313 to reproduce the operation of the operating device 33 by the trainer. In other words, the operating device 33 operates to reproduce the amount of operation performed by the trainer, which is the model data. The trainee can feel the amount of operation performed by the trainer on the operating device 33, controlled by the first motor 335 and the second motor 336, via the lever 331. In other words, the first motor 335 and the second motor 336 act on the trainee via the lever 331.

[0048] The input unit 311 acquires operation data from the operating device 33 and attitude data measured by the IMU of the head-mounted display 35 (step S206).

[0049] Based on the operation data entered in step S206, the simulator 314 simulates the behavior of the avatar machine 100A after a predetermined frame time (step S207). At this time, the simulator 314 calculates the position and orientation of the vehicle 110, the slewing body 120, the boom 131, the arm 132, and the bucket 133. Based on the position and orientation of the work machine 100, as well as the orientation data entered in step S206, the simulator 314 determines the position and orientation of the rendering camera in the virtual space V (step S208).

[0050] The reproduction unit 313 reproduces the behavior of the ghost machine 100G by calculating the attitude of the ghost machine 100G based on the time series of joint angles of the work machine 130 and the rotation angle of the slewing body 120 included in the model data acquired in step S203, and the playback speed of the ghost machine 100G indicated by the setting data (step S209).

[0051] Next, the rendering unit 315 renders the virtual space V in which the avatar machine 100A and ghost machine 100G are located, using the rendering camera at the position and orientation determined in step S208 (step S210). Figure 8 is an example of image data showing the virtual space V in which the avatar machine 100A and ghost machine 100G are located according to the first embodiment.

[0052] The display control unit 316 outputs the image data drawn in step S210 to the head-mounted display 35 (step S211). Since the rendering unit 315 generates image data for each frame time, the image data is treated as a frame image of a moving image. Furthermore, by rendering the virtual space V in which the avatar machine 100A and the ghost machine 100G are placed, the rendering unit 315 can be said to have generated moving image data that simultaneously displays the first moving image showing the avatar machine 100A and the second moving image showing the ghost machine 100G. Furthermore, the rendering unit 315 can be said to have generated moving image data that overlays the first moving image showing the avatar machine 100A and the second moving image showing the ghost machine 100G onto the same virtual space V.

[0053] The generation unit 317 generates behavioral recording data for one frame based on the operation data acquired in step S206 and the pose of the avatar machine 100A calculated in step S207 (step S212).

[0054] The generation unit 317 determines whether or not training has been completed (step S213). For example, the generation unit 317 may determine that training has been completed when a certain amount of time has elapsed since the playback position of the model data reached the end of the time series. If training has not been completed (step S213: NO), the reproduction unit 313 determines whether or not an operation resisting the first motor 335 or the second motor 336 has been performed, based on the operation data acquired in step S206 (step S214). An operation resisting the first motor 335 or the second motor 336 is an operation that moves the lever 331 to a position different from the operation position positioned by the first motor 335 and the second motor 336. For example, the reproduction unit 313 may determine that an operation resisting has been performed when the difference between the operation position reproduced in step S205 and the operation position indicated by the operation data acquired in step S206 exceeds a predetermined threshold. Furthermore, for example, the reproduction unit 313 may determine that a resisting operation has been performed if it is estimated from the change in voltage of the first motor 335 or the second motor 336 that an external force is acting on the motor. If the first motor 335 and the second motor 336 are hydraulic motors, the reproduction unit 313 may determine that a resisting operation has been performed if the internal pressure of the circuit including the motors exceeds the relief pressure.

[0055] If no resistance operation is performed on the first motor 335 or the second motor 336 (step S214: NO), the reproduction unit 313 determines whether or not the reproduction of the operating device 33 has been stopped (step S215). If the reproduction of the operating device 33 has not been stopped (step S215: NO), the process returns to step S205. This allows the training simulator 30 to continue reproducing the trainer's operation on the operating device 33.

[0056] On the other hand, if an operation is performed that resists the first motor 335 or the second motor 336 (step S214: YES), the reproduction unit 313 temporarily records in memory that the reproduction of the operating device 33 has been stopped (step S216) and returns to step S206. Also, if the reproduction of the operating device 33 has already been stopped (step S215: YES), returns to step S206. As a result, the training simulator 30 skips the reproduction of the trainer's operation on the operating device 33.

[0057] On the other hand, if training is completed (step S213: YES), the generation unit 317 compiles the generated behavioral recording data of multiple frames as work data (step S217). The evaluation unit 318 calculates an evaluation value to assess the trainee's operation based on the difference between the work data generated in step S217 and the model data acquired in step S203 (step S218). The transmission unit 319 transmits the work data generated in step S217, the model ID of the model data acquired in step S203, and the evaluation value calculated in step S218 to the data server 10 (step S219). As a result, the data server 10 assigns a work ID to the received work data and records the work ID, work data, model ID, and evaluation value in association in the work data table T2. Furthermore, the display control unit 316 outputs an evaluation screen to the head-mounted display 35 that displays the evaluation values ​​calculated in step S218 (step S220). The evaluation screen may display evaluation values ​​for both the slewing operation and the work equipment 130 operation.

[0058] Action / Effect As described above, the training system 1 according to the first embodiment performs the following processes. The operating device 33 is a device that can be operated in the positive and negative directions relative to the operating axis. The operating device 33 comprises a lever 331 that is movable relative to the operating axis, a first rotation angle sensor 337 and a second rotation angle sensor 338 that detect the amount of operation of the lever 331 relative to the operating axis, and a first motor 335 and a second motor 336 that move the lever 331 relative to the operating axis. The calculation device 31 drives the first motor 335 and the second motor 336 to reproduce the operation by the trainer based on model data including operation data previously input by the trainer. The calculation device 31 simulates changes in the posture of the work machine according to the amount of operation detected from the first rotation angle sensor 337 and the second rotation angle sensor 338. The calculation device 31 displays a moving image representing the simulated change in the posture of the work machine 100 on the head-mounted display 35. This allows the trainee to experience the amount of operation of the operating device 33 by the trainer through the operating device 33. Therefore, trainees can easily compare their own operation with that of the trainer during training on operating work machinery.

[0059] Furthermore, the training system 1 according to the first embodiment allows the trainee to perform operations on the operating device 33 that resist the motor. When an operation that resists the motor is performed on the operating device 33, the training system 1 stops reproducing the operation of the operating device 33 based on the model data. This allows the training system 1 to prioritize and accept the trainee's proactive operations and display the simulation results according to the trainee's operations. The training system 1 according to the first embodiment stops reproducing the operation of the operating device 33 based on the model data when an operation that resists the motor is performed on the operating device 33. This is because once an operation deviates from the model operation performed by the trainer, subsequent operations may not necessarily be model operations.

[0060] In the first embodiment, the training system 1 renders both a ghost machine 100G representing the trainer's operations and an avatar machine 100A representing the trainee's operations. However, in other embodiments, the training system 1 may render only one of the ghost machine 100G or the avatar machine 100A. The training system 1 ensures that the trainee's operations and the trainer's operations match unless the trainee resists the lever 331. Therefore, in other embodiments, the training system 1 can reduce computational complexity by rendering only one of the ghost machine 100G or the avatar machine 100A. Furthermore, the training system 1 in the first embodiment renders a graphic representing the trainer's operation of the lever 331, as shown in Figure 8, but is not limited to this. In other embodiments, the training system 1 may not render a graphic representing the trainer's operation of the lever 331, and may only display the trainee's actual operation of the lever 331.

[0061] <Second Embodiment> In the first embodiment, the training system 1 stops reproducing the operation of the operating device 33 based on the model data when an operation resisting the motor is performed on the operating device 33. On the other hand, in the second embodiment, the training system 1 simulates the operation of the operating device 33 based on the model data and the operation performed by the trainee when an operation resisting the motor is performed on the operating device 33.

[0062] Figure 9 is a schematic perspective view showing the external configuration of the operating device 33 according to the second embodiment. The first motor 335 and the second motor 336 according to the second embodiment are motors capable of torque control. A first torque sensor 340 is provided between the first motor 335 and the lever 331 on the shaft 334a that transmits power from the first motor 335. A second torque sensor 341 is provided between the second motor 336 and the lever 331 on the shaft 333a that transmits power from the second motor 336. The first torque sensor 340 measures the magnitude of the torque generated on the shaft 334a by the left-right operation of the lever 331 by the trainee and the rotation of the first motor 335. The second torque sensor 341 measures the magnitude of the torque generated on the shaft 333a by the front-back operation of the lever 331 by the trainee and the rotation of the second motor 336.

[0063] In the second embodiment, the reproduction unit 313 can determine whether or not an operation resisting the first motor 335 or the second motor 336 has been performed, based on the measurement data from the first torque sensor 340 and the second torque sensor 341. In the second embodiment, the reproduction unit 313 does not stop reproducing the operation of the operating device 33 even if an operation resisting the first motor 335 or the second motor 336 has been performed.

[0064] Instead, the reproduction unit 313 according to the second embodiment changes the torque of the first motor 335 and the second motor 336 based on the measurement data of the first torque sensor 340 and the second torque sensor 341. For example, the reproduction unit 313 determines the resistance torque due to the trainee by calculating the root of the sum of the squares of the torque magnitudes indicated by the measurement data of the first torque sensor 340 and the second torque sensor 341. Next, the reproduction unit 313 determines a target value for the motor torque by multiplying the motor's reference torque by the ratio of the current torque of the motor to the sum of the current torque and the resistance torque. The reproduction unit 313 controls the torque of the first motor 335 and the second motor 336 according to the determined target value.

[0065] The reproduction unit 313 increases the motor torque when the trainee's operation is weak, and decreases the motor torque as the trainee's operation becomes stronger. This allows the reproduction unit 313 to simulate the operation of the control device 33 based on the model data and the operation performed by the trainee, while reflecting the trainee's intentions. In other words, when the trainee operates the control device 33 with intent, the reproduction unit 313 reduces the motor torque so as not to hinder the trainee's operation, and when the trainee places their hand on the control device 33 and imitates the model data, the reproduction unit 313 increases the motor torque to reproduce the trainer's operation.

[0066] Since the simulator 314 simulates the behavior of the avatar machine 100A based on operation data, according to the second embodiment, the avatar machine 100A is simulated according to an operation that combines the operation of the operating device 33 based on model data and the operation by the trainee.

[0067] In another embodiment, the reproduction unit 313 may not change the torque of the first motor 335 and the second motor 336 based on the trainee's operation, and may be configured so that the lever 331 behaves according to the model data even when an external force is applied. In this case, the reproduction unit 313 may estimate the force the trainee applies to the lever 331 based on the measurement data of the first torque sensor 340 and the second torque sensor 341, and perform a simulation by combining the trainee's operation amount estimated from this force with the operation amount of the model data. The simulator 314 simulates the behavior of the avatar machine 100A based on the operation amount obtained by combining the operation amount estimated from the force the trainee applies to the lever 331 with the operation amount of the model data. In this case, the trainee cannot move the lever 331, but can change the behavior of the avatar machine 100A according to the force applied to the lever 331.

[0068] In other embodiments, the reproduction unit 313 may control the motor according to the acceleration components of the operating device 33 shown in the model data. In this case, the reproduction unit 313 controls the motor to vibrate the lever 331 at the start and stop timings of lever 331 operation, and at the timing of changes in the direction of operation. This allows the reproduction unit 313 to communicate to the trainee the timing of changes in lever 331 operation, etc.

[0069] <Other Embodiments> Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to that described above, and various design changes are possible. In other embodiments, the order of the above-described processes may be changed as appropriate. Also, some processes may be executed in parallel.

[0070] The training system 1 according to the above-described embodiment comprises a data server 10 and a training simulator 30, but is not limited thereto. For example, the training system 1 according to another embodiment may consist of the training simulator 30 alone. In this case, the training simulator 30 can perform training based on pre-recorded model data. The training system 1 according to another embodiment may also perform training while referring to work data previously operated by the trainee. In another embodiment, part of the training simulator 30 may be provided on an external computer. For example, in another embodiment, the training simulator 30 may only be responsible for inputting operation data and displaying calculation results, while the simulation calculations are performed by an external device.

[0071] The training system 1 according to the above embodiment uses a head-mounted display 35 as a display device, but is not limited thereto. For example, in another embodiment, a large display used for remote control of the work machine 100 may be used as a display device. In this case, the input unit 311 may accept the operator's specification of the position and orientation of the rendering camera. Examples of the specification of the rendering camera's position and orientation include subjective viewpoint (driver's viewpoint), objective viewpoint (diagonally behind the driver), overhead viewpoint (side of the work machine 100), and free viewpoint. The rendering unit 315 renders the virtual space V according to the specified position and orientation of the rendering camera.

[0072] The training system 1 according to the above embodiment reproduces the trainer's operation by driving the lever 331 with a first motor 335 and a second motor 336 provided on the operating device 33, but is not limited to this. For example, the training system 1 according to another embodiment may present the trainer's operation to the trainee by driving an actuator acting on the trainee's hand (for example, a glove-type tactile presentation device or a tactile presentation device provided on the upper surface of the lever 331) according to model data without driving the lever 331.

[0073] In other embodiments of the training system 1, the reproduction speed of the model data by the reproduction unit 313 may be changed. In this case, the evaluation unit 318 calculates an evaluation value by aligning the time axes of the work data and the model data and determining the similarity.

[0074] The training system 1 according to the above embodiment reproduces the trainer's operations by including operation data previously input by the trainer in the model data, but is not limited to this. For example, the model data of the training system 1 according to other embodiments may include operation data obtained by optimization calculations or machine learning. For example, the training system 1 according to other embodiments may generate model data by performing optimization calculations that generate candidate operation data using an objective function that minimizes the execution time of the task content indicated by the summary data and repeat simulations. Alternatively, the training system 1 according to other embodiments may generate model data by repeatedly running simulations in reinforcement learning where a reward is given when the task content indicated by the summary data is performed. Furthermore, the training system 1 according to other embodiments may generate model data from a time series of operation data previously input by the trainer using statistical methods.

[0075] <Computer Configuration> Figure 10 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. The computer 90 includes a processor 91, main memory 92, storage 93, and an interface 94. The training simulator 30 described above is implemented in the computer 90. The operation of each processing unit described above is stored in storage 93 in the form of a program. The processor 91 reads the program from storage 93, loads it into main memory 92, and executes the above processing according to the program. The processor 91 also allocates memory areas in main memory 92 corresponding to each of the above-mentioned memory units according to the program. Examples of the processor 91 include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a microprocessor.

[0076] The program may be for implementing some of the functions that the computer 90 is to perform. For example, the program may perform functions in combination with other programs already stored in storage, or in combination with other programs implemented on other devices. In other embodiments, the computer 90 may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to, or instead of, the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions implemented by the processor 91 may be implemented by the integrated circuit. Such an integrated circuit is also included as an example of a processor. In other embodiments, the computer 90 may be virtualized on one or more computers.

[0077] Examples of storage 93 include magnetic disks, magneto-optical disks, optical disks, and semiconductor memory. Storage 93 may be an internal medium directly connected to the bus of the computer 90, or it may be an external medium connected to the computer 90 via an interface 94 or a communication line. Furthermore, if this program is delivered to the computer 90 via a communication line, the computer 90 that receives the delivery may load the program into the main memory 92 and execute the above processing. In at least one embodiment, storage 93 is a tangible storage medium that is not temporary.

[0078] Furthermore, the program may be intended to implement some of the functions described above. In addition, the program may be a so-called differential file (differential program) that implements the functions described above in combination with other programs already stored in storage 93. [Explanation of Symbols]

[0079] 1...Training system 10...Data server 100...Working machine 100A...Avatar machine 100G...Ghost machine 110...Roading body 120...Slewing body 121...Operator's cab 130...Working machine 131...Boom 132...Arm 133...Bucket 30...Training simulator 31...Calculation unit 311...Input unit 312...Acquisition unit 313...Reproduction unit 314...Simulator 315...Rendering unit 316...Display control unit 317...Generation unit 318...Evaluation unit 319...Transmission unit 320...Setting memory unit 321...Comment unit 322...Authentication unit 33...Operating device 331...Lever 331a...Axis 332...First support frame 333...Second support frame 333a...Axis 334...Third support frame 334a...Axis 335...First motor 336…Second motor 337…First rotation angle sensor 338…Second rotation angle sensor 35…Head-mounted display 90…Computer 91…Processor 92…Main memory 93…Storage 94…Interface T1…Model data table T2…Work data table T3…User table V…Virtual space

Claims

1. A training system for operating work machines placed in a virtual space, An operating device comprising an operating part that is movable relative to an operating shaft, and a sensor that detects the amount of operation of the operating part relative to the operating shaft, An actuator that acts on the operator when the operator operates the operating device, Display device and Control device and Equipped with, The actuator is driven based on the model data. The change in the posture of the work machine is simulated according to the amount of operation detected by the aforementioned sensor. The first moving image representing the simulated change in the posture of the work machine is displayed on the display device. Training system.

2. The actuator moves the operating part relative to the operating shaft. The training system according to claim 1.

3. Based on the aforementioned model data, a second video is generated that reproduces the behavior of the work machine. The first video and the second video are displayed superimposed on each other. The training system according to claim 1.

4. If an operation that resists the actuator is performed in the aforementioned operating device, the operation of the actuator based on the example data is stopped. The training system according to claim 2.

5. If an operation resisting the actuator is performed in the control device, the operation of the actuator based on the model data is stopped until the second video, which reproduces the behavior of the work machine based on the model data, ends. The training system according to claim 4.

6. Even after an operation is performed in the aforementioned operating device that resists the actuator, the simulation of the change in the posture of the work machine based on the amount of operation continues. The training system according to claim 4 or claim 5.

7. The operating device is capable of operation in both the positive and negative directions relative to the operating shaft. The training system according to claim 1.

8. The aforementioned operating unit is a lever that is movable with respect to the left-right operating axis and the front-back operating axis, The actuator moves the lever with respect to at least one of the left-right operating axis and the front-rear operating axis, The actuator is driven to reproduce the operation shown in the aforementioned example data. The training system according to claim 1.

9. If the lever is operated in a manner that resists the actuator, the operation of the actuator based on the model data is stopped. The training system according to claim 8.

10. If the difference between the operating position of the lever positioned by the actuator and the position of the lever operated by the operator exceeds a predetermined threshold, it is determined that an operation resisting the actuator has been performed. The training system according to claim 9.

11. The aforementioned model data includes operation data entered in the past. The training system according to claim 1.

12. The aforementioned model data includes operational data obtained by calculation. The training system according to claim 1.

13. The actuator is driven based on the acceleration component of the operating device shown in the aforementioned example data. The training system according to claim 11 or claim 12.

14. The actuator is controlled to vibrate the operating part at the timing when the operating direction of the operating device changes based on the acceleration component of the operating device shown in the aforementioned model data. The training system according to claim 13.

15. An operating device comprising an operating part that is movable relative to an operating shaft, and a sensor that detects the amount of operation of the operating part relative to the operating shaft, An actuator that acts on the operator when the operator operates the operating device, Display device and A control method for a training system for operating a work machine located in a virtual space, comprising: Computers The actuator is driven based on the model data. The change in the posture of the work machine is simulated according to the amount of operation detected by the aforementioned sensor. The first moving image representing the simulated change in the posture of the work machine is displayed on the display device. A method for controlling a training system.

Images