Reproduction system, reproduction method, display control system, and display control method

Abstract

This reproduction system comprises: an acquisition unit that acquires log information about a work machine associated with time; an image generation unit that generates an operation image representing the operation of the work machine based on the log information by sequentially applying, to a three-dimensional model of the work machine, operation information representing the operation of the work machine; and a display control unit that displays the operation image on a prescribed display unit. The three-dimensional model includes an operation unit model representing an operation unit provided in a cab of the work machine. The image generation unit generates the operation image by setting the three-dimensional model to a viewpoint facing the operation unit in the cab.

Description

Reproduction system, reproduction method, display control system, and display control method

[0001] This application claims priority to Japanese Patent Application Nos. 2023-203133, 2023-203150, and 2023-203165, filed on November 30, 2023, the contents of which are incorporated herein by reference.

[0002] Patent Document 1 describes a playback device that makes it possible to grasp the movements of two different construction machines (hereinafter referred to as work machines). The playback device described in Patent Document 1 plays back the operation of the work machine by sequentially applying work machine angle information contained in the work machine's log information to a 3D (three-dimensional) model of the work machine. The playback device described in Patent Document 1 synchronizes playback of a 3D model based on first log information with playback of a 3D model based on second log information. The playback device described in Patent Document 1 also uses a two-dimensional operation panel model that represents changes in operation amount as orthogonal movements in the up / down and left / right directions to play back animations of input operations made by the operator of the work machine to various operation levers and travel levers.

[0003] Patent Document 1 describes a playback device that makes it possible to grasp the movements of two different construction machines (hereinafter referred to as work machines). The playback device described in Patent Document 1 plays back the operation of the work machine by sequentially applying angle information of the work machine contained in log information of the work machine to a 3D (three-dimensional) model of the work machine. In the playback device described in Patent Document 1, playback of a 3D model based on first log information and playback of a 3D model based on second log information are performed in synchronization.

[0004] Patent Document 1 describes a playback device that makes it possible to grasp the movements of two different construction machines (hereinafter referred to as work machines). The playback device described in Patent Document 1 plays back the operation of the work machine by sequentially applying angle information of the work machine contained in log information of the work machine to a 3D (three-dimensional) model of the work machine. In the playback device described in Patent Document 1, playback of a 3D model based on first log information and playback of a 3D model based on second log information are performed in synchronization.

[0005] Japanese Patent Application Laid-Open No. 2020-183615

[0006] As described above, the playback device described in Patent Document 1 uses a two-dimensional operation panel model to play back input operations to the control levers and travel levers as animation. However, there is room for improvement in terms of playing back the movements of the work machine in a way that makes them easier to understand.

[0007] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a playback system and playback method that can easily grasp and play back the movement of a work machine.

[0008] Incidentally, when analyzing the work of a work machine using a 3D model, there is a need to be able to confirm, for example, the loading condition of the excavation target, such as soil and sand, inside a work tool such as a bucket used to excavate the target, and the surrounding conditions, such as the terrain on which the work machine is located.

[0009] The present disclosure has been made in consideration of the above circumstances, and has an object to provide a playback system and a playback method that are capable of analyzing work performed by a work machine in detail.

[0010] In analyzing work performed by a work machine, there is a need to analyze the flow of a series of operations, such as excavation, turning from the excavation position to the loading position, loading, and turning from the loading position to the excavation position. The playback device described in Patent Document 1 makes it possible to grasp this series of movements by, for example, repeatedly playing them back as an animation using a 3D model. However, there is room for improvement in order to more effectively analyze work performed by a work machine.

[0011] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a display control system and a display control method that can more effectively analyze work performed by a work machine.

[0012] The playback system of the present disclosure includes an acquisition unit that acquires log information of a work machine associated with a time, an image generation unit that generates operation images that represent the operation of the work machine by sequentially applying operation information that represents the operation of the work machine based on the log information to a three-dimensional model of the work machine, and a display control unit that displays the operation images on a specified display unit, wherein the three-dimensional model includes an operation unit model that represents an operation unit provided in a cab of the work machine, and the image generation unit generates the operation images by setting the three-dimensional model at a viewpoint facing the operation unit in the cab.

[0013] The playback method disclosed herein includes the steps of acquiring log information of a work machine associated with a time, generating a motion image representing the motion of the work machine by sequentially applying motion information representing the motion of the work machine based on the log information to a three-dimensional model of the work machine, and displaying the motion image on a specified display unit, wherein the three-dimensional model includes an operation unit model representing an operation unit provided in a cab of the work machine, and when generating the motion image, the three-dimensional model is set to a viewpoint facing the operation unit in the cab to generate the motion image.

[0014] The playback system of the present disclosure includes an acquisition unit that acquires work machine log information associated with a time and video information representing a captured video image taken by the work machine and associated with a time, and a playback unit that sequentially plays back motion images representing the motion of the work machine by sequentially applying motion information representing the motion of the work machine based on the log information to a three-dimensional model of the work machine, and sequentially plays back captured images based on the video information in synchronization with the motion images.

[0015] The playback method disclosed herein includes the steps of acquiring log information of a work machine associated with a time and video information representing a captured video image taken by the work machine and associated with a time, sequentially applying motion information representing the motion of the work machine based on the log information to a three-dimensional model of the work machine to sequentially play back motion images representing the motion of the work machine, and sequentially playing back captured images based on the video information in synchronization with the motion images.

[0016] The display control system of the present disclosure includes an acquisition unit that acquires work machine log information associated with time, an image generation unit that sequentially generates operation images that represent the operation of the work machine by sequentially applying operation information that represents the operation of the work machine based on the log information to a three-dimensional model of the work machine, and sequentially generates movement trajectory images that represent the movement trajectory of a specified location of the work machine in association with the operation images, based on the operation information, and a display control unit that superimposes the operation images and the movement trajectory images and displays them on a specified display unit.

[0017] The display control method disclosed herein includes the steps of acquiring log information of a work machine associated with time, sequentially applying operation information representing the operation of the work machine based on the log information to a three-dimensional model of the work machine to sequentially generate operation images representing the operation of the work machine, and sequentially generating movement trajectory images representing the movement trajectory of a predetermined location of the work machine in association with the operation images based on the operation information, and displaying the operation images and the movement trajectory images superimposed on each other on a predetermined display unit.

[0018] The playback system and playback method of the present disclosure make it possible to play back the movement of a work machine in a way that makes it easier to understand. The display control device and display control method of the present disclosure make it possible to analyze work performed by a work machine in more detail. The display control system and display control method of the present disclosure make it possible to analyze work performed by a work machine more effectively.

[0019] FIG. 1 is a diagram showing the overall configuration of an analysis support system according to a first embodiment. FIG. 2 is a diagram showing the structure of a work machine according to the first embodiment. FIG. 3 is a diagram showing the functional configuration of a playback system according to the first embodiment. FIG. 4 is a diagram showing the processing flow of the playback system according to the first embodiment. FIG. 5 is a first diagram showing example of log information according to the first embodiment. FIG. 6 is a second diagram showing example of log information according to the first embodiment. FIG. 7 is a third diagram showing example of log information according to the first embodiment. FIG. 8 is a diagram showing a heat map used for estimating work content according to the first embodiment. FIG. 9 is a diagram showing an example of a work machine model according to the first embodiment. FIG. 10 is a diagram showing an example of the relationship between an operation unit 3D model and a projection surface according to the first embodiment. FIG. 11 is a diagram showing an example of the relationship between the display surface of a display unit and a projection surface according to the first embodiment. FIG. 12 is a diagram showing an example of the relationship between the display surface of a display unit and a projection surface according to the first embodiment. FIG. 13 is a diagram showing an example of a display image according to the first embodiment. FIG. 14 is a diagram showing an example of a display image according to the first embodiment. FIG. 15 is a diagram showing an example of a display image according to the first embodiment. FIG. 16 is a diagram showing an example of a display image according to the first embodiment. FIG. 17 is a diagram showing an example of a display image according to the first embodiment. FIG. 1 is a diagram showing the functional configuration of a playback system according to a second embodiment. FIG. 2 is a diagram showing a processing flow of the playback system according to the second embodiment. FIG. 3 is a diagram showing an example of log information according to the second embodiment. FIG. 4 is a diagram showing a heat map used for estimating work content according to the second embodiment. FIG. 5 is a diagram showing an example of a work machine model according to the second embodiment. FIG. 6 is a diagram showing an example of a display image according to the second embodiment. FIG. 7 is a diagram showing an example of a display image according to the second embodiment. FIG. 8 is a diagram showing an example of a display image according to the second embodiment. FIG. 9 is a diagram showing the functional configuration of a playback system according to a third embodiment. FIG. 10 is a diagram showing the processing flow of the playback system according to the third embodiment. FIG. 11 is a diagram showing an example of a display image according to the third embodiment. FIG. 12 is a diagram showing the overall configuration of an analysis support system according to a fourth embodiment.FIG. 10 is a diagram showing the structure of a work machine according to a fourth embodiment. FIG. 11 is a diagram showing the configuration of a cab of a work machine according to the fourth embodiment. FIG. 12 is a diagram showing the functional configuration of a playback system according to the fourth embodiment. FIG. 13 is a diagram showing the processing flow of the playback system according to the fourth embodiment. FIG. 14 is a first diagram showing an example of log information according to the fourth embodiment. FIG. 15 is a second diagram showing an example of log information according to the fourth embodiment. FIG. 16 is a third diagram showing an example of log information according to the fourth embodiment. FIG. 17 is a diagram showing a heat map used for estimating work content according to the fourth embodiment. FIG. 18 is a diagram showing an example of a work machine model according to the fourth embodiment. FIG. 19 is a diagram showing an example of a display image according to the fourth embodiment. FIG. 19 is a diagram showing an example of a display image according to the fourth embodiment.

[0020] First Embodiment A display control device and a display control method according to a first embodiment will be described in detail below with reference to FIGS.

[0021] (Overall Configuration of Analysis Support System) Fig. 1 is a diagram showing the overall configuration of an analysis support system according to Embodiment 1. The analysis support system 11 has a playback system 110 and a data logger 120 mounted on each of a plurality of work machines 13.

[0022] The work machine 3 is the subject of work analysis by the playback system 110. Examples of the work machine 13 include hydraulic excavators, wheel loaders, and bulldozers. In the following description, a hydraulic excavator will be used as an example of the work machine 13. Each work machine 13 is equipped with multiple sensors. The data logger 120 chronologically records and accumulates information indicating the status of the work machine 13 obtained by the sensors. Hereinafter, information recorded by the data logger 120 indicating the status of the work machine 13 at each time will also be referred to as log information. In addition, if the operation mechanism that operates the work machine 13 is configured to operate the work machine 13 using electrical operation signals, information on the operation signals of the work machine 3 may be recorded and accumulated chronologically and included in the log information. In addition, the data logger 120 transmits the recorded log information to the playback system 110 via a wide area communication network at regular time intervals. In addition, the regular time intervals may be, for example, every five minutes. The playback system 110 records the log information received from the data logger 120 on a recording medium. The function of the playback system 110 will be described later.

[0023] (Structure of Work Machine) Figure 2 is a diagram showing the structure of a work machine according to the first embodiment. The work machine 13, which is a hydraulic excavator, excavates earth and sand and levels the ground at work sites and the like. As shown in Figure 2, the work machine 13, which is a hydraulic excavator, comprises a lower running body 131 for traveling, and a rotatable upper rotating body 132 that is installed above the lower running body 131. The upper rotating body 132 is also provided with a driver's cab 132A, a work implement 132B, and two GNSS (Global Navigation Satellite System) antennas 1G1 and 1G2.

[0024] The undercarriage 131 has a left crawler track 1CL and a right crawler track 1CR. The work machine 13 moves forward, turns, and reverses by rotation of the left crawler track 1CL and the right crawler track 1CR.

[0025] The operator's cab 132A is a place where an operator of the work machine 13 gets in and operates the work machine 13. The operator's cab 132A is installed, for example, on the left side of the front end of the upper rotating body 132. The internal configuration of the operator's cab 132A will be described later.

[0026] The work implement 132B consists of a boom 1BM, an arm 1AR, and a bucket 1BK. The boom 1BM is attached to the front end of the upper rotating body 132. An arm 1AR is attached to the boom 1BM. A bucket 1BK is attached to the arm 1AR. A boom cylinder 1SL1 is attached between the upper rotating body 132 and the boom 1BM. By driving the boom cylinder 1SL1, the boom 1BM can be moved relative to the upper rotating body 132. An arm cylinder 1SL2 is attached between the boom 1BM and the arm 1AR. By driving the arm cylinder 1SL2, the arm 1AR can be moved relative to the boom 1BM. A bucket cylinder 1SL3 is attached between the arm 1AR and the bucket 1BK. By driving the bucket cylinder 1SL3, the bucket 1BK can be moved relative to the arm 1AR. The above-described upper rotating body 132, boom 1BM, arm 1AR, and bucket 1BK provided on the work machine 13, which is a hydraulic excavator, are one aspect of the movable parts of the work machine 13. The bucket 1BK is also one example of a configuration of a "work tool" according to the present disclosure. The bucket 1BK is provided with a cutting edge 1BKT used for excavation, etc.

[0027] (Configuration of the Operator's Cabin) FIG. 3 is a diagram showing the configuration of the operator's cab 132A of the work machine according to the first embodiment.

[0028] As shown in Figure 3, the operator's cab 132A is provided with operation levers 1L1 and 1L2, foot pedals 1F1 and 1F2, and travel levers 1R1 and 1R2. The operation levers 1L1 and 1L2, foot pedals 1F1 and 1F2, and travel levers 1R1 and 1R2 are included in an operation unit 132AO for operating movable parts of the work machine 13. The operation levers 1L1 and 1L2 are located on the left and right sides of the seat 1ST inside the operator's cab 132A. The foot pedals 1F1 and 1F2 are located on the floor inside the operator's cab 132A, in front of the seat 1ST.

[0029] An example of an operation pattern showing the correspondence between input operations on the operation levers 1L1, 1L2 and travel levers 1R1, 1R2 and the operation of the work machine 13, which is a hydraulic excavator, is as follows.

[0030] The operation lever 1L1, located on the left side as viewed from the front of the cab, is an operation mechanism for performing the swing operation of the upper rotating body 132 and the excavation / dumping operation of the arm 1AR. Specifically, when the operator of the work machine 13 tilts the operation lever 1L1 forward, the arm 1AR performs a dumping operation. When the operator of the work machine 13 tilts the operation lever 1L1 rearward, the arm 1AR performs an excavation operation. When the operator of the work machine 13 tilts the operation lever 1L1 to the right, the upper rotating body 132 swings right. When the operator of the work machine 13 tilts the operation lever 1L1 to the left, the upper rotating body 132 swings left. When the operation lever 1L1 is tilted forward or backward, the upper rotating body 132 swings right or left, and when the operation lever 1L1 is tilted left or right, the arm 1AR performs a dumping operation or an excavation operation.

[0031] The control lever 1L2, located on the right side as viewed from the front of the cab, is an operating mechanism for performing the excavation / dump operation of the bucket 1BK and the raising / lowering operation of the boom 1BM. Specifically, when the operator of the work machine 13 tilts the control lever 1L2 forward, the boom 1BM is lowered. When the operator of the work machine 13 tilts the control lever 1L2 rearward, the boom 1BM is raised. When the operator of the work machine 13 tilts the control lever 1L2 to the right, the bucket 1BK is dumped. When the operator of the work machine 13 tilts the control lever 1L2 to the left, the bucket 1BK is excavated.

[0032] Additionally, the travel levers 1R1 and 1R2 are operating mechanisms for controlling the operation of the undercarriage 131, i.e., for controlling the travel of the work machine 13. The travel lever 1R1, located on the left side as one faces the front of the cab, corresponds to the rotational drive of the left crawler 1CL of the undercarriage 131. Specifically, when the operator of the work machine 13 tilts the travel lever 1R1 forward, the left crawler 1CL rotates in the forward direction. On the other hand, when the operator of the work machine 13 tilts the travel lever 1R1 rearward, the left crawler 1CL rotates in the reverse direction.

[0033] The travel lever 1R2, located on the right side as viewed from the front of the cab, corresponds to the rotational drive of the right crawler 1CR of the undercarriage 131. Specifically, when the operator of the work machine 13 tilts the travel lever 1R2 forward, the right crawler 1CR rotates in the forward direction. Conversely, when the operator of the work machine 13 tilts the travel lever 1R2 rearward, the right crawler 1CR rotates in the reverse direction. The foot pedals 1F1 and 1F2 are linked to the travel levers 1R1 and 1R2, respectively, and travel can also be controlled by the foot pedals 1F1 and 1F2.

[0034] The above-described operation pattern is merely an example, and is not limited to the above-described pattern depending on the model of the hydraulic excavator, etc.

[0035] Depending on the embodiment, the work machine 3 described using FIG. 2 may not be equipped with the GNSS antennas 1G1 and 1G2.

[0036] (Functional Configuration of Playback System) Fig. 4 is a diagram showing the functional configuration of the playback system according to the first embodiment. Hereinafter, functions of the playback system 110 according to the first embodiment will be described with reference to Fig. 4. As shown in Fig. 4, the playback system 110 includes a CPU 1100, a memory 1101, a display unit 1102, an operation reception unit 1103, a communication interface 1104, and storage 1105. Note that the CPU (Central Processing Unit) 1100 may be a processor such as an FPGA or a GPU instead of a CPU.

[0037] The CPU 1100 is a processor that controls the overall operation of the playback system 110. The various functions of the CPU 1100 will be described later.

[0038] The memory 1101 is a so-called main storage device, and stores instructions and data necessary for the CPU 1100 to operate based on a program.

[0039] The display unit 1102 is a display device capable of visually displaying information, and is, for example, a liquid crystal display or an organic EL display.

[0040] The operation reception unit 1103 is an input device, such as a general mouse, keyboard, or touch sensor.

[0041] The communication interface 1104 is a communication interface for communicating with the data logger 120 .

[0042] The storage 1105 is a so-called auxiliary storage device, such as a hard disk drive (HDD) or solid state drive (SSD). The storage 1105 stores log information 1TL received from the data logger 120, the vehicle type of the work machine 13, and a work machine model 1TM, which is a 3D model prepared in advance for each model. The work machine model 1TM will be described later. The storage 1105 also stores a unit task prediction model 1PM1 and an element task prediction model 1PM2, which are trained machine learning models used when estimating the work content of the work machine 13, heat maps (1H1, 1H2) generated during the estimation process, the estimated work content R of the work machine 13, and three-dimensional point cloud information 1MAP representing topographical information of the work site. The unit task prediction model 1PM1, element task prediction model 1PM2, and heat maps (1H1, 1H2) will be described later. The three-dimensional point cloud information 1MAP includes, for example, three-dimensional point cloud information measured at the work site using a drone, etc., as well as information on topography that is continuously updated from the cutting edge coordinate information of the work machine 13 in operation, and is recorded in storage 1105.

[0043] The functions of the CPU 1100 of the playback system 110 according to the first embodiment will be described in detail. The CPU 1100 operates based on a predetermined program to function as an acquisition unit 11000, a reception unit 11001, an extraction unit 11002, an image generation unit 11003, an estimation unit 11004, a determination unit 11005, a synchronization unit 11006, and a display control unit 11007. The predetermined program may be for implementing some of the functions of the playback system 110. For example, the program may be combined with another program already stored in the storage 1105 or with another program implemented in another device to perform the functions. In another embodiment, the playback system 110 may include a custom large-scale integrated circuit (LSI) such as a programmable logic device (PLD) in addition to or instead of the above configuration. Examples of PLDs include programmable array logic (PAL), generic array logic (GAL), complex programmable logic device (CPLD), and field programmable gate array (FPGA). In this case, some or all of the functions realized by a processor may be realized by the integrated circuit.

[0044] The acquisition unit 11000 acquires the log information 1TL to be played back from among the multiple log information 1TL recorded and accumulated in the storage 1105. Here, it is assumed that the multiple log information 1TL are recorded in the storage 1105 as files recorded with different file names. The acquisition unit 11000, for example, acquires one log information 1TL to be analyzed or acquires two log information 1TL to be compared, in accordance with instructions from an operator. Below, an example is described in which the acquisition unit 11000 acquires two log information 1TL to be compared. One of the two log information acquired by the acquisition unit 11000 is the log information designated by the operator as the target for playback, for example, by a file name. Hereinafter, this log information will be referred to as designated log information 1TL1 (first log information). The other of the two log information is the log information to be played back simultaneously with playback based on the designated log information 1TL1. Hereinafter, this log information will be referred to as comparison log information 1TL2 (second log information). The comparison log information 1TL2 may be log information selected in advance as a "model" for the work content for each of various work contents and conditions, for example, from the viewpoint of good fuel economy, short work time, driving by an experienced operator, etc. Alternatively, the comparison log information 1TL2 may be, for example, past log information recorded for a previous operation performed by the same operator as the designated log information 1TL1. In this case, by comparing the two log information, it is possible to analyze, for example, the operator's proficiency.

[0045] The acquisition unit 11000 according to this embodiment automatically searches for and acquires comparison log information 1TL2 appropriate as a comparison target for the specified log information 1TL1, based on the determination result by the determination unit 11005 (described below). However, the acquisition unit 11000 according to other embodiments is not limited to the above-described aspect. For example, the acquisition unit 11000 may acquire log information specified by an operator using a file name or the like as the comparison log information 1TL2. Furthermore, if only one piece of comparison log information 1TL2 is recorded, the acquisition unit 11000 may acquire that one piece of comparison log information 1TL2.

[0046] The reception unit 11001 receives a predetermined regeneration instruction from the operator of the regeneration system 110. For example, the reception unit 11001 receives a regeneration instruction for the work machine 13 from the operator of the regeneration system 110.

[0047] The extraction unit 11002 extracts, from the acquired log information 1TL, angle information to be used for playback of the work machine 13. The angle information is an example of "operation information (motion information)" according to the present disclosure.

[0048] The image generation unit 11003 generates operational images that represent the operation of the work machine 13 by sequentially applying operation information that represents the operation of the work machine 13 based on the log information 1TL to a work machine model 1TM, which is a three-dimensional model of the work machine 13. As will be described later, in this embodiment, the work machine model 1TM includes an operation unit 3D model 1M1 (an example of an operation unit model in the present disclosure) that represents an operation unit 132AO provided in the cab 132A of the work machine 13. The image generation unit 11003 also generates operational images by setting the viewpoint 1PT when projecting the work machine model 1TM onto a two-dimensional projection surface to a position facing the operation unit 132AO within the cab 132A. The image generation unit 11003 generates operational images by setting the viewpoint 1PT of the work machine model 1TM to a position facing the operation unit 132AO within the cab 132A.

[0049] The estimation unit 11004 estimates the work content of the work machine 13 at each time from the acquired log information 1TL.

[0050] The determining unit 11005 determines whether or not the comparison log information 1TL2 is appropriate as a comparison target for the specified log information 1TL1, based on the information included in the specified log information 1TL1.

[0051] The synchronization unit 11006 performs processing to synchronize the playback of the work machine model 1TM based on the specified log information 1TL1 with the playback of the work machine model 1TM based on the comparison log information 1TL2. Specifically, the synchronization unit 11006 specifies the playback start time on the timeline of the animation of the work machine model 1TM based on the work content estimated by the estimation unit 11004.

[0052] The display control unit 11007 causes the display unit 1102 to display the operation image generated by the image generation unit 11003. The display control unit 11007 may further display, on the display unit 1102, a work content image showing a time series of estimated work content. The display control unit 11007 may further display, on the display unit 1102, an operation amount image showing a time series of values ​​corresponding to the operation amount of the operation unit 132AO included in the log information 1TL. The display control unit 11007 may further display, on the display unit 1102, an instantaneous fuel consumption image showing a time series of instantaneous fuel consumption included in the log information 1TL. The display control unit 11007 may further display, on the display unit 1102, an image showing the topography (terrain image) based on the three-dimensional point cloud information 1MAP, for example, by combining it with the operation image and displaying it on the display unit 1102.

[0053] (Processing Flow of the Reproduction System) Hereinafter, the specific processing flow performed by the reproduction system 110 will be described in detail with reference to FIGS.

[0054] 5 starts when a dedicated application is started by the operator of the playback system 110. When the dedicated application is started by the operator's operation, the acquisition unit 11000 of the CPU 1100 expands and acquires the designated log information 1TL1 designated as the target for playback in the memory 1101 (step 1S00).

[0055] Here, the log information 1TL (designated log information 1TL1, comparison log information 1TL2) will be described with reference to FIGS.

[0056] As shown in Figures 6 to 8, the log information 1TL includes work machine identification information. Specifically, the work machine identification information is an individual identification number for individually identifying the work machine 13. In Figures 6 to 8, the work machine identification information is allocated to correspond to the vehicle type, model, type, and serial number of the work machine 13, indicating a hydraulic excavator, wheel loader, bulldozer, etc. Note that the work machine identification information may be numbers, letters, symbols, or a combination of these, in addition to numbers.

[0057] As shown in Figure 6, the log information 1TL includes information indicating the position and attitude of the work machine 13 at each time, and angle information of the movable parts of the work machine 13. Specifically, the log information 1TL records the position of the work machine 13, the roll angle of the work machine 13, which is the left-right tilt of the machine body, the pitch angle, which is the fore-and-aft tilt of the machine body, the slewing angle, the boom angle, the arm angle, and the bucket angle for each time. Here, the data logger 120 mounted on the work machine 13 identifies and records the position of the work machine 13 based on positioning information indicating latitude, longitude, and altitude, which is information obtained by receiving from the GNSS antennas 1G1, 1G2, for example. The data logger 120 also calculates and records the roll angle and pitch angle of the work machine 13 based on measurement results from an IMU (Inertial Measurement Unit) mounted on the work machine 13. The data logger 120 also calculates and records the rotation angle of the upper rotating body 132 based on positioning information obtained from each of the GNSS antennas 1G1 and 1G2 provided on the upper rotating body 132. Furthermore, the data logger 120 calculates and records the boom angle, arm angle, and bucket angle based on the extension and retraction degrees of the boom cylinder 1SL1, arm cylinder 1SL2, and bucket cylinder 1SL3. Note that the boom angle, arm angle, bucket angle, and rotation angle may be acquired, for example, by attaching IMUs to the boom, arm, bucket, and upper rotating body and using these IMUs.

[0058] The position, roll angle, and pitch angle are information necessary to identify the position and attitude of the work machine 13 itself. Therefore, for example, in an embodiment in which only the movements of the movable parts of the work machine 13, i.e., the upper rotating body 132, boom 1BM, arm 1AR, and bucket 1BK, are played back as animation, and the position and attitude of the work machine 13 itself are not reproduced, information on the position, roll angle, and pitch angle does not need to be included in the log information.

[0059] Log information 1TL shown in Figure 6 corresponds to the "operation information" according to the present disclosure. Operation information is, for example, information that represents the operation of movable parts of the work machine 13, such as the work implement 132B, upper rotating body 132, and lower traveling body 131, which are operated by the operation unit 132AO of the work machine 13. Alternatively, log information TL shown in Figure 6 is information that represents the operation of the work machine 13 that can be seen from outside the work machine 13, and can also be referred to as external operation information. That is, in this embodiment, operation information or external operation information includes position information, roll angle, pitch angle, swing angle, boom angle, arm angle, and bucket angle of the work machine 13. In this embodiment, operation information (or external operation information) differs from drive mechanism information and operation information, which will be described later, in that it is less susceptible to being affected by differences in the class of the work machine 13 as information used when estimating work content.

[0060] As shown in FIG. 7 , the log information 1TL also includes the degree of input by the operator to the control levers 1L1, 1L2, etc. at each time, i.e., the degree of lever tilt and the degree of pedal depression, as indicated by the pilot oil pressure (PPC (Proportional Pressure Control) pressure). Specifically, the log information 1TL records, for each time, the PPC pressure of the control levers 1L1, 1L2, travel levers 1R1, 1R2, or foot pedals 1F1, 1F2, which correspond to each of the operator's operations: left / right swing, arm digging / dumping, boom raising / lowering, bucket digging / dumping, right track forward / reverse, and left track forward / reverse. Note that the times shown in FIG. 7 correspond to the times shown in FIG. 6 . In this embodiment, the information shown in FIG. 7 is also referred to as operation information.

[0061] As shown in FIG. 8 , the log information 1TL includes information indicating the status of the main drive mechanisms of the work machine 13, such as the engine and hydraulic pump, at each time. Specifically, the log information 1TL records the engine coolant temperature, engine output, instantaneous fuel consumption, and hydraulic pump oil temperature for each time. The items shown in FIG. 8 are merely examples, and, for example, items such as the engine coolant temperature and hydraulic pump oil temperature may be omitted, or other items that affect fuel economy, such as the fuel throttle opening, may be included. As another example, in an electric construction machine, items such as instantaneous power consumption and remaining battery capacity may be included. The times shown in FIG. 8 correspond to the times in FIGS. 6 and 7 . In this embodiment, the information shown in FIG. 8 is also referred to as drive mechanism information.

[0062] Returning to FIG. 5, the estimation unit 11004 of the CPU 1100 estimates the work content of the work machine 13 at each time based on the designated log information 1TL1 acquired in step 1S00 (step 1S01).

[0063] Here, the procedure by which the estimation unit 11004 estimates the work content of the work machine 13 from the log information 1TL will be described with reference to Figure 9. The estimation unit 11004 estimates the work content of the work machine 13 for both unit tasks and element tasks. A unit task is a task that accomplishes one work purpose. An element task is an element that makes up a unit task and represents a series of actions or tasks categorized by purpose.

[0064] Examples of unit task classifications include "excavation and loading," "plowing," "slope (from below)," "load collection," "travel," and "parking / restoring" as shown in FIG. 9 , as well as "ditch digging," "backfilling," and "slope (from above)." Excavation and loading involves digging and scraping away soil or rocks, and loading the scraped soil or rocks onto the bed of a transport vehicle. Excavation and loading is a unit task consisting of excavation, loading and unloading, unloading, unloading and waiting for unloading, and bed holding. Plowing is the task of scraping away excess ground irregularities to a predetermined height. Plowing is a unit task consisting of excavation and unloading, or excavation, loading and unloading, unloading and unloading, and may include leveling and sweeping. Slope (from below) is the task of creating a slope using a work machine 13 located below the target area. Compacting a slope (from below) is a unit operation consisting of rolling, excavation, loaded rotation, soil removal, and empty rotation, and may include pushing and leveling. Load collection is the operation of collecting soil and sand excavated by excavation, etc. before loading it onto a transport vehicle. Load collection is a unit operation consisting of excavation, loaded rotation, soil removal, and empty rotation, and may include pushing and leveling. Traveling is the operation of moving the work machine 13. Traveling as a unit operation is a unit operation consisting of traveling as an element operation. Stopping / idling is a state in which the bucket 1BK is free of soil and rocks and is stopped for a predetermined period of time or more. Stopping / idling as a unit operation is a unit operation consisting of stopping as an element operation. Trench excavation is the operation of digging a long, narrow trench in the ground and scraping it away. Trench excavation is a unit operation consisting of excavation, loaded rotation, soil removal, and empty rotation, and may include pushing and leveling. Backfilling is the process of filling existing trenches or holes in the ground with soil and sand to fill them flat. Backfilling is a unit operation consisting of excavation, loading and turning, soil removal, compaction, and empty turning, and may also include leveling and brooming. Slope (from above) is the process of creating a slope using a work machine 13 positioned above the target area. Slope (from above) is a unit operation consisting of compaction, excavation, loading and turning, soil removal, and empty turning, and may also include leveling.

[0065] Examples of classifications of elemental work include "excavation," "loaded rotation," "waiting for soil discharge," "soil discharge," "empty load rotation," and "load carrier holddown" shown in FIG. 9 , as well as "rolling," "leveling," and "brooming." Excavation is the work of digging up and scraping away soil or rocks with the bucket 1BK. Loaded rotation is the work of rotating the upper rotating body 132 while the bucket 1BK holds the scraped soil or rocks. Waiting for soil discharge is the work of waiting for a transport vehicle to load the scraped soil or rocks while the bucket 1BK holds the scraped soil or rocks. Soil discharge is the work of lowering the scraped soil or rocks from the bucket 1BK onto a transport vehicle or a predetermined location. Empty load rotation is the work of rotating the upper rotating body 132 when the bucket 1BK is empty of soil or rocks. Load carrier holddown is the work of pressing down and leveling the soil loaded on the load carrier of the transport vehicle with the bucket 1BK from above. Compaction is the process of pushing soil and sand into disturbed ground with the bucket 1BK to shape and strengthen the ground. Leveling is the process of sweeping and leveling the soil and sand with the bottom of the bucket 1BK. Brooming is the process of sweeping and leveling the soil and sand with the side of the bucket 1BK.

[0066] The estimation unit 11004 inputs the log information 1TL to the unit task prediction model 1PM1 in chronological order to obtain a time series of likelihoods related to the unit task. The unit task prediction model 1PM1 is a model that outputs likelihoods related to the unit task when the log information 1TL is input, for example, by learning using training data, and may be stored in the storage 1105, for example.

[0067] The estimation unit 11004 also obtains a time series of likelihoods related to element works by inputting the log information 1TL to an element work prediction model 1PM2 in chronological order. The element work prediction model 1PM2 is a model that outputs likelihoods related to element works when the log information 1TL is input, for example, by learning using teacher data, and may be stored in the storage 1105, for example.

[0068] The estimation unit 11004 smooths the time series of likelihoods for unit tasks and element tasks by applying a time averaging filter to each of them, and generates a unit task heat map 1H1 representing the time series of likelihoods for the smoothed unit tasks and an element task heat map 1H2 representing the time series of likelihoods for the smoothed element tasks, as shown in FIG. 9 . The heat maps 1H1 and 1H2 are maps in which, based on the time series of smoothed likelihoods, the task task divisions are plotted on the vertical axis and the time on the horizontal axis, and colors representing the likelihoods of the task divisions are applied. For example, the colors of the heat maps may be closer to blue as the likelihood of the task divisions decreases, and closer to red as the likelihood of the task divisions increases. The estimation unit 11004 stores the heat maps 1H1 and 1H2 in the storage 1105.

[0069] The estimation unit 11004 identifies a time period during which the likelihood of a unit task is dominant, based on the time series of the smoothed likelihood, and estimates the work content of the work machine 13 for that time period. For example, in a time period during which the likelihood of the unit task "digging and loading" is dominant, the estimation unit 11004 estimates the work content of the work machine 13 to be "digging and loading". Similarly, the estimation unit 11004 identifies a time period during which the likelihood of an element task is dominant, based on the time series of the smoothed likelihood, and estimates the work content of the work machine 13 for that time period. For example, in a time period during which the likelihood of the element task "excavation" is dominant, the estimation unit 11004 estimates the work content of the work machine 13 to be "excavation". The estimation unit 11004 stores information on the work content 1R estimated for the work machine 13 in the storage 1105.

[0070] 5, the acquisition unit 11000 then determines whether or not to display information (comparison log information 1TL2) to be compared with the designated log information 1TL1 designated as the target for playback, based on a predetermined instruction operation from the operator (step 1S02A). If the information is to be displayed (step 1S02A: YES), the acquisition unit 11000 executes the processing of step 1S02B and step 1S03, and then proceeds to step 1S04. If the information is not to be displayed (step 1S02A: NO), the acquisition unit 11000 proceeds to step 1S04 without executing step 1S02B and step 1S03. If the information is to be displayed (step 1S02A: YES), the acquisition unit 11000 acquires comparison log information 1TL2 appropriate as a comparison target for the designated log information 1TL1 acquired in step 1S00 (step 1S02B). At this time, the determination unit 11005 performs a determination process flow for determining whether the plurality of pieces of comparison log information 1TL2 recorded in the storage 1105 are suitable as comparison targets for the designated log information 1TL1 acquired in step 1S00. In this determination process flow, the determination unit 11005 first selects one piece of comparison log information 1TL2 from the plurality of pieces of comparison log information 1TL2 previously recorded in the storage 1105. Then, based on the log information 1TL, the determination unit 11005 determines whether, for example, the vehicle size, the amount of change in the swing angle (swing magnitude), the bucket height of the work machine 13 while waiting to unload, etc. are similar. If it determines that they are not similar, the determination unit 11005 selects one piece of new comparison log information 1TL2 and performs the same determination process as above. Furthermore, the determination unit 11005 according to other embodiments may narrow down playback candidates based on the work position, work time, work category, etc., in addition to the above determination process.

[0071] 5, next, the synchronization unit 11006 performs processing to synchronize the animation playback of the work machine model 1TM based on the designated log information 1TL1 with the animation playback of the work machine model 1TM based on the comparison log information 1TL2 (step 1S03). Specifically, the synchronization unit 11006 specifies the playback start time on a timeline in the animation of the work machine model 1TM based on the designated log information 1TL1 and the playback start time on a timeline in the animation of the work machine model 1TM based on the comparison log information 1TL2.

[0072] The method for specifying the playback start time for synchronizing the two is as follows. That is, the synchronization unit 11006 extracts the timing of unit task switching in the specified log information 1TL1 based on the unit task heat map 1H1 for the specified log information 1TL1 generated by the estimation unit 11004. Next, the synchronization unit 11006 extracts the timing of unit task switching in the comparison log information 1TL2 based on the unit task heat map 1H1 for the comparison log information 1TL2 generated by the estimation unit 11004. The synchronization unit 11006 selects one of the extracted timings and specifies it as the playback start time for the respective animation. Note that the playback system 110 according to other embodiments is not limited to the above aspect. A playback system 110 according to another embodiment may, for example, compare a combination of angle information of the boom, arm, bucket, etc. in the comparison log information 1TL2 with a combination of angle information of the boom, arm, bucket, etc. in the specified log information 1TL1, extract the timing at which the two angles are close to each other, and identify this as the playback start time for each animation.

[0073] After step 1S03, or if step 1S02A is "NO", the reception unit 11001 receives a regeneration instruction from the operator (step 1S04). One form of the regeneration instruction may be an operation such as pressing a regeneration button. The regeneration instruction may also include information that will be the start point of regeneration, such as the time, the position of the work machine 13, or various events such as the occurrence of an abnormality in the work machine 13.

[0074] Next, the acquisition unit 11000 selects and reads out from the storage 1105 the work machine model 1TM corresponding to the referenced work machine identification information based on the work machine identification information as the type of work machine 13 received by the reception unit 11001 (step 1S05).

[0075] The work machine model 1TM will now be described with reference to Figure 10. As shown in Figure 10, the work machine model 1TM is information that includes work machine identification information, an exterior 3D model 1M0 of the work machine 13 indicated in the work machine identification information, and an operation unit 3D model 1M1 that is a three-dimensional model of the operation unit 132AO. The exterior 3D model 1M0 is a 3D model that represents the work machine 13, and is constructed for each part of the work machine 13, such as the undercarriage and upper rotating body. For example, the exterior 3D model 1M0 represents the shape of the work machine 13. For example, the exterior 3D model 1M0 is composed of a lower running body exterior model 1M01 representing the lower running body 131 of the work machine 13, an upper rotating body exterior model 1M02 representing the upper rotating body 132, a boom exterior model 1M03 representing the boom 1BM, an arm exterior model 1M04 representing the arm 1AR, and a bucket exterior model 1M05 representing the bucket 1BK.

[0076] The operation unit 3D model 1M1 is a 3D model representing the operation unit 132AO, and represents the cab 132A equipped with the operation unit 132AO, the seat 1ST in the cab 132A, and a three-dimensional shape representing the operation unit 132AO. Among these, models 1M11, 1M12, 1M13, 1M14, etc. representing the operation levers 1L1, 1L2, driving levers 1R1, 1R2, etc. included in the operation unit 132AO are models whose inclination changes based on the viewpoint 1PT or a line or plane passing through the viewpoint 1PT when projecting the three-dimensional model onto a two-dimensional projection surface 1PS according to the log information 1TL. The inclination of the cab 132A, seat 1ST, etc. does not change with respect to the viewpoint 1PT or a line or plane passing through the viewpoint 1PT. In FIG. 10, the projection surface 1PS is a two-dimensional plane when projecting a three-dimensional shape onto a two-dimensional plane. The viewpoint 1PT is a position from which the three-dimensional shape projected onto the projection surface 1PS is viewed. In this embodiment, the viewpoint 1PT is set at a position facing the operation unit 132AO in the driver's cab 132A. The viewpoint 1PT can be set, for example, at a position corresponding to the eye position of the operator when sitting in the seat 1ST. The vector 1PTD represents the reference direction when viewing the projection surface 1PS from the viewpoint 1PT, for example, the forward direction.

[0077] Returning to FIG. 5 , the extraction unit 11002 extracts information to be used for playback from each of the designated log information 1TL1 and the comparison log information 1TL2, or from the designated log information 1TL1 (step 1S06). For example, the extraction unit 11002 extracts various angle information, such as the boom angle, arm angle, and bucket angle, as well as the PPC pressures of the control levers 1L1 and 1L2, the travel levers 1R1 and 1R2, and the foot pedals 1F1 and 1F2, as information to be used for playback. Alternatively, in steps 1S00 and 1S02B, only the information to be used for playback may be acquired. The following description of the processing flow takes as an example a case where the designated log information 1TL1 and the comparison log information 1TL2 are compared and played back. If playback using the comparison log information 1TL2 is not performed, this can be achieved by essentially omitting the processing related to the comparison log information 1TL2.

[0078] Next, the image generation unit 11003 and the display control unit 11007 simultaneously play back an animation of the work machine model 1TM based on the designated log information 1TL1 and an animation of the work machine model 1TM based on the comparison log information 1TL2 (step 1S07). Here, the image generation unit 11003 generates a motion image that represents the operation of the work machine 13 while applying the various information recorded in the designated log information 1TL1 to the work machine model 1TM in chronological order of the timestamps from the playback start time specified in the synchronization processing of step 1S03 (step 1S07a). Furthermore, simultaneously with generating the motion images for this animation, the image generation unit 11003 generates a motion image that represents the operation of the work machine 13 while applying the various information recorded in the comparison log information 1TL2 to the work machine model 1TM in chronological order of the timestamps from the playback start time specified in the synchronization processing of step 1S03 (step 1S07a). In addition, in step 1S07a, the display control unit 11007 causes the display unit 1102 to display the operation image generated by the image generation unit 11003.

[0079] The operation image generation process performed by the image generation unit 11003 in step 1S07a will be described in detail below. The image generation unit 11003 changes the angle of the corresponding portion of the exterior 3D model 1M0 based on various angle information such as the swing angle and boom angle indicated in the log information 1TL (designated log information 1TL1, comparative log information 1TL2). For example, the image generation unit 11003 recreates the position and posture of the bucket 1BK of the work machine 13 by tilting the bucket exterior model 1M05 around the rotation axis defined by the connection position with the arm exterior model 1M04 so that the bucket angle is indicated in the log information 1TL.

[0080] Similarly, the image generation unit 11003 reproduces the position and posture of the arm 1AR of the work machine 13 by tilting the arm exterior model 1M04 around the rotation axis defined at the connection position with the boom exterior model 1M03 so that the arm angle is the angle indicated in the log information 1TL.

[0081] Similarly, the image generation unit 11003 reproduces the position and posture of the upper rotating body 132 of the work machine 13 by tilting the upper rotating body exterior model 1M02 around the rotation axis defined at the connection position with the lower running body exterior model 1M01 to the rotation angle indicated in the log information TL.

[0082] Similarly, the image generation unit 11003 reproduces the posture of the upper rotating body 132 of the work machine 13 by tilting the lower running body exterior model 1M01 around the roll rotation axis defined in the lower running body exterior model 1M01 to the roll angle indicated in the log information 1TL, and tilting it around the pitch rotation axis defined in the lower running body exterior model 1M01 to the pitch angle indicated in the log information 1TL.

[0083] Furthermore, the playback system 110 according to the first embodiment can play back animation of the running of the work machine 13 based on the PPC pressures for the right track forward / reverse and the left track forward / reverse at each time, which are included in the log information 1TL.

[0084] Specifically, the exterior 3D model 1M0 moves forward, backward, left / right forward, and left / right backward based on the PPC pressures for the right track forward / reverse and the left track forward / reverse. For example, the exterior 3D model 1M0 moves forward based on the PPC pressure values ​​for the right track forward and the left track forward. The speed of movement may be changed based on the PPC pressure values.

[0085] Furthermore, the exterior 3D model 1M0 is moved backward based on the numerical values ​​of the PPC pressure for the right track reverse and the left track reverse. Furthermore, the exterior 3D model 1M0 is moved so as to curve forward in the left and right directions based on the difference between the numerical values ​​of the PPC pressure for the right track forward and the left track forward. For example, if the numerical value of the PPC pressure for the right track forward is greater than the numerical value of the PPC pressure for the left track forward, the exterior 3D model 1M0 is moved so as to curve forward in the left direction. The speed of movement and the magnitude of the curve may be changed depending on the numerical values ​​of the PPC pressure for each track and the difference between the numerical values ​​of the PPC pressure.

[0086] Similarly, the exterior 3D model 1M0 is moved so as to curve backward in the left and right directions based on the difference between the PPC pressure values ​​for the right and left track retractions. For example, if the PPC pressure value for the right track retraction is greater than the PPC pressure value for the left track retraction, the exterior 3D model 1M0 is moved so as to curve backward in the left direction. The speed of movement and the magnitude of the curve may be changed depending on the PPC pressure values ​​and the difference between the PPC pressure values.

[0087] Note that by using position information in addition to the PPC pressures for right track forward / reverse and left track forward / reverse, it is possible to more accurately animate the traveling of the work machine 3. In this case, by using position information, it is possible to more accurately represent the speed and position of the movement of the work machine 3. Furthermore, by playing back animation of the work machine 13 based on the roll angle, or the pitch angle, or both the roll angle and the pitch angle, in addition to the PPC pressures for right track forward / reverse and left track forward / reverse, it is possible to reproduce the left-right tilt of the work machine 13 or the front-to-back tilt of the work machine 3 while traveling.

[0088] Furthermore, the image generation unit 11003 changes the angle of the corresponding portion of the exterior 3D model 1M0 based on the PPC pressure indicated in the log information 1TL (designated log information 1TL1, comparative log information 1TL2). For example, the image generation unit 11003 recreates the position and posture of the bucket 1BK of the work machine 13 by tilting the bucket exterior model 1M05 around the rotation axis defined at the connection position with the arm exterior model 1M04 so that the bucket angle is the one indicated in the log information 1TL.

[0089] Similarly, the image generation unit 11003 reproduces the position and posture of the arm 1AR of the work machine 13 by tilting the arm exterior model 1M04 around the rotation axis defined at the connection position with the boom exterior model 1M03 so that the arm angle is the angle indicated in the log information 1TL.

[0090] Similarly, the image generation unit 11003 reproduces the position and posture of the upper rotating body 132 of the work machine 13 by tilting the upper rotating body exterior model 1M02 around the rotation axis defined at the connection position with the lower running body exterior model 1M01 to the rotation angle indicated in the log information 1TL.

[0091] Similarly, the image generation unit 11003 reproduces the posture of the upper rotating body 132 of the work machine 13 by tilting the lower running body exterior model 1M01 around the roll rotation axis defined in the lower running body exterior model 1M01 to the roll angle indicated in the log information 1TL, and tilting it around the pitch rotation axis defined in the lower running body exterior model 1M01 to the pitch angle indicated in the log information 1TL.

[0092] Similarly, the image generation unit 11003 changes the inclination of a model 1M11 representing the operating lever 1L1, a model 1M12 representing the operating lever 1L2, a model 1M13 representing the driving lever 1R1, and a model 1M14 representing the driving lever 1R2 based on the PPC pressure.

[0093] 11 , the image generation unit 11003 may rotate the angle of the projection surface 1PS in the direction of arrow 1RA based on the roll angle, and change the position (height) of the projection surface 1PS based on the pitch angle. That is, for example, as shown in FIG. 12 , the angular relationship with the projection surface 1PS may be changed based on the roll angle, such as to surface 1PS1, surface 1PS2, etc., using an area 1DS on the display screen of the display unit 1102 as a reference. Alternatively, for example, as shown in FIG. 13 , the position (height) relationship with the projection surface 1PS may be changed based on the pitch angle, such as to surface 1PS3, surface 1PS4, etc., using an area 1DS on the display screen of the display unit 1102 as a reference. In this way, by changing the angle and position of the projection surface 1PS based on the roll angle and pitch angle, it is possible to reflect changes in the roll angle and pitch angle, which are changes in the attitude of the work machine 13 viewed from the outside, in a two-dimensional image generated using a viewpoint 1PT inside the driver's seat 132A of the work machine 13. In this case, for example, in response to operation of the operation unit 132AO, the occurrence of vibrations or the like that change the roll angle or pitch angle of the work machine 13 can be reflected as changes in, for example, the operational images representing the models 1M11-1M14, the terrain images, etc. Note that the image generation unit 11003 may be able to select, for example, whether the roll angle and pitch angle used when changing the angle or position of the projection surface 1PS should be values ​​based on the specified log information 1TL1 or values ​​based on the comparison log information 1TL2 in accordance with a predetermined operation by the operator during image playback. With this configuration, for example, the extent of occurrence of vibrations or the like can be compared by playing back images while switching between the selections.

[0094] Also, in step 1S07, the display control unit 11007 combines a topographical image based on the three-dimensional point cloud information 1MAP with the action image and displays it on the display unit 1102 (step 1S07b).

[0095] In addition, in step 1S07, the display control unit 11007 may display a supplemental image showing the work content estimation result and the time series of various information included in the log information 1TL on the display unit 1102 (step 1S07c).

[0096] The image generation unit 11003 determines whether to end animation playback while playing back the animation of the operations of the two work machines 13 (step 1S08). For example, the image generation unit 11003 determines to end animation playback when it receives an instruction to end playback based on pressing a stop button, etc. It may also determine to end animation playback after a predetermined period of time has elapsed since animation playback began. If animation playback has not ended (step 1S08; NO), the image generation unit 11003 continues simultaneous animation playback of the two work machine models 1TM. On the other hand, if animation playback is to end (step 1S08; YES), the image generation unit 11003 ends the animation playback process.

[0097] Of the processing flows described using FIG. 5, steps 1S00, 1S01, 1S03, 1S04, 1S05, 1S06, and 1S08 are not essential components of the playback system 110, and other embodiments may not include such steps.

[0098] (Display Images of the Playback System) FIGS. 14 to 18 show examples of display images of the display unit 1102 according to the first embodiment. FIGS. 14 to 17 show examples of display images based on the designation log information 1TL1 and the comparison log information 1TL2 of the playback system according to the first embodiment. FIG. 18 shows an example of a display image based on the designation log information 1TL1 of the playback system according to the first embodiment. Display image 1D2 shown in FIG. 15 includes action image 1D21, which is a state in which a certain time has passed since the action image 1D11 included in display image 1D1 shown in FIG. 14. Action image 1D11 and action image 1D21 include images representing models 1M11, 1M12, and 1M03, etc., and are also combined with a terrain image 1D13. In the example of operation shown in FIG. 5, action image 1D11 and action image 1D21 are displayed in step 1S7a, and terrain image 1D13 is displayed in step 1S7b. Additionally, the operation image 1D11 and the operation image 1D21 display buttons 1B1 and 1B2 for selecting the reference log information 1TL. FIG. 14 shows a state in which button 1B1 is selected. In this case, the roll angle and pitch angle are set based on the specified log information 1TL1, and an image is generated. FIG. 15 shows a state in which button 1B2 is selected. In this case, the roll angle and pitch angle are set based on the comparison log information 1TL2, and an image is generated. In the examples shown in FIGS. 14 and 15 , changes in the roll angle and pitch angle are represented, for example, as changes in the inclination and position of the terrain image D13. By switching the selection state of buttons 1B1 and 1B2, it is possible to compare the state when the specified log information 1TL1 was recorded with the state when the comparison log information 1TL2 was recorded. Note that the image indicated by the solid line is an image based on the specified log information 1TL1. The image indicated by the dashed line is an image based on the comparison log information 1TL2.

[0099] Furthermore, display images 1D1 and 1D2 include supplemental images 1D12 and 1D22 generated in step 1S7c. Fig. 16 shows supplemental image 1D12 shown in Fig. 14. As shown in Fig. 16, supplemental image 1D12 includes a work content image 1D121 representing the timeline of the work content estimated based on designated log information 1TL1, a work content image 1D122 representing the timeline of the work content estimated based on comparison log information 1TL2, a playback time icon 1D129, and a scroll bar 1D120.

[0100] The supplemental image 1D12 also includes an operation amount image 1D123 and an operation amount image 1D124 that represent a time series of values ​​corresponding to the operation amount of the operation unit 132AO included in the log information 1TL. Note that the solid line represents the operation amount based on the specified log information 1TL1, and the dashed line represents the operation amount based on the comparison log information 1TL2. The operation amount image 1D123 represents the boom raising lever pressure. The operation amount image 1D124 represents the arm excavation lever pressure.

[0101] FIG. 17 shows an example of a supplemental image 1D12 when the scroll bar 1D120 shown in FIG. 16 is scrolled downward. The supplemental image 1D12 shown in FIG. 18 includes operation amount images 1D125, 1D126, and 1D128, and an instantaneous fuel consumption image 1D127 showing a time series of instantaneous fuel consumption included in log information 1TL. The operation amount image 1D125 represents the arm excavation lever pressure. The operation amount image 1D126 represents the boom raising lever pressure. The operation amount image 1D128 represents the bucket dump lever pressure. Note that the content of the time series information included in the supplemental image 1D12 shown in FIGS. 16 and 17 is merely an example and is not limited to this example. For example, some of the items shown in FIGS. 16 and 17 may be omitted.

[0102] Moreover, the display image 1D1A shown in FIG. 18 includes an action image 1D11 and a supplemental image 1D12 based on the designated log information 1TL1.

[0103] (Actions and Effects) As described above, the playback system 110 according to the first embodiment comprises an acquisition unit 11000 that acquires log information 1TL of the work machine 13 associated with time; an image generation unit 11003 that generates operation images that represent the operation of the work machine 13 by sequentially applying operation information that represents the operation of the work machine 13 based on the log information 1TL to a work machine model 1TM, which is a three-dimensional model of the work machine 13; and a display control unit 11007 that displays the operation images on the display unit 1102. The work machine model 1TM includes an operation unit 3D model 1M1 that represents the operation unit 132AO provided in the cab 132A of the work machine 13. The image generation unit 11003 also generates operation images by setting the work machine model 1TM to a viewpoint facing the operation unit 132AO within the cab 132A. With this configuration, the movement of the operation unit 132AO can be played back based on, for example, the operator's viewpoint. Therefore, according to this embodiment, the movement of the work machine 13 can be played back in a way that makes it easy to understand.

[0104] Furthermore, in the playback system 110 according to the first embodiment, the operation images include images representing the operation unit 132AO, making it possible to easily analyze and compare, for example, lever operations. Furthermore, the playback system 110 can display, on the display unit 1102, supplemental images, including, along with the operation images, task content images representing a time series of task content, operation amount images representing a time series of operation amounts of the operation unit, and instantaneous fuel consumption images representing a time series of instantaneous fuel consumption. This configuration allows for a more detailed understanding of, for example, the temporal changes in the operation of the operation unit 132AO for each task item, compared to a case where supplemental images are not provided. Examples of the temporal changes include, for example, whether the lever operation includes not only fully open or fully closed but also intermediate operation amounts for fine adjustment, and the flow in which each lever is operated for each operation item.

[0105] (Modification) The contents of the log information 1TL (FIGS. 6 to 8) according to the first embodiment are not limited to those in other embodiments. For example, if the work machine 13 is not a hydraulic excavator but a different vehicle type, log information 1TL according to that vehicle type is recorded. Examples of other vehicle types include a wheel loader, a bulldozer, etc.

[0106] Furthermore, the regeneration system 110 according to the first embodiment has been described as being installed at a location away from the work machine 13 and connected to the data logger 120 mounted on the work machine 13 via a wide area communication network, but other embodiments are not limited to this configuration.

[0107] For example, in a replay system 110 according to another embodiment, part or all of the configuration of the replay system 110 may be installed inside the work machine 13. In this case, the data logger 120 may transmit the log information 1TL to the replay system 110 via an internal network of the work machine 13, rather than via a wide area communication network. In this way, the operator on board the work machine 13 can check the movements of the work machine 13 being operated by the operator on the spot by playing back an animation. Furthermore, by playing back model movements of the work machine 13 for the operator of the work machine 13, this can be used as guidance.

[0108] Note that the playback system 110 installed inside the work machine 13 may acquire the log information 1TL of other work machines 13 via a wide area communication network, etc. In this way, it is possible to play back animations of the states of work machines 13 other than the work machine 13 on which the playback system 110 is installed.

[0109] Furthermore, the playback system 110 according to another embodiment may be installed in a location away from the work machine 13, and may transmit and display the video information generated by the animation playback process on a monitor mounted on the work machine 13.

[0110] Furthermore, in other embodiments, one aspect of the playback instruction received from the operator may be, for example, a playback period. For example, the playback period may be a playback start time and a playback end time. In this case, the playback system 110 performs playback of the work machine 13 for the received playback period. Furthermore, in other embodiments, it is not essential to specify a playback end time. For example, in other embodiments, a playback instruction may be received from the operator specifying only a playback start time, and playback may be performed for a certain period of time from the playback start time, or playback may continue as long as log information exists, or playback may be stopped in response to the occurrence of various other events.

[0111] The acquired log information 1TL (FIGS. 6 to 8) does not need to be arranged in chronological order. In this case, the image generation unit 11003 only needs to apply the information to be used for playback from the log information 1TL to the work machine model 1TM in chronological order.

[0112] The various processing steps of the playback system 110 described above are stored in the form of a program on a computer-readable recording medium, and the computer reads and executes this program to perform the various processing steps. Computer-readable recording media include magnetic disks, magneto-optical disks, CD-ROMs, DVD-ROMs, and semiconductor memories. Alternatively, the computer program may be distributed to a computer via a communication line, and the computer that receives the program may execute the program.

[0113] The program may be one that realizes part of the above-mentioned functions, or may be one that realizes the above-mentioned functions in combination with a program already recorded in the computer system, such as a so-called differential file or differential program.

[0114] Some or all of the functions of the playback system 110 described above may be assigned to the work machine 13. For example, some or all of the functions of the acquisition unit 11000, reception unit 11001, extraction unit 11002, image generation unit 11003, estimation unit 11004, determination unit 11005, synchronization unit 11006, display control unit 11007, memory 1101, display unit 1102, operation reception unit 1103, communication interface 1104, and storage 1105 may be assigned to the work machine 13.

[0115] Although several embodiments of the present disclosure have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the inventions described in the claims and their equivalents, as well as in the scope and spirit of the inventions.

[0116] (Additional Note) The playback system 110 according to the present disclosure can be understood, for example, as follows.

[0117] (1) A playback system 110 according to a first aspect comprises an acquisition unit 11000 that acquires log information 1TL of a work machine 13 associated with time, an image generation unit 11003 that generates operation images that represent the operation of the work machine by sequentially applying operation information that represents the operation of the work machine based on the log information to a three-dimensional model of the work machine (work machine model 1TM), and a display control unit 11007 that displays the operation images on a predetermined display unit 1102, the three-dimensional model including an operation unit model (operation unit 3D model 1M1) that represents an operation unit 132AO provided in a cab 132A of the work machine, and the image generation unit generates the operation images by setting a viewpoint 1PT when projecting the three-dimensional model onto a two-dimensional projection surface 1PS to a position facing the operation unit within the cab. According to this aspect and each of the following aspects, the movement of the work machine 13 can be played back in a way that makes it easy to understand.

[0118] (2) A playback system 110 according to a second aspect is the playback system of (1), in which the operation image includes an image representing the operation unit.

[0119] (3) A playback system 110 according to a third aspect is the playback system of (1) or (2), and further includes an estimation unit 11004 that estimates the work content of the work machine at each time based on the log information, and the display control unit further displays a work content image 1D121 on the display unit, the work content image 1D121 representing the time series of the estimated work content.

[0120] (4) A playback system 110 according to a fourth aspect is a playback system according to any one of (1) to (3), wherein the display control unit further displays, on the display unit, an operation amount image representing a time series of values ​​corresponding to the operation amount of the operation unit contained in the log information.

[0121] (5) The playback system 110 according to the fifth aspect is a playback system according to any one of (1) to (4), wherein the display control unit further displays an instantaneous fuel consumption image on the display unit, the instantaneous fuel consumption image representing a time series of the instantaneous fuel consumption included in the log information.

[0122] (6) A playback system 110 according to a sixth aspect is a playback system according to any one of (1) to (5), wherein the image generation unit rotates the projection surface based on the roll angle of the work machine and changes the height of the projection surface based on the pitch angle of the work machine.

[0123] Second Embodiment A playback system and a playback method according to a second embodiment will be described in detail below with reference to FIGS.

[0124] (Overall configuration of analysis support system) Fig. 19 is a diagram showing the overall configuration of an analysis support system according to embodiment 2. The analysis support system 21 has a playback system 210, and a data logger 220 and an imaging device 2CAM mounted on each of a plurality of work machines 23.

[0125] The work machine 23 is the subject of work analysis by the playback system 210. Examples of the work machine 23 include hydraulic excavators, wheel loaders, and bulldozers. In the following description, a hydraulic excavator will be used as an example of the work machine 23. Each work machine 23 is equipped with multiple sensors. The data logger 220 chronologically records and accumulates information obtained by the sensors indicating the status of the work machine 23. Hereinafter, information recorded by the data logger 220 indicating the status of the work machine 23 at each time will also be referred to as log information. This log information is associated with time. In addition, if the operating mechanism that operates the work machine 23 is configured to operate the work machine 23 using electrical operating signals, information on the operating signals of the work machine 23 may be recorded and accumulated chronologically and included in the log information. In addition, the data logger 220 transmits the recorded log information to the playback system 210 via a wide area communication network at regular time intervals. In addition, the regular time intervals may be, for example, every five minutes. The playback system 210 records the log information received from the data logger 220 on a recording medium. The imaging device 2CAM captures moving images of the outside, for example, in the forward direction of the work machine 23. The imaging device 2CAM may store moving image information representing the captured moving images in an internal or external storage device of the imaging device 2CAM. The moving image information is associated with time information (time). The imaging device 2CAM also transmits the captured moving image information to the playback system 210 via a wide area communication network at regular time intervals, or distributes it to the playback system 210 in approximately real time. The playback system 210 records the moving image information received from the imaging device CAM on a recording medium or plays it back in approximately real time. The following describes an example in which the playback system 210 temporarily stores the moving image information. The functions of the playback system 210 will be described later.

[0126] (Structure of Work Machine) Figure 20 is a diagram showing the structure of a work machine according to the second embodiment. The work machine 23, which is a hydraulic excavator, excavates earth and sand and levels the ground at work sites and the like. As shown in Figure 20, the work machine 23, which is a hydraulic excavator, comprises a lower running body 231 for traveling, and a rotatable upper rotating body 232 that is installed above the lower running body 231. In addition, the upper rotating body 232 is provided with a driver's cab 232A, a work implement 232B, and two GNSS (Global Navigation Satellite System) antennas 2G1 and 2G2.

[0127] The undercarriage 231 has a left crawler track 2CL and a right crawler track 2CR. The work machine 23 moves forward, turns, and reverses by rotation of the left crawler track 2CL and the right crawler track 2CR.

[0128] The cab 232A is where the operator of the work machine 23 gets in and operates the work machine 23. The cab 232A is installed, for example, on the left side of the front end of the upper rotating body 232. An imaging device 2CAM that captures moving images in the forward direction of the work machine 23 is also attached inside the cab 232A. However, the attachment position of the imaging device 2CAM to the work machine 23 and the number of devices are not limited to this example. The configuration of the operation section inside the cab 232A will be described later.

[0129] The work implement 232B consists of a boom 2BM, an arm 2AR, and a bucket 2BK. The boom 2BM is attached to the front end of the upper rotating body 232. An arm 2AR is attached to the boom 2BM. A bucket 2BK is attached to the arm 2AR. A boom cylinder 2SL1 is attached between the upper rotating body 232 and the boom 2BM. By driving the boom cylinder 2SL1, the boom 2BM can be moved relative to the upper rotating body 232. An arm cylinder 2SL2 is attached between the boom 2BM and the arm 2AR. By driving the arm cylinder 2SL2, the arm 2AR can be moved relative to the boom 2BM. A bucket cylinder 2SL3 is attached between the arm 2AR and the bucket 2BK. By driving the bucket cylinder 2SL3, the bucket 2BK can be moved relative to the arm 2AR. The above-described upper rotating body 232, boom 2BM, arm 2AR, and bucket 2BK provided on the work machine 23, which is a hydraulic excavator, are one aspect of the movable parts of the work machine 23. The bucket 2BK is also one example of a configuration of a "work tool" according to the present disclosure. The bucket 2BK is provided with a cutting edge 2BKT used for excavation, etc.

[0130] (Configuration of the Operator's Cabin) FIG. 21 is a diagram showing the configuration of an operator's cab 232A of a work machine according to the second embodiment.

[0131] As shown in Figure 21, the operator's cab 232A is provided with operation levers 2L1 and 2L2, foot pedals 2F1 and 2F2, and travel levers 2R1 and 2R2. The operation levers 2L1 and 2L2, foot pedals 2F1 and 2F2, and travel levers 2R1 and 2R2 are included in an operation unit 232AO for operating movable parts of the work machine 23. The operation levers 2L1 and 2L2 are located on the left and right sides of the seat 2ST inside the operator's cab 232A. The foot pedals 2F1 and 2F2 are located on the floor inside the operator's cab 232A, in front of the seat 2ST.

[0132] An example of an operation pattern showing the correspondence between input operations on the control levers 2L1, 2L2 and travel levers 2R1, 2R2 and the operation of the work machine 23, which is a hydraulic excavator, is as follows.

[0133] The operation lever 2L1, located on the left side facing the front of the cab, is an operation mechanism for performing the swing operation of the upper rotating body 232 and the excavation / dumping operation of the arm 2AR. Specifically, when the operator of the work machine 23 tilts the operation lever 2L1 forward, the arm 2AR performs a dumping operation. When the operator of the work machine 23 tilts the operation lever 2L1 rearward, the arm 2AR performs an excavation operation. When the operator of the work machine 23 tilts the operation lever 2L1 to the right, the upper rotating body 232 swings to the right. When the operator of the work machine 23 tilts the operation lever 2L1 to the left, the upper rotating body 232 swings to the left. When the operation lever 2L1 is tilted forward or backward, the upper rotating body 232 swings to the right or left, and when the operation lever 2L1 is tilted left or right, the arm 2AR performs a dumping operation or an excavation operation.

[0134] The control lever 2L2, located on the right side as viewed from the front of the cab, is an operating mechanism for performing the excavation / dump operation of the bucket 2BK and the raising / lowering operation of the boom 2BM. Specifically, when the operator of the work machine 23 tilts the control lever 2L2 forward, the boom 2BM is lowered. When the operator of the work machine 23 tilts the control lever 2L2 rearward, the boom 2BM is raised. When the operator of the work machine 23 tilts the control lever 2L2 to the right, the bucket 2BK is dumped. When the operator of the work machine 23 tilts the control lever 2L2 to the left, the bucket 2BK is excavated.

[0135] Additionally, the travel levers 2R1 and 2R2 are operating mechanisms for controlling the operation of the undercarriage 231, i.e., for controlling the travel of the work machine 23. The travel lever 2R1, located on the left side as one faces the front of the cab, corresponds to the rotational drive of the left crawler 2CL of the undercarriage 231. Specifically, when the operator of the work machine 23 tilts the travel lever 2R1 forward, the left crawler 2CL rotates in the forward direction. On the other hand, when the operator of the work machine 23 tilts the travel lever 2R1 rearward, the left crawler 2CL rotates in the reverse direction.

[0136] The travel lever 2R2, located on the right side as viewed from the front of the cab, corresponds to the rotational drive of the right crawler 2CR of the undercarriage 231. Specifically, when the operator of the work machine 23 tilts the travel lever 2R2 forward, the right crawler 2CR rotates in the forward direction. Conversely, when the operator of the work machine 23 tilts the travel lever 2R2 rearward, the right crawler 2CR rotates in the reverse direction. The foot pedals 2F1 and 2F2 are linked to the travel levers 2R1 and 2R2, respectively, and travel can also be controlled by the foot pedals 2F1 and 2F2.

[0137] The above-described operation pattern is merely an example, and is not limited to the above-described pattern depending on the model of the hydraulic excavator, etc.

[0138] Depending on the embodiment, the work machine 23 described using FIG. 20 may not be equipped with the GNSS antennas 2G1, 2G2.

[0139] (Functional Configuration of Playback System) Fig. 22 is a diagram showing the functional configuration of a playback system according to the second embodiment. Hereinafter, functions of a playback system 210 according to the second embodiment will be described with reference to Fig. 22. As shown in Fig. 22, the playback system 210 includes a CPU 2100, a memory 2101, a display unit 2102, an operation reception unit 2103, a communication interface 2104, and storage 2105. Note that the CPU (Central Processing Unit) 2100 may be a processor such as an FPGA or a GPU instead of a CPU.

[0140] The CPU 2100 is a processor that controls the overall operation of the playback system 210. The various functions of the CPU 2100 will be described later.

[0141] The memory 2101 is a so-called main storage device, and the memory 2101 stores instructions and data necessary for the CPU 2100 to operate based on a program.

[0142] The display unit 2102 is a display device capable of visually displaying information, and is, for example, a liquid crystal display or an organic EL display.

[0143] The operation reception unit 2103 is an input device, such as a general mouse, keyboard, or touch sensor.

[0144] The communication interface 2104 is a communication interface for communicating with the data logger 220 .

[0145] The storage 2105 is a so-called auxiliary storage device, such as a hard disk drive (HDD) or solid state drive (SSD). The storage 2105 stores log information 2TL received from the data logger 220, a work machine model 2TM which is a 3D model prepared in advance for each vehicle type and model of the work machine 23, and video image information 2MV captured by the work machine 23. The work machine model 2TM will be described later. The storage 2105 also stores a unit task prediction model 2PM1 and an element task prediction model 2PM2 which are trained machine learning models used when estimating the work content of the work machine 23, heat maps (2H1, 2H2) generated during the estimation process, estimated work content 2R of the work machine 23, and three-dimensional point cloud information 2MAP which represents topographical information of the work site. The unit task prediction model 2PM1, element task prediction model 2PM2, and heat maps (2H1, 2H2) will be described later. The three-dimensional point cloud information 2MAP includes, for example, three-dimensional point cloud information measured at the work site using a drone, etc., as well as information on topography that is continuously updated from the cutting edge coordinate information of the work machine 23 in operation, and is recorded in storage 2105.

[0146] The functions of the CPU 2100 of the playback system 210 according to the second embodiment will be described in detail. The CPU 2100 operates based on a predetermined program to function as an acquisition unit 21000, a reception unit 21001, an extraction unit 21002, a playback unit 21003, an estimation unit 21004, a determination unit 21005, and a synchronization unit 21006. The predetermined program may be for implementing some of the functions of the playback system 210. For example, the program may be combined with another program already stored in the storage 2105 or with another program implemented in another device to perform the functions. In another embodiment, the playback system 210 may include a custom large-scale integrated circuit (LSI) such as a programmable logic device (PLD) in addition to or instead of the above configuration. Examples of PLDs include programmable array logic (PAL), generic array logic (GAL), complex programmable logic device (CPLD), and field programmable gate array (FPGA). In this case, some or all of the functions realized by a processor may be realized by the integrated circuit.

[0147] The acquisition unit 21000 acquires the log information 2TL to be played back from among the multiple pieces of log information 2TL recorded and accumulated in the storage 2105. The acquisition unit 21000 also acquires video information 2MV corresponding to the log information 2TL to be played back. Here, it is assumed that the multiple pieces of log information 2TL and multiple pieces of video information 2MV are recorded in the storage 2105 as separate files with different file names. For example, the acquisition unit 21000 acquires one piece of log information 2TL and video information 2MV to be analyzed, or acquires two pieces of log information 2TL and two pieces of video information 2MV to be compared, in accordance with instructions from an operator. The following describes an example in which the acquisition unit 21000 acquires two pieces of log information 2TL and two pieces of video information 2MV to be compared. Note that the video information 2MV is acquired in a manner that corresponds to the acquired log information 2TL, and therefore, the video information 2MV acquired when acquiring the log information 2TL will not be described as appropriate. One of the two pieces of log information acquired by the acquisition unit 21000 is the log information designated by the operator as the target for playback, for example, by a file name. Hereinafter, this log information will be referred to as designated log information 2TL1 (first log information). The other of the two pieces of log information is log information that is played back simultaneously with playback based on the designated log information 2TL1. Hereinafter, this log information will be referred to as comparison log information 2TL2 (second log information). The comparison log information 2TL2 may be log information selected in advance as a "model" for various work content and conditions, for example, from the perspective of good fuel economy, short work time, or being driven by an experienced operator. Alternatively, the comparison log information 2TL2 may be, for example, past log information recorded during a previous operation performed by the same operator as the designated log information 2TL1. In this case, by comparing the two pieces of log information, it is possible to analyze, for example, the operator's proficiency.

[0148] The acquisition unit 21000 according to this embodiment automatically searches for and acquires comparison log information 2TL2 appropriate as a comparison target for the specified log information 2TL1, based on the determination result by the determination unit 21005 (described later). However, the acquisition unit 21000 according to other embodiments is not limited to the above-described aspect. For example, the acquisition unit 21000 may acquire log information specified by an operator using a file name or the like as the comparison log information 2TL2. Furthermore, if only one piece of comparison log information 2TL2 is recorded, the acquisition unit 21000 may acquire that one piece of comparison log information 2TL2.

[0149] The reception unit 21001 receives a predetermined regeneration instruction from the operator of the regeneration system 210. For example, the reception unit 21001 receives a regeneration instruction for the work machine 23 from the operator of the regeneration system 210.

[0150] The extraction unit 21002 extracts, from the acquired log information 2TL, angle information to be used for playback of the work machine 23. The angle information is an example of "operation information (motion information)" according to the present disclosure.

[0151] The playback unit 21003 sequentially applies operation information that represents the operation of the work machine 23 based on the log information 2TL to a three-dimensional model of the work machine 23, thereby sequentially playing back operation images that represent the operation of the work machine 23 in a first display area of ​​the display screen of the display unit 2102. In addition to displaying the operation images, the playback unit 21003 also sequentially plays back captured images based on the video information 2MV that corresponds to the log information 2TL in a second display area different from the first display area of ​​the display screen of the display unit 2102, in synchronization with the operation images. Ensuring synchronization between the playback of operation images (playback of the work machine model 2TM) and the playback of captured video of the same work machine 23 can be performed based on the time information included in the log information 2TL and the time information included in the video information 2MV. The playback unit 21003 adjusts the playback timing of the operation images and the captured video so that the recording time corresponding to the operation information included in the log information 2TL and the capture time of each frame included in the video information 2MV approximately coincide.

[0152] In addition, the playback unit 21003 may display the estimation results of the work content by the estimation unit 21004 as supplementary information on the display unit 2102, or may combine an image representing the terrain (terrain image) based on the three-dimensional point cloud information 2MAP with an action image and display it on the display unit 2102.

[0153] The estimation unit 21004 estimates the work content of the work machine 23 at each time from the acquired log information 2TL.

[0154] The determining unit 21005 determines whether or not the comparison log information 2TL2 is appropriate as a comparison target for the specified log information 2TL1, based on the information included in the specified log information 2TL1.

[0155] The synchronization unit 21006 performs processing to synchronize the playback of the work machine model 2TM (playback of operation images) based on the specified log information 2TL1 with the playback of the work machine model 2TM (playback of operation images) based on the comparison log information 2TL2. Specifically, the synchronization unit 21006 specifies the playback start time on the timeline of the animation of the work machine model 2TM based on the work content estimated by the estimation unit 21004.

[0156] (Processing Flow of the Playback System) Hereinafter, the specific processing flow performed by the playback system 210 will be described in detail with reference to FIGS.

[0157] 23 starts when a dedicated application is started by the operator of the playback system 210. When the dedicated application is started by the operator's operation, the acquisition unit 21000 of the CPU 2100 expands and acquires the designated log information 2TL1 designated as the target for playback in the memory 2101 (step 2S00).

[0158] Here, the log information 2TL (designated log information 2TL1, comparison log information 2TL2) will be described with reference to FIGS. 24 to 26. FIG.

[0159] As shown in Figures 24 to 26, the log information 2TL includes work machine identification information. Specifically, the work machine identification information is an individual identification number for individually identifying the work machine 23. In Figures 24 to 26, the work machine identification information is assigned to correspond to the vehicle type, model, type, and serial number of the work machine 23, indicating a hydraulic excavator, wheel loader, bulldozer, etc. Note that the work machine identification information may be numbers, letters, symbols, or a combination of these, in addition to numbers.

[0160] As shown in Figure 24, the log information 2TL includes information indicating the position and attitude of the work machine 23 at each time, and angle information of the movable parts of the work machine 23. Specifically, the log information TL records the position of the work machine 23, the roll angle of the work machine 23, which is the left-right tilt of the machine body, the pitch angle, which is the fore-and-aft tilt of the machine body, the slewing angle, boom angle, arm angle, and bucket angle for each time. Here, the data logger 20 mounted on the work machine 3 identifies and records the position of the work machine 23 based on positioning information indicating latitude, longitude, and altitude, which is information obtained by receiving from the GNSS antennas 2G1, 2G2, for example. In addition, the data logger 220 calculates and records the roll angle and pitch angle of the work machine 23 based on the measurement results of an IMU (Inertial Measurement Unit) mounted on the work machine 23. The data logger 220 also calculates and records the rotation angle of the upper rotating body 232 based on the positioning information obtained from each of the GNSS antennas 2G1 and 2G2 provided on the upper rotating body 232. Furthermore, the data logger 220 calculates and records the boom angle, arm angle, and bucket angle based on the extension and retraction degrees of the boom cylinder 2SL1, arm cylinder 2SL2, and bucket cylinder 2SL3. Note that the boom angle, arm angle, bucket angle, and rotation angle may be acquired, for example, by attaching IMUs to the boom, arm, bucket, and upper rotating body and using these IMUs.

[0161] The position, roll angle, and pitch angle are information necessary to identify the position and attitude of the work machine 23 itself. Therefore, for example, in an embodiment in which only the movements of the movable parts of the work machine 23, i.e., the upper rotating body 232, boom 2BM, arm 2AR, and bucket 2BK, are played back as animation, and the position and attitude of the work machine 23 itself are not reproduced, information on the position, roll angle, and pitch angle does not need to be included in the log information.

[0162] The log information 2TL shown in Figure 24 corresponds to the "operation information" according to the present disclosure. The operation information is, for example, information that represents the operation of movable parts operated by the operation unit 232AO of the work machine 23, such as the work implement 232B, upper rotating body 232, and lower traveling body 231, which are operated by the operation unit of the work machine 23. Alternatively, the log information 2TL shown in Figure 24 is information that represents the operation of the work machine 23 that can be seen from outside the work machine 23, and can also be referred to as external operation information. That is, in this embodiment, the operation information or external operation information includes the position information, roll angle, pitch angle, swing angle, boom angle, arm angle, and bucket angle of the work machine 23. In this embodiment, the operation information (or external operation information) differs from the drive mechanism information and operation information described below in that, as information used when estimating work content, it is less susceptible to the influence of differences in the vehicle class of the work machine 23.

[0163] As shown in FIG. 25 , the log information 2TL also includes the degree of input by the operator to the control levers 2L1, 2L2, etc. at each time, i.e., the degree of tilt of the levers and the degree of pedal depression, as indicated by the pilot oil pressure (PPC (Proportional Pressure Control) pressure). Specifically, the log information 2TL records, for each time, the PPC pressure of the control levers 2L1, 2L2, travel levers 2R1, 2R2, or foot pedals 2F1, 2F2 corresponding to each type of operation performed by the operator: left / right swing, arm digging / dumping, boom raising / lowering, bucket digging / dumping, right track forward / reverse, and left track forward / reverse. Note that the times shown in FIG. 25 correspond to the times shown in FIG. 24 . In this embodiment, the information shown in FIG. 25 is also referred to as operation information.

[0164] As shown in FIG. 26 , the log information TL includes information indicating the status of the main drive mechanisms of the work machine 23, such as the engine and hydraulic pump, at each time. Specifically, the log information 2TL records the engine coolant temperature, engine output, instantaneous fuel consumption, and hydraulic pump oil temperature for each time. The items shown in FIG. 26 are merely examples, and, for example, items such as the engine coolant temperature and hydraulic pump oil temperature may be omitted, or other items that affect fuel economy, such as the fuel throttle opening, may be included. As yet another example, in an electric construction machine, items such as instantaneous power consumption and remaining battery capacity may be included. The times shown in FIG. 26 correspond to the times in FIGS. 24 and 25 . In this embodiment, the information shown in FIG. 26 is also referred to as drive mechanism information.

[0165] Returning to FIG. 23, the estimation unit 21004 of the CPU 2100 estimates the work content of the work machine 23 at each time based on the designated log information 2TL1 acquired in step 2S00 (step 2S01).

[0166] Here, the procedure by which the estimation unit 21004 estimates the work content of the work machine 23 from the log information 2TL will be described with reference to Figure 27. The estimation unit 21004 estimates the work content of the work machine 23 for both unit tasks and element tasks. A unit task is a task that accomplishes one work purpose. An element task is an element that makes up a unit task and represents a series of actions or tasks categorized by purpose.

[0167] Examples of unit task classifications include "digging and loading," "plowing," "slope (from below)," "load collection," "driving," and "parking / restoring" as shown in FIG. 27 , as well as "ditch digging," "backfilling," and "slope (from above)." Excavation and loading involves digging and scraping away soil or rocks, and loading the scraped soil or rocks onto the bed of a transport vehicle. Excavation and loading is a unit task consisting of excavation, loading and unloading, unloading, waiting for unloading, and holding down the bed. Plowing is the task of scraping away excess ground irregularities to a predetermined height. Plowing is a unit task consisting of excavation and unloading, or excavation, loading and unloading, and unloading, and may include leveling and sweeping. Slope (from below) is the task of creating a slope using a work machine 23 located below the target area. Compacting a slope (from below) is a unit operation consisting of rolling, excavation, loaded rotation, soil removal, and empty rotation, and may include pushing and leveling. Load collection is the operation of collecting soil and sand excavated by excavation, etc. before loading it onto a transport vehicle. Load collection is a unit operation consisting of excavation, loaded rotation, soil removal, and empty rotation, and may include pushing and leveling. Traveling is the operation of moving the work machine 23. Traveling as a unit operation is a unit operation consisting of traveling as an element operation. Stopping / idling is a state in which the bucket 2BK is free of soil and rocks and is stopped for a predetermined period of time or more. Stopping / idling as a unit operation is a unit operation consisting of stopping as an element operation. Trench excavation is the operation of digging a long, narrow trench in the ground and scraping it away. Trench excavation is a unit operation consisting of excavation, loaded rotation, soil removal, and empty rotation, and may include pushing and leveling. Backfilling is the process of filling existing trenches or holes in the ground with soil and sand to fill them flat. Backfilling is a unit operation consisting of excavation, loading and turning, soil removal, compaction, and empty turning, and may also include leveling and brooming. Slope (from above) is the process of creating a slope using a work machine 23 positioned above the target area. Slope (from above) is a unit operation consisting of compaction, excavation, loading and turning, soil removal, and empty turning, and may also include leveling.

[0168] Examples of classifications of elemental work include "excavation," "loaded rotation," "waiting for soil discharge," "soil discharge," "empty rotation," and "load carrier holddown" shown in FIG. 27 , as well as "rolling," "leveling," and "brooming." Excavation is the work of digging up and scraping away soil or rocks using the bucket 2BK. Loaded rotation is the work of rotating the upper rotating body 232 while the bucket 2BK holds the scraped soil or rocks. Waiting for soil discharge is the work of waiting for a transport vehicle to load the scraped soil or rocks while the bucket 2BK holds the scraped soil or rocks. Soil discharge is the work of lowering the scraped soil or rocks from the bucket 2BK onto a transport vehicle or a predetermined location. Empty rotation is the work of rotating the upper rotating body 232 when the bucket 2BK is empty of soil or rocks. Load carrier holddown is the work of pressing down and leveling the soil loaded on the load carrier of the transport vehicle with the bucket 2BK from above. Compaction is the work of pushing soil and sand into disturbed ground with the bucket 2BK to shape and strengthen the ground. Leveling is the work of sweeping and leveling the soil and sand with the bottom of the bucket 2BK. Brooming is the work of sweeping and leveling the soil and sand with the side of the bucket 2BK.

[0169] The estimation unit 21004 obtains a time series of likelihoods related to a unit task by inputting the log information 2TL to a unit task prediction model 2PM1 in chronological order. The unit task prediction model 2PM1 is a model that outputs likelihoods related to a unit task when the log information 2TL is input, for example, by learning using training data, and may be stored in the storage 2105, for example.

[0170] The estimation unit 21004 also obtains a time series of likelihoods related to element works by inputting the log information 2TL to the element work prediction model 2PM2 in chronological order. The element work prediction model 2PM2 is a model that outputs likelihoods related to element works when the log information 2TL is input, for example, by learning using teacher data, and may be stored in the storage 2105, for example.

[0171] The estimation unit 21004 smooths the time series of likelihoods for unit tasks and element tasks by applying a time averaging filter to each of them, and generates a unit task heat map 2H1 representing the time series of likelihoods for the smoothed unit tasks and an element task heat map 2H2 representing the time series of likelihoods for the smoothed element tasks, as shown in FIG. 27 . The heat maps 2H1 and 2H2 are maps in which colors representing the likelihoods of task tasks are applied to a plane with task tasks on the vertical axis and time on the horizontal axis, based on the time series of smoothed likelihoods. For example, the colors of the heat maps may be closer to blue as the likelihood of the task task is lower, and closer to red as the likelihood of the task task is higher. The estimation unit 21004 stores the heat maps 2H1 and 2H2 in the storage 2105.

[0172] The estimation unit 21004 identifies a time period during which the likelihood of a unit task is dominant, based on the time series of the smoothed likelihood, and estimates the work content of the work machine 23 for that time period. For example, in a time period during which the likelihood of the unit task "digging and loading" is dominant, the estimation unit 21004 estimates the work content of the work machine 23 to be "digging and loading". Similarly, the estimation unit 21004 identifies a time period during which the likelihood of an element task is dominant, based on the time series of the smoothed likelihood, and estimates the work content of the work machine 23 for that time period. For example, in a time period during which the likelihood of the element task "digging" is dominant, the estimation unit 21004 estimates the work content of the work machine 23 to be "digging". The estimation unit 21004 stores information on the work content R estimated for the work machine 23 in the storage 2105.

[0173] 23 , the acquisition unit 21000 then determines whether or not to display information (comparison log information 2TL2) to be compared with the designated log information 2TL1 designated as the target of playback, based on a predetermined instruction operation from the operator (step 2S02A). If the information is to be displayed (step 2S02A: YES), the acquisition unit 21000 executes the processes of steps 2S02B and 2S03, and then proceeds to step 2S04. If the information is not to be displayed (step 2S02A: NO), the acquisition unit 21000 proceeds to step 2S04 without executing steps 2S02B and 2S03. If the information is to be displayed (step 2S02A: YES), the acquisition unit 21000 acquires comparison log information 2TL2 appropriate as a comparison target for the designated log information 2TL1 acquired in step 2S00 (step 2S02B). At this time, the determination unit 21005 performs a determination process flow for determining whether the plurality of pieces of comparison log information 2TL2 recorded in the storage 2105 are suitable for comparison with the designated log information 2TL1 acquired in step 2S00. In this determination process flow, the determination unit 21005 first selects one piece of comparison log information 2TL2 from the plurality of pieces of comparison log information 2TL2 previously recorded in the storage 2105. Then, based on the log information 2TL, the determination unit 21005 determines whether, for example, the vehicle size, the amount of change in the swing angle (swing magnitude), the bucket height of the work machine 23 while waiting to unload, etc. are similar. If it determines that they are not similar, it selects one new piece of comparison log information 2TL2 and performs the same determination process as above. Furthermore, the determination unit 21005 according to other embodiments may narrow down the candidates for playback based on the work position, work time, work category, etc., in addition to the above determination process.

[0174] 23 , next, the synchronization unit 21006 performs processing to synchronize the animation playback of the work machine model 2TM based on the designated log information 2TL1 with the animation playback of the work machine model 2TM based on the comparison log information 2TL2 (step 2S03). Specifically, the synchronization unit 21006 specifies the playback start time on a timeline in the animation of the work machine model 2TM based on the designated log information 2TL1 and the playback start time on a timeline in the animation of the work machine model 2TM based on the comparison log information 2TL2.

[0175] The method for specifying the playback start time for synchronizing the two is as follows. That is, the synchronization unit 21006 extracts the timing of unit task switching in the specified log information 2TL1 based on the unit task heat map 2H1 for the specified log information 2TL1 generated by the estimation unit 21004. Next, the synchronization unit 21006 extracts the timing of unit task switching in the comparison log information 2TL2 based on the unit task heat map 2H1 for the comparison log information 2TL2 generated by the estimation unit 21004. The synchronization unit 21006 selects one of the extracted timings and specifies it as the playback start time for the respective animation. Note that the playback system 210 according to other embodiments is not limited to the above aspect. A playback system 210 (synchronization unit 21006) according to another embodiment may, for example, compare a combination of angle information of the boom, arm, bucket, etc. in the comparison log information 2TL2 with a combination of angle information of the boom, arm, bucket, etc. in the specified log information 2TL1, extract the timing at which the two angles are close to each other, and identify this as the playback start time for each animation.

[0176] After step 2S03, or if step 2S02A is "NO", the reception unit 21001 receives a playback instruction from the operator (step 2S04). One form of the playback instruction may be an operation such as pressing a playback button. The playback instruction may also include information that will be the start point of playback, such as the time, the position of the work machine 23, or various events such as the occurrence of an abnormality in the work machine 23. When the reception unit 21001 receives the playback instruction, the playback range designation unit 21007 may also designate the work content for which a movement trajectory image is to be generated.

[0177] Next, the acquisition unit 21000 selects and reads out from the storage 2105 the work machine model 2TM corresponding to the referenced work machine identification information based on the work machine identification information as the type of work machine 23 received by the reception unit 21001 (step 2S05).

[0178] The work machine model 2TM will now be described with reference to Figure 28. As shown in Figure 28, the work machine model 2TM is information that includes work machine identification information and an exterior 3D model 2M0 of the work machine 23 indicated in the work machine identification information. The exterior 3D model 2M0 is a 3D model that represents the work machine 23, and is constructed for each part of the work machine 23, such as the lower traveling body and upper rotating body. For example, the exterior 3D model 2M0 represents the shape of the work machine 23. For example, the exterior 3D model 2M0 is made up of a lower traveling body exterior model 2M01 that represents the lower traveling body 231 of the work machine 23, an upper rotating body exterior model 2M02 that represents the upper rotating body 232, a boom exterior model 2M03 that represents the boom 2BM, an arm exterior model 2M04 that represents the arm 2AR, and a bucket exterior model 2M05 that represents the bucket 2BK.

[0179] Returning to FIG. 23 , the extraction unit 21002 extracts information to be used for playback from each of the designated log information 2TL1 and the comparison log information 2TL2, or from the designated log information 2TL1 (step 2S06). For example, the extraction unit 21002 extracts various angle information, such as the boom angle, arm angle, and bucket angle, as information to be used for playback. The pilot oil pressure shown in FIG. 25 may also be extracted as information to be used for playback. Furthermore, in steps 2S00 and 2S02B, only the information to be used for playback may be acquired. The following description of the process flow takes as an example a case where the designated log information 2TL1 and the comparison log information 2TL2 are compared and played back. When playback using the comparison log information 2TL2 is not performed, the process related to the comparison log information 2TL2 can be basically omitted.

[0180] Next, the playback unit 21003 simultaneously plays back animation of the work machine model 2TM based on the designated log information 2TL1 and animation of the work machine model 2TM based on the comparison log information 2TL2, plays back moving images based on the moving image information 2MV corresponding to the designated log information 2TL1, and plays back moving images based on the moving image information 2MV corresponding to the comparison log information 2TL2 (step 2S07). Here, the playback unit 21003 plays back operational images showing the operation of the work machine 23 while applying the various information recorded in the designated log information 2TL1 to the work machine model 2TM in chronological order of oldest timestamp from the playback start time identified in the synchronization processing of step 2S03 (step 2S07a). Furthermore, at the same time as playing back the motion images for this animation, the playback unit 21003 plays back motion images showing the operation of the work machine 23 while applying the various information recorded in the comparison log information 2TL2 to the work machine model 2TM in order of oldest timestamp from the playback start time identified in the synchronization processing of step 2S03 (step 2S07a).

[0181] The following describes in detail the contents of the operation image generation process performed by the playback unit 21003. The playback unit 21003 changes the angle of the corresponding portion of the exterior 3D model 2M0 based on various angle information such as the swing angle and boom angle indicated in the log information 2TL (designated log information 2TL1, comparison log information 2TL2). For example, the playback unit 21003 reproduces the position and posture of the bucket 2BK of the work machine 23 by tilting the bucket exterior model 2M05 around the rotation axis defined at the connection position with the arm exterior model 2M04 so that the bucket angle is indicated in the log information 2TL.

[0182] Similarly, the reproduction unit 21003 reproduces the position and posture of the arm 2AR of the work machine 23 by tilting the arm exterior model 2M04 around the rotation axis defined at the connection position with the boom exterior model 2M03 so that the arm angle is the angle indicated in the log information 2TL.

[0183] Similarly, the reproduction unit 21003 reproduces the position and posture of the upper rotating body 232 of the work machine 23 by tilting the upper rotating body exterior model 2M02 around the rotation axis defined at the connection position with the lower running body exterior model 2M01 to the rotation angle indicated in the log information 2TL.

[0184] Similarly, the reproduction unit 21003 reproduces the posture of the upper rotating body 232 of the work machine 23 by tilting the lower running body exterior model 2M01 around the roll rotation axis defined in the lower running body exterior model 2M01 to the roll angle indicated in the log information 2TL, and tilting it around the pitch rotation axis defined in the lower running body exterior model 2M01 to the pitch angle indicated in the log information 2TL.

[0185] Furthermore, the playback system 210 according to the second embodiment can play back animation of the work machine 23 traveling based on the PPC pressures for the right track forward / reverse and the left track forward / reverse at each time, which are included in the log information 2TL.

[0186] Specifically, the exterior 3D model 2M0 moves forward, backward, left / right forward, and left / right backward based on the PPC pressures for the right track forward / reverse and the left track forward / reverse. For example, the exterior 3D model 2M0 moves forward based on the PPC pressure values ​​for the right track forward and the left track forward. The speed of movement may be changed based on the PPC pressure values.

[0187] Furthermore, the exterior 3D model 2M0 is moved backward based on the numerical values ​​of the PPC pressure for the right track reverse and the left track reverse. Furthermore, the exterior 3D model 2M0 is moved so as to curve forward in the left and right directions based on the difference between the numerical values ​​of the PPC pressure for the right track forward and the left track forward. For example, if the numerical value of the PPC pressure for the right track forward is greater than the numerical value of the PPC pressure for the left track forward, the exterior 3D model 2M0 is moved so as to curve forward in the left direction. The speed of movement and the magnitude of the curve may be changed depending on the numerical values ​​of the PPC pressure for each track and the difference between the numerical values ​​of the PPC pressure.

[0188] Similarly, the exterior 3D model 2M0 is moved so as to curve backward in the left and right directions based on the difference between the PPC pressure values ​​for the right and left track retractions. For example, if the PPC pressure value for the right track retraction is greater than the PPC pressure value for the left track retraction, the exterior 3D model 2M0 is moved so as to curve backward in the left direction. The speed of movement and the magnitude of the curve may be changed depending on the PPC pressure values ​​and the difference between the PPC pressure values.

[0189] Note that by using position information in addition to the PPC pressures for the right track forward / reverse and the left track forward / reverse, it is possible to more accurately animate the travel of the work machine 23. In this case, by using position information, it is possible to more accurately represent the speed and position of the movement of the work machine 23. Furthermore, by playing back animation of the work machine 23 based on the roll angle, or the pitch angle, or both the roll angle and the pitch angle, in addition to the PPC pressures for the right track forward / reverse and the left track forward / reverse, it is possible to reproduce the left-right tilt or the fore-and-aft tilt of the work machine 23 while traveling.

[0190] Also, in step 2S07, the playback unit 21003 acquires video information 2MV corresponding to log information 2TL (specified log information 2TL1, comparison log information 2TL2) using the acquisition unit 21000 in synchronization with the playback of the motion image in step 2S07a, and sequentially plays back the captured images based on the video information 2MV acquired by the acquisition unit 21000 (step 2S07b).

[0191] Also, in step 2S07, the playback unit 21003 combines a terrain image representing the terrain based on the three-dimensional point cloud information 2MAP with the action image and displays it on the display unit 2102 (step 2S07c).

[0192] Furthermore, in step 2S07, the playback unit 21003 causes the display unit 2102 to display supplementary information such as the estimated results of the work content at each time (step 2S07d).

[0193] The playback unit 21003 determines whether to end animation playback while playing back the animation of the operations of the two work machines 23 (step 2S08). For example, the playback unit 21003 determines to end animation playback when it receives an instruction to end playback based on pressing a stop button, etc. It may also determine to end animation playback after a predetermined period of time has elapsed since animation playback began. If animation playback has not ended (step 2S08; NO), the playback unit 21003 continues simultaneous playback of the animation of the two work machine models 2TM. On the other hand, if animation playback is to end (step 2S08; YES), the playback unit 21003 ends the animation playback process.

[0194] Of the processing flows described using FIG. 23, steps 2S00, 2S01, 2S03, 2S04, 2S05, 2S06, and 2S08 are not essential components of the playback system 210, and other embodiments may not include such steps.

[0195] (Display image of playback system) Figures 29 and 30 are diagrams showing examples of display images based on designated log information 2TL1 and comparison log information 2TL2 of the playback system according to the second embodiment. Figure 31 is a diagram showing an example of a display image based on designated log information 2TL1 of the playback system according to the second embodiment. When comparing designated log information 2TL1 and comparison log information 2TL2, the CPU 2100 of the playback system 210 according to the second embodiment causes the display unit 2102 to display display images 2D1 and 2D2 as shown in Figures 29 and 30, for example.

[0196] The display image 2D1 shown in FIG. 29 includes a first display region 2DR1 and a second display region 2DR2. An action image 2D11 is displayed in the first display region 2DR1. The action image 2D11 includes an action image 2D111 (shown in solid lines) based on designated log information 2TL1 and an action image 2D112 (shown in dashed lines) based on comparison log information 2TL2, superimposed, for example, in different display colors. The action image 2D11 also includes a terrain image 2D113 generated to combine the action images 2D111 and 2D112. The captured image 2D12 is displayed in the second display region 2DR2. The captured image 2D12 includes a captured image 2D121 based on video information 2MV corresponding to designated log information 2TL1 and a captured image 2D122 based on video information 2MV corresponding to comparison log information 2TL2. In this example, the captured image 2D12 includes images of the dump truck bed, work equipment, and bucket load, which are the loading targets for excavated materials, etc. In this example, by comparing the captured image 2D121 and the captured image 2D122, it is possible to grasp, for example, the state of the bucket load.

[0197] The display image 2D1 also includes supplemental information (image) 2D13. The supplemental information 2D13 includes a timeline 2D131 of the results of estimation of the work content based on the specified log information 2TL1, a timeline 2D132 of the results of estimation of the work content based on the comparison log information 2TL2, and a playback time icon 2D133. Note that a playback system 210 according to another embodiment may display the two action images 2D111 and 2D112 side by side, rather than displaying them overlapping (superimposing) each other. In this case, the images may be displayed on two displays arranged side by side.

[0198] The display image 2D2 shown in FIG. 30 includes a first display area 2DR1 and a second display area 2DR2. An action image 2D21 is displayed in the first display area 2DR1. The action image 2D21 has a different viewpoint from the action image 2D11 shown in FIG. 29 . For example, the viewpoint can be changed by performing a predetermined mouse operation on the action image 2D11. The action image 2D21 includes an action image 2D211 (shown in solid lines) based on the designated log information 2TL1 and an action image 2D212 (shown in dashed lines) based on the comparison log information 2TL2, superimposed, for example, in different display colors. The action image 2D21 also includes a terrain image 2D213 generated to be combined with the action image 2D211 and the action image 2D212. The captured image 2D22 is displayed in the second display area 2DR2. The captured image 2D22 includes a captured image 2D221 based on the moving image information 2MV corresponding to the designated log information 2TL1, and a captured image 2D222 based on the moving image information 2MV corresponding to the comparison log information 2TL2.

[0199] The display image 2D2 also includes supplemental information (image) 2D23. The supplemental information 2D23 includes a time series 2D231 of the results of work content estimation based on the specified log information 2TL1, a time series 2D232 of the results of work content estimation based on the comparison log information 2TL2, and a playback time icon 2D233.

[0200] The display image 2D1A shown in FIG. 31 includes a first display area 2DR1 and a second display area 2DR2. An action image 2D11 is displayed in the first display area 2DR1. The action image 2D11 is an action image 2D111 based on the designated log information 2TL1. The action image 2D11 also includes a terrain image 2D113 generated to be composited with the action image 2D111. A captured image 2D12 is displayed in the second display area 2DR2. The captured image 2D12 is a captured image 2D121 based on the moving image information 2MV corresponding to the designated log information 2TL1. In this example, the captured image 2D12 includes images of the dump truck bed, work equipment, and bucket load, which are the loading targets for excavated materials, etc. In this example, the captured image 2D121 allows the user to grasp, for example, the state of the bucket load. The display image 2D1 also includes supplemental information (image) 2D13. The supplemental information 2D13 includes a time series 2D131 of the results of work content estimation based on the specified log information 2TL1, and a playback time icon 2D133.

[0201] (Actions and Effects) As described above, the playback system 210 according to the second embodiment comprises an acquisition unit 21000 and a playback unit 21003. The acquisition unit 21000 acquires log information 2TL of the work machine 23 associated with a time, and video information 2MV representing captured video captured by the work machine 23 and associated with a time. The playback unit 21003 sequentially applies operation information representing the operation of the work machine 23 based on the log information 2TL to a work machine model 2TM, which is a three-dimensional model of the work machine 23, to sequentially play back operation images 2D11 representing the operation of the work machine 23 in a first display area 2DR1 of the display unit 2102, and sequentially plays back captured images 2D12 based on the video information 2MV in a second display area 2DR2, different from the first display area 2DR1 of the display unit 2102, in synchronization with the operation images 2D11. With this configuration, by referencing the captured images captured by the work machine 23, it is possible to analyze in detail the work performed by the work machine 23.

[0202] Furthermore, in this embodiment, the acquisition unit 21000 acquires each piece of log information 2TL and each piece of moving image information 2MV of a plurality of work machines 23, and the playback unit 21003 superimposes and sequentially plays back each operation image 2D111, 2D112 of each work machine 23 in the first display area 2DR1, and also plays back each captured image 2D121, 2D122 based on each piece of moving image information 2MV side by side in the second display area 2DR2. With this configuration, it is possible to simultaneously display an animation of a plurality of 3D models based on a plurality of pieces of log information 2TL and a video of a plurality of captured images.

[0203] <Third Embodiment> A playback system and playback method according to the third embodiment will be described in detail below with reference to Figs. 32 to 34. Fig. 32 is a diagram showing the functional configuration of a playback system according to the third embodiment. Fig. 33 is a diagram showing the processing flow of the playback system according to the third embodiment. Fig. 34 is a diagram showing an example of a display image according to the third embodiment. A playback system 210a according to the third embodiment targets multiple work machines 23 working (having worked) at the same time at the same work site, for example, and displays each work machine 23 on an overhead image representing the work site, so that the work machine 23 to be played back can be switched by identifying the work machine 23 to be played back on the overhead image.

[0204] The playback system 210a of the third embodiment shown in Fig. 32 differs from the playback system 210 of the second embodiment shown in Fig. 22 in the following configuration. That is, part of the content of the acquisition unit 21000a shown in Fig. 32 is different from that of the acquisition unit 21000 shown in Fig. 22. Furthermore, the CPU 2100 shown in Fig. 32 newly includes a playback range designation unit 21007 and an overhead image display unit 21008.

[0205] The playback range designation unit 21007 designates the range to be played back by, for example, designating the area of ​​the work site to be played back and the date and time to be played back in accordance with the operator's operation. The area of ​​the work site can be designated by a coordinate range or by specifying the registered name of a pre-registered coordinate range.

[0206] The overhead image display unit 21008 generates an overhead image including multiple work machines 23 located at the same time within the same work site area specified by the playback range specification unit 21007, and displays the overhead image including each work machine 23 in a selectable form within a third display area 2DR3 that is different from the first display area 2DR1 and the second display area 2DR2 of the display unit 2102.

[0207] The acquisition unit 21000a acquires all log information 2TL that matches the conditions specified by the playback range designation unit 21007 as the target for estimation by the estimation unit 21004, and also acquires log information 2TL and video image information 2MV of the work machine 23 selected using the overhead image displayed by the overhead image display unit 21008 as the target for playback by the playback unit 21003.

[0208] Other configurations within the CPU 2100 are the same in the second and third embodiments.

[0209] As shown in FIG. 33 , in the playback system 210a of the third embodiment, first, the playback range designation unit 21007 designates the range to be played back (step 2S101). Next, the acquisition unit 21000a acquires all log information 2TL that satisfies the playback conditions (step 2S102). Next, the estimation unit 21004 estimates the work content based on all the acquired log information 2TL (step 2S103). Next, the reception unit 21001 receives a playback instruction (step 2S104). Next, the overhead image display unit 21008 displays an overhead image (step 2S105). Next, the overhead image display unit 21008 identifies the work machine 23 to be played back based on mouse operations or the like on the overhead image (step 2S106). Note that in the subsequent processing of step 2S106, the work machine 23 to be played back may or may not be re-identified.

[0210] Next, the acquisition unit 21000 selects and reads out from the storage 2105 the work machine model 2TM that corresponds to the referenced work machine identification information, based on the work machine identification information as the type of the specified work machine 23 (step 2S107). Next, the extraction unit 21002 extracts information to be used for playback from the log information 2TL1 (step 2S108).

[0211] Next, the playback unit 21003 performs playback processing (step 2S109). In step 2S109, the action image playback (2S109a), video image playback (2S109b), topographic image display (2S109c), and supplemental information display (2S109d) correspond to the action image playback (2S07a), video image playback (2S07b), topographic image display (2S07c), and supplemental information display (2S07d) shown in Fig. 23, respectively, and their explanations will be omitted.

[0212] The determination process in step 2S110 is the same as the determination process in step 2S08 shown in Fig. 23. However, in the case of step 2S110, if playback is not to be ended, the process returns to step 2S105.

[0213] Fig. 34 shows an example of a display image according to the third embodiment. The display image 2D3 shown in Fig. 34 includes a first display area 2DR1, a second display area 2DR2, and a third display area 2DR3. The first display area 2DR1 includes an operation image 2D31. In this example, the operation image 2D31 includes an operation image 2D311 of one identified work machine and a terrain image 2D313. The second display area 2DR2 includes a captured image 2D32. The third display area 2DR3 includes an overhead image 2D34. In this case, the overhead image 2D34 includes icons 2D341 and 2D342 corresponding to two work machines 23. The icons 2D341 and 2D342 indicate the imaging direction with triangular dashed areas and the movement direction with dashed arrows.

[0214] The display image 2D3 also includes supplemental information 2D33. The supplemental information 2D33 includes a time series of the estimation results of the work content based on the specified log information 2TL, a time series indicating the control state of the automatic control (e.g., the state of positional accuracy), and a playback time icon 2D333.

[0215] According to the present embodiment, the playback system 210a is newly equipped with an overhead image display unit 21008. The overhead image display unit 21008 generates an overhead image 2D34 including multiple work machines 23 located in the same work site area at the same time, and displays the overhead image 2D34 including each work machine in a selectable form in a third display area 2DR3, which is different from the first display area 2DR1 and the second display area 2DR2 of the display unit 2102. The acquisition unit 21000a also acquires log information 2TL and moving image information 2MV of a work machine 23 selected in the overhead image 2D34. With this configuration, for example, when multiple work machines 23 are to be analyzed (or monitored), it is possible to easily select the work machine 23 to be played back.

[0216] (Modification) The contents of the log information 2TL according to the above embodiment are not limited to those in other embodiments. For example, if the work machine 23 is not a hydraulic excavator but a different vehicle type, log information 2TL according to that vehicle type is recorded. Examples of other vehicle types include a wheel loader and a bulldozer.

[0217] Furthermore, the regeneration system 210 according to the above embodiment has been described as being installed at a location away from the work machine 23 and connected to the data logger 220 mounted on the work machine 23 via a wide area communication network, but other embodiments are not limited to this configuration.

[0218] For example, in a replay system 210 according to another embodiment, part or all of the configuration of the replay system 210 may be installed inside the work machine 23. In this case, the data logger 220 may transmit the log information 2TL to the replay system 210 via an internal network of the work machine 23, rather than via a wide area communication network. In this way, the operator on board the work machine 23 can check the movements of the work machine 23 that he or she is operating on the spot by playing back an animation. In addition, by playing back a model movement of the work machine 23 for the operator of the work machine 23, it can be used as guidance.

[0219] The playback system 210 installed inside the work machine 23 may also acquire the log information 2TL of other work machines 23 via a wide area communication network, etc. In this way, it is possible to play back animations of the states of work machines 23 other than the work machine 23 on which the playback system 210 is installed.

[0220] Furthermore, the playback system 210 according to another embodiment may be installed in a location away from the work machine 23, and may transmit and display the video information generated by the animation playback process on a monitor mounted on the work machine 23.

[0221] Furthermore, in other embodiments, one aspect of the playback instruction received from the operator may be, for example, a playback period. For example, the playback period may be a playback start time and a playback end time. In this case, the playback system 210 performs playback of the work machine 23 for the received playback period. Furthermore, in other embodiments, it is not essential to specify a playback end time. For example, in other embodiments, a playback instruction from the operator may be a mode in which only a playback start time is received and playback is performed for a certain period of time from the playback start time, or a mode in which playback continues as long as log information exists, or playback may be stopped in response to the occurrence of various other events.

[0222] The acquired log information 2TL does not need to be arranged in chronological order. In this case, the playback unit 21003 may apply the information to be used for playback from the log information 2TL to the work machine model 2TM in chronological order.

[0223] The various processing steps of the playback system 210 described above are stored in the form of a program on a computer-readable recording medium, and the computer reads and executes this program to perform the various processing steps. Computer-readable recording media include magnetic disks, magneto-optical disks, CD-ROMs, DVD-ROMs, and semiconductor memories. Alternatively, the computer program may be distributed to a computer via a communication line, and the computer that receives the program may execute the program.

[0224] The program may be one that realizes part of the above-mentioned functions, or may be one that realizes the above-mentioned functions in combination with a program already recorded in the computer system, such as a so-called differential file or differential program.

[0225] Some or all of the functions of the playback system 210 described above may be assigned to the work machine 23. For example, some or all of the functions of the acquisition unit 21000, reception unit 21001, extraction unit 21002, playback unit 21003, estimation unit 21004, determination unit 21005, synchronization unit 21006, memory 2101, display unit 2102, operation reception unit 2103, communication interface 2104, and storage 2105 may be assigned to the work machine 23.

[0226] Although several embodiments of the present disclosure have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the inventions described in the claims and their equivalents, as well as in the scope and spirit of the inventions.

[0227] (Additional Note) The playback systems 210 and 210a according to the present disclosure can be understood, for example, as follows.

[0228] (7) A playback system 210, 210a according to a seventh aspect comprises an acquisition unit 21000, 21000a that acquires log information 2TL of a work machine 23 associated with a time and video information 2MV representing captured video captured by the work machine and associated with a time, and a playback unit 21003 that sequentially applies operation information representing the operation of the work machine based on the log information to a three-dimensional model of the work machine (work machine model 2TM) to sequentially play back operation images 2D11, 2D21, 2D31 representing the operation of the work machine in a first display area 2DR1 of a specified display unit 2102, and sequentially plays back captured images 2D12, 2D22, 2D32 based on the video information 2MV in a second display area 2DR2 different from the first display area of ​​the display unit, in synchronization with the operation images. According to this aspect and each of the following aspects, work performed by a work machine can be analyzed in more detail.

[0229] (8) A playback system 210a according to an eighth aspect is the playback system of (7), and further includes an overhead image display unit 21008 that generates an overhead image 2D34 including multiple work machines located in the same work site area at the same time, and displays the overhead image including each of the work machines in a selectable form in a third display area 2DR3 that is different from the first and second display areas of the display unit, and the acquisition unit 21000a acquires the log information and the video image information of the work machine selected in the overhead image.

[0230] (9) The playback system 210, 210a according to a ninth aspect is the playback system of (7) or (8), wherein the acquisition unit 21000, 21000a acquires the log information and the video image information of a plurality of work machines, and the playback unit 21003 superimposes the operation images of the work machines and plays them sequentially in the first display area, and also arranges and plays the captured images based on the video image information side by side in the second display area.

[0231] Fourth Embodiment A display control system and a display control method according to a fourth embodiment will be described in detail below with reference to FIGS.

[0232] (Overall configuration of analysis support system) Figure 35 is a diagram showing the overall configuration of an analysis support system according to the fourth embodiment. The analysis support system 31 has a playback system 310 and a data logger 320 mounted on each of a plurality of work machines 33. In this embodiment, the playback system 310 is an example configuration of a display control system according to the present disclosure.

[0233] The work machine 33 is the subject of work analysis by the playback system 310. Examples of the work machine 33 include hydraulic excavators, wheel loaders, and bulldozers. In the following description, a hydraulic excavator will be used as an example of the work machine 33. Each work machine 33 is equipped with multiple sensors. The data logger 320 chronologically records and accumulates information indicating the status of the work machine 33 obtained by the sensors. Hereinafter, information recorded by the data logger 320 indicating the status of the work machine 33 at each time will also be referred to as log information. Note that if the operating mechanism that operates the work machine 33 is configured to operate the work machine 33 using electrical operating signals, information on the operating signals of the work machine 33 may be recorded and accumulated chronologically and included in the log information. The data logger 320 also transmits the recorded log information to the playback system 310 via a wide area communication network at regular time intervals. Note that the regular time intervals may be, for example, every five minutes. The playback system 310 records the log information received from the data logger 320 on a recording medium. The function of the playback system 310 will be described later.

[0234] (Structure of Work Machine) Figure 36 is a diagram showing the structure of a work machine according to a fourth embodiment. A work machine 33, which is a hydraulic excavator, excavates earth and sand and levels the ground at work sites and the like. As shown in Figure 36, the work machine 33, which is a hydraulic excavator, comprises a lower running body 331 for traveling, and a rotatable upper rotating body 332 that is installed above the lower running body 331. In addition, the upper rotating body 332 is provided with a driver's cab 332A, a work implement 332B, and two GNSS (Global Navigation Satellite System) antennas 3G1 and 3G2.

[0235] The undercarriage 331 has a left crawler track 3CL and a right crawler track 3CR. The work machine 33 moves forward, turns, and reverses by rotation of the left crawler track 3CL and the right crawler track 3CR.

[0236] The operator's cab 332A is a place where an operator of the work machine 33 gets in and operates the work machine 33. The operator's cab 332A is installed, for example, on the left side of the front end of the upper rotating body 332. The internal configuration of the operator's cab 332A will be described later.

[0237] The work implement 332B consists of a boom 3BM, an arm 3AR, and a bucket 3BK. The boom 3BM is attached to the front end of the upper rotating body 332. An arm 3AR is attached to the boom 3BM. A bucket 3BK is attached to the arm 3AR. A boom cylinder 3SL1 is attached between the upper rotating body 332 and the boom 3BM. By driving the boom cylinder 3SL1, the boom 3BM can be moved relative to the upper rotating body 332. An arm cylinder 3SL2 is attached between the boom 3BM and the arm 3AR. By driving the arm cylinder 3SL2, the arm 3AR can be moved relative to the boom 3BM. A bucket cylinder 3SL3 is attached between the arm 3AR and the bucket 3BK. By driving the bucket cylinder 3SL3, the bucket 3BK can be moved relative to the arm 3AR. The above-described upper rotating body 332, boom 3BM, arm 3AR, and bucket 3BK provided on the work machine 33, which is a hydraulic excavator, are one aspect of the movable parts of the work machine 33. The bucket 3BK is also one example of a configuration of a "work tool" according to the present disclosure. The bucket 3BK is provided with a cutting edge 3BKT used for excavation, etc.

[0238] (Configuration of the Operator's Cabin) FIG. 37 is a diagram showing the configuration of the operator's cab of a work machine according to the fourth embodiment.

[0239] 37, the driver's cab 332A is provided with operation levers 3L1 and 3L2, foot pedals 3F1 and 3F2, and travel levers 3R1 and 3R2. The operation levers 3L1 and 3L2 are located on the left and right sides of the seat 3ST inside the driver's cab 332A. The foot pedals 3F1 and 3F2 are located on the floor inside the driver's cab 332A, in front of the seat 3ST.

[0240] An example of an operation pattern showing the correspondence between input operations on the control levers 3L1, 3L2 and travel levers 3R1, 3R2 and the operation of the work machine 33, which is a hydraulic excavator, is as follows.

[0241] The control lever 3L1, located on the left side facing the front of the cab, is an operating mechanism for performing the swing operation of the upper rotating body 332 and the excavation / dumping operation of the arm 3AR. Specifically, when the operator of the work machine 33 tilts the control lever 3L1 forward, the arm 3AR performs a dumping operation. When the operator of the work machine 33 tilts the control lever 3L1 rearward, the arm 3AR performs an excavation operation. When the operator of the work machine 33 tilts the control lever 3L1 to the right, the upper rotating body 332 swings to the right. When the operator of the work machine 33 tilts the control lever 3L1 to the left, the upper rotating body 332 swings to the left. When the control lever 3L1 is tilted forward or backward, the upper rotating body 332 swings to the right or left, and when the control lever 3L1 is tilted left or right, the arm 3AR performs a dumping operation or an excavation operation.

[0242] The control lever 3L2, located on the right side as viewed from the front of the cab, is an operating mechanism for performing the excavation / dump operation of the bucket 3BK and the raising / lowering operation of the boom 3BM. Specifically, when the operator of the work machine 33 tilts the control lever 3L2 forward, the boom 3BM is lowered. When the operator of the work machine 33 tilts the control lever 3L2 rearward, the boom 3BM is raised. When the operator of the work machine 33 tilts the control lever 3L2 to the right, the bucket 3BK is dumped. When the operator of the work machine 33 tilts the control lever 3L2 to the left, the bucket 3BK is excavated.

[0243] The travel levers 3R1, 3R2 are operating mechanisms for controlling the operation of the undercarriage 331, i.e., for controlling the travel of the work machine 33. The travel lever 3R1, located on the left side as one faces the front of the cab, corresponds to the rotational drive of the left crawler 3CL of the undercarriage 331. Specifically, when the operator of the work machine 33 tilts the travel lever 3R1 forward, the left crawler 3CL rotates in the forward direction. When the operator of the work machine 33 tilts the travel lever 3R1 rearward, the left crawler 3CL rotates in the reverse direction.

[0244] The travel lever 3R2, located on the right side facing the front of the cab, corresponds to the rotational drive of the right crawler 3CR of the undercarriage 331. Specifically, when the operator of the work machine 33 tilts the travel lever 3R2 forward, the right crawler 3CR rotates in the forward direction. Conversely, when the operator of the work machine 33 tilts the travel lever 3R2 rearward, the right crawler 3CR rotates in the reverse direction. The foot pedals 3F1, 3F2 are linked to the travel levers 3R1, 3R2, respectively, and travel can also be controlled by the foot pedals 3F1, 3F2.

[0245] The above-described operation pattern is merely an example, and is not limited to the above-described pattern depending on the model of the hydraulic excavator, etc.

[0246] It should be noted that, depending on the embodiment, the work machine 33 described using Figure 36 may not be equipped with GNSS antennas 3G1, 3G2.

[0247] (Functional Configuration of Playback System) Fig. 38 is a diagram showing the functional configuration of a playback system according to the fourth embodiment. Hereinafter, functions of a playback system 310 according to the fourth embodiment will be described with reference to Fig. 38. As shown in Fig. 38, the playback system 310 includes a CPU 3100, a memory 3101, a display unit 3102, an operation reception unit 3103, a communication interface 3104, and storage 3105. Note that the CPU (Central Processing Unit) 3100 may be a processor such as an FPGA or GPU instead of a CPU.

[0248] The CPU 3100 is a processor that controls the overall operation of the playback system 310. The various functions of the CPU 3100 will be described later.

[0249] The memory 3101 is a so-called main storage device, and the memory 3101 stores instructions and data necessary for the CPU 3100 to operate based on a program.

[0250] The display unit 3102 is a display device capable of visually displaying information, and is, for example, a liquid crystal display or an organic EL display.

[0251] The operation reception unit 3103 is an input device, such as a general mouse, keyboard, or touch sensor.

[0252] The communication interface 3104 is a communication interface for communicating with the data logger 320 .

[0253] The storage 3105 is a so-called auxiliary storage device, such as a hard disk drive (HDD) or solid state drive (SSD). The storage 3105 stores log information 3TL received from the data logger 320, the vehicle type of the work machine 33, and a work machine model 3TM, which is a 3D model prepared in advance for each model. The work machine model 3TM will be described later. The storage 3105 also stores a unit task prediction model 3PM1 and an element task prediction model 3PM2, which are trained machine learning models used when estimating the work content of the work machine 33, heat maps (3H1, 3H2) generated during the estimation process, estimated work content 3R of the work machine 33, and three-dimensional point cloud information 3MAP representing topographical information of the work site. The unit task prediction model 3PM1, element task prediction model 3PM2, and heat maps (3H1, 3H2) will be described later. The three-dimensional point cloud information 3MAP includes, for example, three-dimensional point cloud information measured at the work site using a drone, etc., as well as information on topography that is continuously updated from the cutting edge coordinate information of the work machine 33 in operation, and is recorded in storage 3105.

[0254] The functions of the CPU 3100 of the playback system 310 according to the fourth embodiment will be described in detail. The CPU 3100 operates based on a predetermined program to function as an acquisition unit 31000, a reception unit 31001, an extraction unit 31002, an image generation unit 31003, an estimation unit 31004, a determination unit 31005, a synchronization unit 31006, a display control unit 31007, and a designation unit 31008. The predetermined program may be for implementing part of the functions of the playback system 310. For example, the program may be combined with another program already stored in the storage 3105 or with another program implemented in another device to perform the functions. In other embodiments, the playback system 310 may include a custom large-scale integrated circuit (LSI) such as a programmable logic device (PLD) in addition to or instead of the above configuration. Examples of PLDs include programmable array logic (PAL), generic array logic (GAL), complex programmable logic device (CPLD), and field programmable gate array (FPGA). In this case, some or all of the functions realized by a processor may be realized by the integrated circuit.

[0255] The acquisition unit 31000 acquires the log information 3TL to be played back from among the multiple pieces of log information 3TL recorded and accumulated in the storage 3105. Here, it is assumed that the multiple pieces of log information 3TL are recorded in the storage 3105 as files recorded with different file names. The acquisition unit 31000 acquires, for example, two pieces of log information 3TL to be compared. The following describes an example in which the acquisition unit 31000 acquires two pieces of log information 3TL to be compared. One of the two pieces of log information acquired by the acquisition unit 31000 is the log information designated by the operator as the target for playback by, for example, a file name. Hereinafter, this log information will be referred to as designated log information 3TL1 (first log information). The other of the two pieces of log information is the log information to be played back simultaneously with the playback based on the designated log information 3TL1. Hereinafter, this log information will be referred to as comparison log information 3TL2 (second log information). The comparison log information 3TL2 may be log information selected in advance as a "model" for the work content for each of various work contents and conditions, for example, from the viewpoint of good fuel economy, short work time, driving by an experienced operator, etc. Alternatively, the comparison log information 3TL2 may be, for example, past log information recorded for a previous operation performed by the same operator as the designated log information 3TL1. In this case, by comparing the two log information, it is possible to analyze, for example, the operator's proficiency.

[0256] The acquisition unit 31000 according to this embodiment automatically searches for and acquires comparison log information 3TL2 appropriate as a comparison target for the specified log information 3TL1, based on the determination result by the determination unit 31005 (described below). However, the acquisition unit 31000 according to other embodiments is not limited to the above-described aspect. For example, the acquisition unit 31000 may acquire log information specified by an operator using a file name or the like as the comparison log information 3TL2. Furthermore, if only one piece of comparison log information 3TL2 is recorded, the acquisition unit 31000 may acquire that one piece of comparison log information 3TL2.

[0257] The reception unit 31001 receives a predetermined regeneration instruction from the operator of the regeneration system 310. For example, the reception unit 31001 receives a regeneration instruction for the work machine 33 from the operator of the regeneration system 310.

[0258] The extraction unit 31002 extracts, from the acquired log information TL, angle information to be used for playback of the work machine 33. The angle information is an example of "operation information (motion information)" according to the present disclosure.

[0259] The image generation unit 31003 sequentially generates operation images that represent the operation of the work machine 33 by sequentially applying operation information that represents the operation of the work machine 33 based on the log information 3TL to the work machine model 3TM, which is a three-dimensional model of the work machine 33, and sequentially generates movement trajectory images that represent the movement trajectory of a predetermined location of the work machine 33 in association with the operation images, based on the operation information. The image generation unit 31003 can also generate movement trajectory images when the work content estimated by the estimation unit 31004 corresponds to the work content specified by the designation unit 31008. Note that, when displaying operation images in animation, the movement trajectory is a path that starts at the position of that location at a time that is a predetermined time back from the time of the displayed operation image, passes through each position of that location at each time when time is advanced by a predetermined time up to the current time, and ends at the position of that location at the current time. The movement trajectory images can, for example, be multiple images of a predetermined shape that represent each of the positions, such as a spherical three-dimensional image, or a linear or planar image connecting the positions. Furthermore, the predetermined location on the work machine 33 for displaying the movement trajectory may be, for example, the cutting edge 3BKT of the bucket 3BK (work implement) or a predetermined position on the upper rotating body 332. Furthermore, for other work machines 33, it may be the cutting edge of the blade, a predetermined position such as the ground contact position of the crawler or tire, or the central coordinates of the work machine 33.

[0260] Furthermore, the image generation unit 31003 may generate a part image that represents a specific part of the work implement 332B equipped on the work machine 33 a specific time ago in association with the operation image. When the operation image is displayed as an animation, the part image is a three-dimensional image that represents the part at a time that is a specific time prior to the time of the operation image being displayed. The specific part may be, for example, the bucket 3BK. In this case, the part image is a three-dimensional image that represents the bucket 3BK, and may be displayed, for example, as a semi-transparent image superimposed on the operation image.

[0261] The image generation unit 31003 can also change the display color of the movement trajectory image, the size and thickness of the image representing the movement trajectory, and other aspects based on at least one of the work content estimated based on the log information 3TL, the accuracy of the position information included in the log information 3TL, information indicating whether or not automatic operation control is being performed on the work machine 33 included in the log information 3TL, the workload of the work machine 33 based on the log information 3TL, and information indicating the workload of the work machine 33 based on the log information 3TL. The accuracy of the position information can decrease or become unstable when the number of satellite acquisitions in the GNSS is low or when the baseline length to the reference point is long. Therefore, the display aspect of the movement trajectory can be changed depending on whether the accuracy of the position information is good or poor. Examples of indicators of the accuracy of the position information include the root mean square (RMS) value and the dilution of precision (DOP) value. Automatic operation control refers to fully automatic or semi-automatic control of the operation of the work machine 33. The display aspect of the movement trajectory can be changed depending on whether automatic operation control is being performed or not. The amount of work done by the work machine 33 is a value that indicates, for example, the depth of excavation, the load during excavation, the weight of the cargo, etc. The workload of the work machine 3 is a value that indicates the magnitude of the load on the work, calculated from, for example, the hydraulic pressure of the cylinder, the hydraulic pressure of the pump, the engine torque, the tractive force, etc.

[0262] The estimation unit 31004 estimates the work content of the work machine 33 at each time from the acquired log information 3TL.

[0263] The determining unit 31005 determines whether or not the comparison log information 3TL2 is appropriate as a comparison target for the specified log information 3TL1, based on the information included in the specified log information 3TL1.

[0264] The synchronization unit 31006 performs processing to synchronize the playback of the work machine model 3TM based on the specified log information 3TL1 with the playback of the work machine model 3TM based on the comparison log information 3TL2. Specifically, the synchronization unit 31006 specifies the playback start time on the timeline of the animation of the work machine model 3TM based on the work content estimated by the estimation unit 31004.

[0265] The display control unit 31007 superimposes the action image and the movement trajectory image generated by the image generation unit 31003 and displays them on the display unit 3102. The display control unit 31007 also superimposes the part image on the action image and displays them on the display unit 3102. The display control unit 31007 may also display an image representing the terrain (terrain image) on the display unit 3102 based on the three-dimensional point cloud information 3MAP.

[0266] The designation unit 31008 designates the work content for which a movement trajectory image is to be generated in accordance with instructions from an operator. The work content can be designated, for example, by unit work, by element work, or by a combination of unit work and element work.

[0267] (Processing Flow of the Playback System) Hereinafter, the specific processing flow performed by the playback system 310 will be described in detail with reference to FIGS.

[0268] 39 starts when a dedicated application is started by the operator of the playback system 310. When the dedicated application is started by the operator's operation, the acquisition unit 31000 of the CPU 3100 expands and acquires the designated log information 3TL1 designated as the target for playback in the memory 3101 (step 3S00).

[0269] Here, the log information 3TL (designated log information 3TL1, comparison log information 3TL2) will be described with reference to FIGS.

[0270] As shown in Figures 40 to 42, the log information 3TL includes work machine identification information. Specifically, the work machine identification information is an individual identification number for individually identifying the work machine 33. In Figures 40 to 42, the work machine identification information is assigned to correspond to the vehicle type, model, type, and serial number of the work machine 33, indicating a hydraulic excavator, wheel loader, bulldozer, etc. Note that the work machine identification information may be numbers, letters, symbols, or a combination of these, in addition to numbers.

[0271] As shown in Figure 40, the log information 3TL includes information indicating the position and attitude of the work machine 33 at each time, and angle information of the movable parts of the work machine 33. Specifically, the log information 3TL records the position of the work machine 33, the roll angle of the work machine 33, which is the left-right tilt of the machine body, the pitch angle, which is the fore-and-aft tilt of the machine body, the slewing angle, boom angle, arm angle, and bucket angle for each time. Here, the data logger 320 mounted on the work machine 33 identifies and records the position of the work machine 33 based on positioning information indicating latitude, longitude, and altitude, which is information obtained by receiving from the GNSS antennas 3G1, 3G2, for example. The data logger 320 also calculates and records the roll angle and pitch angle of the work machine 33 based on measurement results from an IMU (Inertial Measurement Unit) mounted on the work machine 33. The data logger 320 also calculates and records the rotation angle of the upper rotating body 332 based on the positioning information obtained from each of the GNSS antennas 3G1 and 3G2 provided on the upper rotating body 332. Furthermore, the data logger 320 calculates and records the boom angle, arm angle, and bucket angle based on the extension and retraction degrees of the boom cylinder 3SL1, arm cylinder 3SL2, and bucket cylinder 3SL3. Note that the boom angle, arm angle, bucket angle, and rotation angle may be acquired, for example, by attaching IMUs to the boom, arm, bucket, and upper rotating body and using these IMUs.

[0272] The position, roll angle, and pitch angle are information necessary to identify the position and attitude of the work machine 33 itself. Therefore, for example, in an embodiment in which only the movements of the movable parts of the work machine 33, i.e., the upper rotating body 332, boom 3BM, arm 3AR, and bucket 3BK, are played back as animation, and the position and attitude of the work machine 33 itself are not reproduced, information on the position, roll angle, and pitch angle does not need to be included in the log information.

[0273] Log information 3TL shown in Figure 40 corresponds to the "operation information" according to the present disclosure. Alternatively, log information 3TL shown in Figure 40 is information that represents the operation of the work machine 33 that can be seen from outside the work machine 33, and can also be referred to as external operation information. That is, in this embodiment, the operation information or external operation information includes the position information, roll angle, pitch angle, swing angle, boom angle, arm angle, and bucket angle of the work machine 33. Unlike drive mechanism information and operation information, which will be described later, in this embodiment, operation information or external operation information has the advantage that, as information used when estimating work content, it is less susceptible to the influence of differences in the vehicle class of the work machine 33.

[0274] As shown in FIG. 41 , the log information 3TL also includes the PPC pressure, which is the pilot oil pressure indicating the degree of input to the control levers 3L1, 3L2, etc. by the operator at each time, i.e., the degree of lever tilt and the degree of pedal depression. Specifically, the log information 3TL records, for each time, the PPC pressure of the control levers 3L1, 3L2, travel levers 3R1, 3R2, or foot pedals 3F1, 3F2 corresponding to each type of operation performed by the operator: left / right swing, arm digging / dumping, boom raising / lowering, bucket digging / dumping, right track forward / reverse, and left track forward / reverse. Note that the times shown in FIG. 41 correspond to the times shown in FIG. 40 . In this embodiment, the information shown in FIG. 41 is also referred to as operation information.

[0275] Furthermore, as shown in Figure 42, the log information 3TL includes information indicating the status of the main drive mechanisms of the engine, hydraulic pump, etc. of the work machine 33 at each time. Specifically, the log information 3TL records the engine coolant temperature, engine output, instantaneous fuel consumption, and hydraulic pump oil temperature for each time. Note that each time shown in Figure 42 corresponds to each time in Figures 40 and 41. In this embodiment, the information shown in Figure 42 is also referred to as drive mechanism information.

[0276] Returning to FIG. 39, the estimation unit 31004 of the CPU 3100 estimates the work content of the work machine 33 at each time based on the designated log information 3TL1 acquired in step 3S00 (step 3S01).

[0277] Here, the procedure by which the estimation unit 31004 estimates the work content of the work machine 33 from the log information 3TL will be described with reference to Figure 43. The estimation unit 31004 estimates the work content of the work machine 33 for both unit tasks and element tasks. A unit task is a task that accomplishes one work purpose. An element task is an element that makes up a unit task, and is a task that represents a series of actions or tasks categorized by purpose.

[0278] Examples of unit task classifications include "excavation and loading," "plowing," "slope (from below)," "load collection," "travel," and "parking / restoring" as shown in FIG. 43 , as well as "ditch digging," "backfilling," and "slope (from above)." Excavation and loading involves digging and scraping away soil or rocks, and loading the scraped soil or rocks onto the bed of a transport vehicle. Excavation and loading is a unit task consisting of excavation, loading and unloading, unloading, waiting for unloading, and holding down the bed. Plowing is the task of scraping away excess ground irregularities to a predetermined height. Plowing is a unit task consisting of excavation and unloading, or excavation, loading and unloading, and unloading, and may include leveling and sweeping. Slope (from below) is the task of creating a slope using a work machine 33 positioned below the target area. Compacting a slope (from below) is a unit operation consisting of rolling, excavation, loaded rotation, soil removal, and empty rotation, and may include pushing and leveling. Load collection is the operation of collecting soil and sand excavated by excavation, etc. before loading it onto a transport vehicle. Load collection is a unit operation consisting of excavation, loaded rotation, soil removal, and empty rotation, and may include pushing and leveling. Traveling is the operation of moving the work machine 33. Traveling as a unit operation is a unit operation consisting of traveling as an element operation. Stopping / standby is a state in which the bucket 3BK is free of soil and rocks and is stopped for a predetermined period of time or more. Stopping / standby as a unit operation is a unit operation consisting of stopping as an element operation. Trench excavation is the operation of digging a long, narrow trench in the ground and scraping it away. Trench excavation is a unit operation consisting of excavation, loaded rotation, soil removal, and empty rotation, and may include pushing and leveling. Backfilling is the process of filling existing trenches or holes in the ground with soil and sand to fill them flat. Backfilling is a unit operation consisting of excavation, loading and turning, soil removal, compaction, and empty turning, and may also include leveling and sweeping. Slope (from above) is the process of creating a slope using a work machine 33 positioned above the target area. Slope (from above) is a unit operation consisting of compaction, excavation, loading and turning, soil removal, and empty turning, and may also include leveling.

[0279] Examples of classifications of elemental work include "excavation," "loaded rotation," "waiting for soil discharge," "soil discharge," "empty rotation," and "load carrier holddown" shown in FIG. 43 , as well as "rolling," "leveling," and "brooming." Excavation is the work of digging up and scraping away soil or rocks using the bucket 3BK. Loaded rotation is the work of rotating the upper rotating body 332 while the bucket 3BK holds the scraped soil or rocks. Waiting for soil discharge is the work of waiting for a transport vehicle to load the scraped soil or rocks while the bucket 3BK holds the scraped soil or rocks. Soil discharge is the work of lowering the scraped soil or rocks from the bucket 3BK onto a transport vehicle or a predetermined location. Empty rotation is the work of rotating the upper rotating body 332 when the bucket 3BK is empty of soil or rocks. Load carrier holddown is the work of pressing down and leveling the soil loaded on the load carrier of a transport vehicle with the bucket 3BK from above. Compaction is the work of pushing soil and sand into disturbed ground with the bucket 3BK to shape and strengthen the ground. Leveling is the work of sweeping and leveling the soil and sand with the bottom of the bucket 3BK. Brooming is the work of sweeping and leveling the soil and sand with the side of the bucket 3BK.

[0280] The estimation unit 31004 obtains a time series of likelihoods related to the unit tasks by inputting the log information TL to the unit task prediction model 3PM1 in chronological order. The unit task prediction model 3PM1 is a model that outputs likelihoods related to the unit tasks when the log information 3TL is input, for example, by learning using teacher data, and may be stored in the storage 3105, for example.

[0281] The estimation unit 31004 also obtains a time series of likelihoods related to element works by inputting the log information 3TL to the element work prediction model 3PM2 in chronological order. The element work prediction model 3PM2 is a model that outputs likelihoods related to element works when the log information 3TL is input, for example, through learning using teacher data, and may be stored in the storage 3105, for example.

[0282] The estimation unit 31004 smooths the time series of likelihoods for unit tasks and element tasks by applying a time averaging filter to each of them, and generates a unit task heat map 3H1 representing the time series of likelihoods for the smoothed unit tasks and an element task heat map 3H2 representing the time series of likelihoods for the smoothed element tasks, as shown in FIG. 43 . The heat maps 3H1 and 3H2 are maps based on the time series of smoothed likelihoods, with task categories on the vertical axis and time on the horizontal axis, and colors representing the likelihoods of task categories. For example, the colors of the heat maps may be closer to blue as the likelihood of the task category decreases and closer to red as the likelihood of the task category increases. The estimation unit 31004 stores the heat maps 3H1 and 3H2 in the storage 3105.

[0283] The estimation unit 31004 identifies a time period during which the likelihood of a unit task is dominant, based on the time series of the smoothed likelihood, and estimates the work content of the work machine 33 during that time period. For example, during a time period during which the likelihood of the unit task "digging and loading" is dominant, the estimation unit 31004 estimates the work content of the work machine 33 to be "digging and loading". Similarly, the estimation unit 31004 identifies a time period during which the likelihood of an element task is dominant, based on the time series of the smoothed likelihood, and estimates the work content of the work machine 33 during that time period. For example, during a time period during which the likelihood of the element task "excavation" is dominant, the estimation unit 31004 estimates the work content of the work machine 33 to be "excavation". The estimation unit 31004 stores information on the work content R estimated for the work machine 33 in the storage 3105.

[0284] Returning to FIG. 39 , the acquisition unit 31000 then acquires comparison log information 3TL2 that is appropriate as a comparison target for the designated log information TL1 acquired in step 3S00 (step 3S02). At this time, the determination unit 31005 performs a determination process flow for determining whether the plurality of pieces of comparison log information 3TL2 recorded in the storage 3105 are appropriate as a comparison target for the designated log information 3TL1 acquired in step 3S00. In this determination process flow, the determination unit 31005 first selects one piece of comparison log information 3TL2 from the plurality of pieces of comparison log information 3TL2 previously recorded in the storage 3105. Then, based on the log information 3TL, the determination unit 31005 determines whether the vehicle size, the amount of change in the swing angle (magnitude of swing), the bucket height of the work machine 33 while waiting to unload, etc. are approximately the same. If it determines that they are not approximately the same, the determination unit 31005 selects one new piece of comparison log information 3TL2 and performs the same determination process as described above. Furthermore, the determination unit 31005 according to another embodiment may narrow down playback candidates based on the work location, work time, work category, and the like, in addition to the above determination processing.

[0285] 39 , next the synchronization unit 31006 performs processing to synchronize the animation playback of the work machine model 3TM based on the designated log information 3TL1 with the animation playback of the work machine model 3TM based on the comparison log information 3TL2. Specifically, the synchronization unit 31006 specifies the playback start time on a timeline in the animation of the work machine model 3TM based on the designated log information 3TL1 and the playback start time on a timeline in the animation of the work machine model 3TM based on the comparison log information 3TL2.

[0286] The method for specifying the playback start time for synchronizing the two is as follows. That is, the synchronization unit 31006 extracts the timing of unit task switching in the specified log information 3TL1 based on the unit task heat map 3H1 for the specified log information 3TL1 generated by the estimation unit 31004. Next, the synchronization unit 31006 extracts the timing of unit task switching in the comparison log information 3TL2 based on the unit task heat map 3H1 for the comparison log information 3TL2 generated by the estimation unit 31004. The synchronization unit 31006 selects one of the extracted timings and specifies it as the playback start time for the respective animation. Note that the playback system 310 according to other embodiments is not limited to the above aspect. A playback system 310 (synchronization unit 31006) according to another embodiment may, for example, compare a combination of angle information of the boom, arm, bucket, etc. in the comparison log information 3TL2 with a combination of angle information of the boom, arm, bucket, etc. in the specified log information 3TL1, extract the timing at which the two angles are close to each other, and identify this as the playback start time for each animation.

[0287] Next, the reception unit 31001 receives a playback instruction from the operator (step 3S04). One form of the playback instruction may be an operation such as pressing a playback button. The playback instruction may also include information that will be the start point of playback, such as the time, the position of the work machine 33, or various events such as the occurrence of an abnormality in the work machine 33. When the reception unit 31001 receives the playback instruction, the designation unit 31008 may also designate the work content for which a movement trajectory image is to be generated.

[0288] Next, the acquisition unit 31000 selects and reads out from the storage 3105 the work machine model 3TM corresponding to the referenced work machine identification information based on the work machine identification information as the type of work machine 33 received by the reception unit 31001 (step 3S05).

[0289] The work machine model 3TM will now be described with reference to Figure 44. As shown in Figure 44, the work machine model 3TM is information that includes work machine identification information and an exterior 3D model 3M0 of the work machine 33 indicated in the work machine identification information. The exterior 3D model 3M0 is a 3D model that represents the work machine 33, and is constructed for each part of the work machine 33, such as the lower traveling body and upper rotating body. For example, the exterior 3D model 3M0 represents the shape of the work machine 33. For example, the exterior 3D model 3M0 is made up of a lower traveling body exterior model 3M01 that represents the lower traveling body 331 of the work machine 33, an upper rotating body exterior model 3M02 that represents the upper rotating body 332, a boom exterior model 3M03 that represents the boom 3BM, an arm exterior model 3M04 that represents the arm 3AR, and a bucket exterior model 3M05 that represents the bucket 3BK.

[0290] 39 , the extraction unit 31002 extracts information to be used for playback from each of the designated log information 3TL1 and the comparison log information 3TL2 (step 3S06). For example, the extraction unit 31002 extracts various types of angle information such as the boom angle, arm angle, and bucket angle as information to be used for playback. Note that the pilot oil pressure shown in FIG. 41 may also be extracted as information to be used for playback. Also, in steps 3S00 and 3S02, only the information to be used for playback may be acquired.

[0291] Next, the image generation unit 31003 and the display control unit 31007 simultaneously play back an animation of the work machine model 3TM based on the designated log information 3TL1 and an animation of the work machine model 3TM based on the comparison log information 3TL2 (step 3S07). Here, the image generation unit 31003 generates a motion image that represents the operation of the work machine 33 while applying the various information recorded in the designated log information 3TL1 to the work machine model 3TM in chronological order of the timestamps from the playback start time specified in the synchronization processing of step 3S03 (step 3S07a). Furthermore, simultaneously with generating the motion images for this animation, the image generation unit 31003 generates a motion image that represents the operation of the work machine 33 while applying the various information recorded in the comparison log information 3TL2 to the work machine model 3TM in chronological order of the timestamps from the playback start time specified in the synchronization processing of step 3S03 (step 3S07a).

[0292] The following describes in detail the contents of the operation image generation process performed by the image generation unit 31003. The image generation unit 31003 changes the angle of the corresponding portion of the exterior 3D model 3M0 based on various angle information such as the swing angle and boom angle indicated in the log information 3TL (designated log information 3TL1, comparison log information 3TL2). For example, the image generation unit 31003 recreates the position and posture of the bucket 3BK of the work machine 33 by tilting the bucket exterior model 3M05 around the rotation axis defined at the connection position with the arm exterior model 3M04 so that the bucket angle is indicated in the log information 3TL.

[0293] Similarly, the image generation unit 31003 reproduces the position and posture of the arm 3AR of the work machine 33 by tilting the arm exterior model 3M04 around the rotation axis defined at the connection position with the boom exterior model 3M03 so that the arm angle is the angle indicated in the log information 3TL.

[0294] Similarly, the image generation unit 31003 reproduces the position and posture of the upper rotating body 332 of the work machine 33 by tilting the upper rotating body exterior model 3M02 around the rotation axis defined at the connection position with the lower running body exterior model 3M01 to the rotation angle indicated in the log information 3TL.

[0295] Similarly, the image generation unit 31003 reproduces the posture of the lower running body 331 of the work machine 33 by tilting the lower running body exterior model 3M01 around the roll rotation axis defined in the lower running body exterior model 3M01 to the roll angle indicated in the log information 3TL, and tilting it around the pitch rotation axis defined in the lower running body exterior model 3M01 to the pitch angle indicated in the log information 3TL.

[0296] Furthermore, the playback system 310 according to the fourth embodiment can play back animation of the running of the work machine 33 based on the PPC pressures for the right track forward / reverse and the left track forward / reverse at each time, which are included in the log information 3TL.

[0297] Specifically, the exterior 3D model 3M0 moves forward, backward, left / right forward, and left / right backward based on the PPC pressures for the right track forward / reverse and the left track forward / reverse. For example, the exterior 3D model 3M0 moves forward based on the PPC pressure values ​​for the right track forward and the left track forward. The speed of movement may be changed based on the PPC pressure values.

[0298] Furthermore, the exterior 3D model 3M0 is moved backward based on the numerical values ​​of the PPC pressure for the right track reverse and the left track reverse. Furthermore, the exterior 3D model 3M0 is moved so as to curve forward in the left and right directions based on the difference between the numerical values ​​of the PPC pressure for the right track forward and the left track forward. For example, if the numerical value of the PPC pressure for the right track forward is greater than the numerical value of the PPC pressure for the left track forward, the exterior 3D model 3M0 is moved so as to curve forward in the left direction. The speed of movement and the magnitude of the curve may be changed depending on the numerical values ​​of the PPC pressure for each track and the difference between the numerical values ​​of the PPC pressure.

[0299] Similarly, the exterior 3D model 3M0 is moved so as to curve backward in the left and right directions based on the difference between the PPC pressure values ​​for the right and left track retractions. For example, if the PPC pressure value for the right track retraction is greater than the PPC pressure value for the left track retraction, the exterior 3D model 3M0 is moved so as to curve backward in the left direction. The speed of movement and the magnitude of the curve may be changed depending on the PPC pressure values ​​and the difference between the PPC pressure values.

[0300] Note that by using position information in addition to the PPC pressures for the right track forward / reverse and the left track forward / reverse, it is possible to more accurately animate the travel of the work machine 33. In this case, by using position information, it is possible to more accurately represent the speed and position of the movement of the work machine 33. Furthermore, by playing back animation of the work machine 33 based on the roll angle, or the pitch angle, or both the roll angle and the pitch angle, in addition to the PPC pressures for the right track forward / reverse and the left track forward / reverse, it is possible to reproduce the left-right tilt of the work machine 33 or the front-to-back tilt of the work machine 33 while traveling.

[0301] Furthermore, in step 3S07, the image generation unit 31003 sequentially generates movement trajectory images showing the movement trajectory of a predetermined location of the work machine 33 (the cutting edge 3BK of the bucket 3BK) in association with the movement image based on the movement information (step 3S07b). Furthermore, the display control unit 31007 causes the display unit 3102 to display the movement image and the movement trajectory image superimposed on each other. Furthermore, in step 3S07, the image generation unit 31003 sequentially generates part images showing a predetermined location of the work machine 332B (the bucket 3BK) in association with each other (step 3S07c). Furthermore, in step 3S07c, the display control unit 31007 causes the display unit 3102 to display the movement image, movement trajectory image, and part image superimposed on each other. Also, in step 3S07, the display control unit 31007 combines a terrain image based on the three-dimensional point cloud information 3MAP with the action image, movement trajectory image, and part image, and displays the result on the display unit 3102 (step 3S07d). Figure 45 schematically shows a display image 3D1 displayed on the display unit 3102 by the display control unit 31007. The display image 3D1 includes an action image 3D11, a movement trajectory image 3D12, a part image 3D13, and a terrain image 3D14, all based on the designation log information 3TL1. Figure 46 schematically shows a display image 3D2 displayed on the display unit 3102 by the display control unit 31007. The display image 3D2 includes an action image 3D11, a movement trajectory image 3D12, a part image 3D13, and a terrain image 3D14 based on the specified log information 3TL1, as well as an action image 3D21, a movement trajectory image 3D22, and a part image 3D23 based on the comparison log information 3TL2.

[0302] The image generation unit 31003 determines whether to end animation playback while playing back the animation of the operations of the two work machines 33 (step 3S08). For example, the image generation unit 31003 determines to end animation playback when it receives an instruction to end playback based on pressing a stop button, etc. It may also determine to end animation playback after a predetermined period of time has elapsed since animation playback began. If animation playback has not ended (step 3S08; NO), the image generation unit 31003 continues simultaneous animation playback of the two work machine models 3TM. On the other hand, if animation playback is to end (step 3S08; YES), the image generation unit 31003 ends the animation playback process.

[0303] Of the processing flows described using Figure 39, steps 3S00, 3S01, 3S03, 3S04, 3S05, 3S06, and 3S08 are not essential components of the playback system 310, and other embodiments may not include such steps.

[0304] (Other Display Examples) FIG. 47 schematically illustrates a display image 3D3 displayed on the display unit 3102 by the display control unit 31007. The display image 3D3 includes an action image 3D31, a movement trajectory image 3D32, and a terrain image 3D34 based on log information 3TL1. The movement trajectory image 3D32 shown in FIG. 47 changes its display mode, for example, its display color, based on the work content. FIG. 48 schematically illustrates a display image 3D4 displayed on the display unit 3102 by the display control unit 31007. The display image 3D4 includes an action image 3D41, a movement trajectory image 3D42, and a terrain image 3D44 based on log information 3TL1. The movement trajectory image 3D42 shown in FIG. 48 changes its display mode, for example, its display color, based on the accuracy of the position information.

[0305] (Actions and Effects) As described above, the playback system 310 according to the fourth embodiment comprises an acquisition unit 31000 that acquires log information 3TL of the work machine 33 associated with time, an image generation unit 31003 that sequentially generates action images 3D11, 3D21 that represent the action of the work machine 33 by sequentially applying action information that represents the action of the work machine 33 based on the log information 3TL to a three-dimensional model of the work machine 33, and that sequentially generates movement trajectory images 3D12, 3D22 that represent the movement trajectory of a predetermined location of the work machine 33 in association with the action images, and a display control unit 31007 that superimposes the action images and the movement trajectory images and displays them on a predetermined display unit 3102. With this configuration, the flow of a series of actions can be easily grasped from the movement trajectory images, enabling more effective analysis of work performed by the work machine 33.

[0306] (Modification) The contents of the log information 3TL (FIGS. 40 to 42) according to the fourth embodiment are not limited to those in other embodiments. For example, if the work machine 33 is not a hydraulic excavator but a different vehicle type, log information 3TL according to that vehicle type is recorded. Examples of other vehicle types include a wheel loader, a bulldozer, etc.

[0307] Furthermore, the regeneration system 310 according to the fourth embodiment has been described as being installed at a location away from the work machine 33 and connected to the data logger 320 mounted on the work machine 33 via a wide area communication network, but other embodiments are not limited to this configuration.

[0308] For example, in a replay system 310 according to another embodiment, part or all of the configuration of the replay system 310 may be installed inside the work machine 33. In this case, the data logger 320 may transmit the log information 3TL to the replay system 310 via an internal network of the work machine 33, rather than via a wide area communication network. In this way, the operator on board the work machine 33 can check the movements of the work machine 33 that he or she is operating by playing back an animation on the spot. Furthermore, by playing back model movements of the work machine 33 for the operator of the work machine 33, this can be used as guidance.

[0309] The playback system 310 installed inside the work machine 33 may also acquire the log information 3TL of other work machines 33 via a wide area communication network, etc. In this way, it is possible to play back animations of the states of work machines 33 other than the work machine 33 on which the playback system 310 is installed.

[0310] Furthermore, the playback system 310 according to another embodiment may be installed in a location away from the work machine 33, and may transmit and display the video information generated by the animation playback process on a monitor mounted on the work machine 33.

[0311] Furthermore, in other embodiments, one aspect of the playback instruction received from the operator may be, for example, a playback period. For example, the playback period may be a playback start time and a playback end time. In this case, the playback system 310 performs playback of the work machine 33 for the received playback period. Furthermore, in other embodiments, it is not essential to specify a playback end time. For example, in other embodiments, a playback instruction from the operator may be a mode in which only a playback start time is received and playback is performed for a certain period of time from the playback start time, or playback may continue as long as log information exists, or playback may be stopped in response to the occurrence of various other events.

[0312] The acquired log information 3TL (FIGS. 40 to 42) does not need to be arranged in chronological order. In this case, the image generation unit 31003 can simply apply the information to be used for playback from the log information 3TL to the work machine model 3TM in chronological order.

[0313] The various processing steps of the playback system 310 described above are stored in the form of a program on a computer-readable recording medium, and the computer reads and executes this program to perform the various processing steps. Computer-readable recording media include magnetic disks, magneto-optical disks, CD-ROMs, DVD-ROMs, and semiconductor memories. Alternatively, the computer program may be distributed to a computer via a communication line, and the computer that receives the program may execute the program.

[0314] The program may be one that realizes part of the above-mentioned functions, or may be one that realizes the above-mentioned functions in combination with a program already recorded in the computer system, such as a so-called differential file or differential program.

[0315] Some or all of the functions of the playback system 310 described above may be assigned to the work machine 33. For example, some or all of the functions of the acquisition unit 31000, reception unit 31001, extraction unit 31002, image generation unit 31003, estimation unit 31004, determination unit 31005, synchronization unit 31006, display control unit 31007, designation unit 31008, memory 3101, display unit 3102, operation reception unit 3103, communication interface 3104, and storage 3105 may be assigned to the work machine 33.

[0316] Although several embodiments of the present disclosure have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the inventions described in the claims and their equivalents, as well as in the scope and spirit of the inventions.

[0317] (Additional Note) The display control system (playback system 310) according to the present disclosure can be understood, for example, as follows.

[0318] (10) A display control system (reproduction system 310) according to a tenth aspect comprises an acquisition unit 31000 that acquires log information 3TL of a work machine 33 associated with time, an image generation unit 31003 that sequentially generates operation images (3D11, 3D21, 3D31, 3D41) that represent the operation of the work machine by sequentially applying operation information that represents the operation of the work machine based on the log information to a three-dimensional model of the work machine (work machine model 3TM), and sequentially generates movement trajectory images (3D12, 3D22, 3D32, 3D42) that represent the movement trajectory of a predetermined location of the work machine in association with the operation images, based on the operation information, and a display control unit 31007 that superimposes the operation images and the movement trajectory images and displays them on a predetermined display unit 3102. According to this aspect and each of the following aspects, work performed by a work machine can be analyzed more effectively.

[0319] (11) A display control system (reproduction system 310) according to an eleventh aspect is a display control system according to (10), further comprising an estimation unit 31004 that estimates the work content of the work machine at each time based on the log information, and a designation unit 31008 that designates the work content for which the movement trajectory image is to be generated, and the image generation unit 31003 generates the movement trajectory image when the estimated work content corresponds to the designated work content.

[0320] (12) A display control system (playback system 310) according to a twelfth aspect is a display control system according to (10) or (11), in which the image generation unit 31003 generates a part image (3D13, 3D23) that represents a specific part of the work equipment provided on the work machine a specific time ago in correspondence with the operation image, and the display control unit 31007 superimposes the part image on the operation image and displays it on the display unit.

[0321] (13) A display control system (playback system 310) according to a thirteenth aspect is a display control system according to any one of (10) to (12), in which the predetermined location corresponds to the cutting edge of a work implement provided on the work machine.

[0322] (14) A display control system (reproduction system 310) according to a fourteenth aspect is a display control system according to any one of (10) to (12), in which the predetermined location corresponds to the ground contact position of a track or tire provided on the work machine.

[0323] (15) A display control system (reproduction system 310) according to a fifteenth aspect is a display control system according to any one of (10) to (14), wherein the image generation unit changes the appearance of the movement trajectory image based on at least one of the work content estimated based on the log information, the accuracy of the position information contained in the log information, information indicating whether or not automatic operation control is in place in the work machine contained in the log information, the amount of work of the work machine based on the log information, and information indicating the workload of the work machine based on the log information.

[0324] The playback system and playback method of the present disclosure make it possible to play back the movement of a work machine in a way that makes it easier to understand. The display control device and display control method of the present disclosure make it possible to analyze work performed by a work machine in more detail. The display control system and display control method of the present disclosure make it possible to analyze work performed by a work machine more effectively.

[0325] 11 Analysis support system, 110 Playback system, 1100 CPU, 11000 Acquisition unit, 11001 Reception unit, 11002 Extraction unit, 11003 Image generation unit, 11004 Estimation unit, 11005 Determination unit, 11006 Synchronization unit, 11007 Display control unit, 1101 Memory, 1102 Display unit, 1103 Operation reception unit, 1104 Communication interface, 1105 Storage, 120 Data logger, 13 Work machine, 1H1 to 1H2 Heat map, 1TM Work machine model, 1M1 Operation unit 3D model, 1TL Log information, 1PM1 Unit work prediction model, 1PM2 Element work prediction model, 1R Estimated work content, 1D11, 1D21, 1D31 Operation image 21 Analysis support system, 210 Reproduction system, 2100 CPU, 21000 acquisition unit, 21001 reception unit, 21002 extraction unit, 21003 reproduction unit, 21004 estimation unit, 21005 determination unit, 21006 synchronization unit, 21007 reproduction range designation unit, 21008 bird's-eye view image display unit, 2101 memory, 2102 display unit, 2103 operation reception unit, 2104 communication interface, 2105 storage, 220 data logger, 23 work machine, 2H1-2H2 heat map, 2TM work machine model, 2TL log information, 2PM1 unit work prediction model, 2PM2 element work prediction model, 2R estimated work content, 2D11, 2D21, 2D31 operation image, 2D12, 2D22, 2D32 captured image 31 Analysis support system, 310 playback system, 3100 CPU, 31000 acquisition unit, 31001 reception unit, 31002 extraction unit, 31003 image generation unit, 31004 estimation unit, 31005 determination unit, 31006 synchronization unit, 31007 display control unit, 31008 designation unit, 3101 memory, 3102 display unit, 3103 operation reception unit, 3104 communication interface, 3105 storage, 320 data logger, 33 work machine, 3H1-3H2 heat map, 3TM work machine model, 3TL log information, 3PM1 unit work prediction model, 3PM2 element work prediction model, 3R estimated work content, 3D11, 3D21, 3D31, 3D41 Movement image, 3D12, 3D22, 3D32, 3D42 Movement trajectory image, 3D13, 3D23 Part image

Claims

1. A playback system comprising: an acquisition unit that acquires log information of a work machine associated with a time; an image generation unit that generates operation images that represent the operation of the work machine by sequentially applying operation information that represents the operation of the work machine based on the log information to a three-dimensional model of the work machine; and a display control unit that displays the operation images on a specified display unit, wherein the three-dimensional model includes an operation unit model that represents an operation unit provided in a cab of the work machine, and the image generation unit generates the operation images by setting the three-dimensional model at a viewpoint facing the operation unit in the cab.

2. The playback system according to claim 1, wherein the operation image includes an image representing the operation unit.

3. The playback system according to claim 2, further comprising an estimation unit that estimates the work content of the work machine at each time based on the log information, and the display control unit further displays a work content image on the display unit, the work content image representing a time series of the estimated work content.

4. The playback system according to claim 3, wherein the display control unit further displays, on the display unit, an operation amount image showing a time series of values ​​corresponding to the operation amount of the operation unit contained in the log information.

5. The playback system according to claim 4, wherein the display control unit further displays, on the display unit, an instantaneous fuel consumption image representing a time series of the instantaneous fuel consumption included in the log information.

6. The reproduction system according to claim 5, wherein the image generation unit rotates a projection plane onto which the three-dimensional model is projected based on the roll angle of the work machine, and changes the height of the projection plane based on the pitch angle of the work machine.

7. A playback method comprising the steps of: acquiring log information of a work machine associated with time; generating a motion image representing the motion of the work machine by sequentially applying motion information representing the motion of the work machine based on the log information to a three-dimensional model of the work machine; and displaying the motion image on a specified display unit, wherein the three-dimensional model includes an operation unit model representing an operation unit provided in a cab of the work machine, and when generating the motion image, the three-dimensional model is set at a viewpoint facing the operation unit in the cab to generate the motion image.

8. A playback system comprising: an acquisition unit that acquires work machine log information associated with a time and video information representing a captured video image taken by the work machine and associated with a time; and a playback unit that sequentially plays back motion images representing the motion of the work machine by sequentially applying motion information representing the motion of the work machine based on the log information to a three-dimensional model of the work machine, and sequentially plays back captured images based on the video information in synchronization with the motion images.

9. The playback system of claim 8, wherein the playback unit sequentially plays back the motion images in a first display area of ​​a specified display unit, and sequentially plays back the captured images in a second display area different from the first display area of ​​the display unit in synchronization with the motion images.

10. The playback system according to claim 9, further comprising an overhead image display unit that generates an overhead image including a plurality of work machines located in the same work site area at the same time, and displays the overhead image including each of the work machines in a selectable form in a third display area different from the first display area and the second display area of ​​the display unit, wherein the acquisition unit acquires the log information and the video image information of the work machine selected in the overhead image.

11. A playback system as described in claim 9 or 10, wherein the acquisition unit acquires the log information and the video image information of a plurality of work machines, and the playback unit superimposes the operation images of the work machines and plays them sequentially in the first display area, and also arranges and plays back the captured images based on the video image information side by side in the second display area.

12. A playback method including the steps of: acquiring log information of a work machine associated with a time; and video information representing a captured video captured by the work machine and associated with a time; sequentially applying motion information representing the motion of the work machine based on the log information to a three-dimensional model of the work machine to sequentially play back motion images representing the motion of the work machine, and sequentially playing back captured images based on the video information in synchronization with the motion images.

13. A display control system comprising: an acquisition unit that acquires log information of a work machine associated with time; an image generation unit that sequentially generates motion images that represent the motion of the work machine by sequentially applying motion information that represents the motion of the work machine based on the log information to a three-dimensional model of the work machine, and sequentially generates motion trajectory images that represent the motion trajectory of a specified location of the work machine in association with the motion images, based on the motion information; and a display control unit that superimposes the motion images and the motion trajectory images and displays them on a specified display unit.

14. A display control system as described in claim 13, further comprising: an estimation unit that estimates work content of the work machine at each time based on the log information; and a designation unit that designates the work content for which the movement trajectory image is to be generated, wherein the image generation unit generates the movement trajectory image when the estimated work content corresponds to the designated work content.

15. A display control system as described in claim 14, wherein the image generation unit generates a part image that represents a specific part of a work implement equipped on the work machine from a specific time ago in correspondence with the operation image, and the display control unit superimposes the part image on the operation image and displays it on the display unit.

16. A display control system as described in claim 15, wherein the predetermined location corresponds to a cutting edge of a tool provided on the work machine.

17. The display control system according to claim 15, wherein the predetermined location corresponds to a ground contact position of a track or tire of the work machine.

18. A display control system as described in claim 16, wherein the image generation unit changes the appearance of the movement trajectory image based on at least one of the work content estimated based on the log information, the accuracy of the position information contained in the log information, information indicating the presence or absence of automatic operation control of the work machine contained in the log information, the workload of the work machine based on the log information, and information indicating the workload of the work machine based on the log information.

19. A display control method comprising: a step of acquiring log information of a work machine associated with time; a step of sequentially generating motion images representing the motion of the work machine by sequentially applying motion information representing the motion of the work machine based on the log information to a three-dimensional model of the work machine, and a step of sequentially generating motion trajectory images representing a motion trajectory of a predetermined location of the work machine in association with the motion images, based on the motion information; and a step of superimposing the motion images and the motion trajectory images and displaying them on a predetermined display unit.

Images