PLAYBACK SYSTEM, PLAYBACK METHOD, DISPLAY CONTROL SYSTEM AND DISPLAY CONTROL METHOD

Abstract

A playback system includes: a capture unit configured to capture log information of a working machine associated with a point in time; an image generation unit configured to generate an operating image representing the operation of the working machine by sequentially applying operating information representing the operation of the working machine, based on the log information, to a three-dimensional model of the working machine; and a display control unit configured to display the operating image on a predetermined display unit, wherein the three-dimensional model includes an operator unit model representing an operator unit provided in a cabin of the working machine, and the image generation unit generates the operating image using the three-dimensional model set to a viewpoint facing the operator unit in the cabin.

Description

TECHNICAL AREA

[0001] The present disclosure relates to a playback system, a playback method, a display control system, and a display control method. The present disclosure claims priority based on Japanese patent applications No. 2023-203133, 2023-203150, and 2023-203165, filed on November 30, 2023, the contents of which are incorporated herein by reference. STATE OF THE ART

[0002] Patent document 1 describes a playback device that enables the detection of the movements of two different construction machines (hereinafter referred to as working machines). The playback device described in patent document 1 reproduces the operation of the working machine by sequentially applying angular information of the working machine, which is contained in the working machine's protocol information, to a three-dimensional (3D) model of the working machine. In the playback device described in patent document 1, the playback of a 3D model based on first protocol information and the playback of a 3D model based on second protocol information are performed synchronously.In addition, the playback device described in patent document 1 reproduces an animation showing input operations on various control levers and drive levers by an operator of the working machine, using a two-dimensional control panel model that represents a change in the amount of input by movements in an up-down direction and a left-right direction that are orthogonal to each other or the like.

[0003] Patent document 1 describes a playback device that enables the detection of the movements of two different construction machines (hereinafter referred to as working machines). The playback device described in patent document 1 reproduces the operation of the working machine by sequentially applying angular information of the working machine, which is contained in the working machine's protocol information, to a three-dimensional (3D) model of the working machine. In the playback device described in patent document 1, the playback of a 3D model based on first protocol information and the playback of a 3D model based on second protocol information are performed synchronously.

[0004] Patent document 1 describes a playback device that enables the detection of the movements of two different construction machines (hereinafter referred to as working machines). The playback device described in patent document 1 reproduces the operation of the working machine by sequentially applying angular information of the working machine, which is contained in the working machine's protocol information, to a three-dimensional (3D) model of the working machine. In the playback device described in patent document 1, the playback of a 3D model based on first protocol information and the playback of a 3D model based on second protocol information are performed synchronously. LIST OF COUNTERPOINTS Patent Literature

[0005] Patent Document 1: JP 2020-183615 A BRIEF DESCRIPTION OF THE INVENTION Technical Problem

[0006] As described above, the animation depicting the input processes at the control levers and the drive levers is reproduced in the playback device described in patent document 1, using the two-dimensional control panel model. Further improvements are needed in the easily understandable representation of the machine's movement.

[0007] The present disclosure is made to solve the above problem, and one of its objectives is to provide a reproduction system and a reproduction method capable of reproducing a movement of a working machine in an easily understandable manner.

[0008] For example, when analyzing the work of the machine using the 3D model, it is necessary to check the loading state of an excavation target in a working tool such as a shovel that has excavated the target such as earth and sand, the condition of the environment such as the terrain in which the machine is located, and the like.

[0009] The present disclosure was made in view of the circumstances described above, and one of its purposes is to provide a reproduction system and a reproduction method that enable a detailed analysis of the work performed by a working machine.

[0010] In analyzing the work performed by a machine, there was a need to analyze the sequence of operations, such as excavation, rotation from an excavation position to a loading position, loading, and rotation from the loading position back to the excavation position. According to the playback device described in patent document 1, such a sequence of movements can be detected, for example, by repeatedly playing back an animation using a 3D model that depicts the sequence. Further improvements are needed for a more effective analysis of the work performed by a machine.

[0011] The present disclosure was made to solve the above problem, and one of its objectives is to provide a display control system and a display control method that enable a more effective analysis of work performed by a working machine. Solution to the problem

[0012] A playback system of the present disclosure includes: a capture unit configured to capture log information of a working machine associated with a point in time; an image generation unit configured to generate an operating image representing the operation of the working machine by sequentially applying operating information representing the operation of the working machine, based on the log information, to a three-dimensional model of the working machine;and a display control unit configured to display the operating image on a predetermined display unit, wherein the three-dimensional model includes an operator unit model representing an operator unit provided in a cabin of the working machine, and the image generation unit generates the operating image using the three-dimensional model set to a viewing point facing the operator unit in the cabin.

[0013] A reproduction method of the present disclosure includes: capturing log information of a working machine that is associated with a point in time; generating an operating image representing an operation of the working machine by sequentially applying operating information representing the operation of the working machine, based on the log information, to a three-dimensional model of the working machine; and displaying the operating image on a predetermined display unit, wherein the three-dimensional model includes an operator unit model representing an operator unit provided in a cabin of the working machine, and generating the operating image includes generating the operating image with the three-dimensional model set to a viewing point facing the operator unit in the cabin.

[0014] A playback system of the present disclosure includes: a capture unit configured to capture log information of a working machine associated with a point in time, and moving image information representing a captured moving image recorded by the working machine and associated with a point in time; and a playback unit configured to sequentially play back an operating image representing an operation of the working machine by sequentially applying operating information representing the operation of the working machine, based on the log information, to a three-dimensional model of the working machine, and sequentially playing back a captured image based on the moving image information in synchronization with the operating image.

[0015] A reproduction method of the present disclosure includes: capturing log information of a working machine that is associated with a point in time, and moving image information that represents a recorded moving image that is captured by the working machine and associated with a point in time; and sequentially reproducing an operating image that represents an operation of the working machine by sequentially applying operating information that represents the operation of the working machine, based on the log information, to a three-dimensional model of the working machine, and sequentially reproducing a recorded image based on the moving image information in synchronization with the operating image.

[0016] A display control system of the present disclosure includes: a capture unit configured to capture log information of a working machine associated with a point in time; an image generation unit configured to sequentially generate an operating image representing the operation of the working machine by sequentially applying operating information representing the operation of the working machine, based on the log information, to a three-dimensional model of the working machine, and sequentially generating a motion trajectory image representing the motion trajectory of a predetermined part of the working machine, in conjunction with the operating image based on the operating information; and a display control unit configured to display the operating image and the motion trajectory image superimposed on a predetermined display unit.

[0017] A display control method of the present disclosure includes: acquiring log information of a working machine that is associated with a point in time; sequentially generating an operating image representing an operation of the working machine by sequentially applying operating information representing the operation of the working machine, based on the log information, to a three-dimensional model of the working machine; and sequentially generating a motion trajectory image representing a motion trajectory of a predetermined part of the working machine, in conjunction with the operating image based on the operating information; and displaying the operating image and the motion trajectory image in a superimposed manner on a predetermined display unit. ADVANTAGEOUS EFFECTS OF THE INVENTION

[0018] According to the reproduction system and reproduction method of the present disclosure, it is possible to reproduce a movement of a working machine in an easily recognizable way.

[0019] According to the display control device and the display control method of the present disclosure, it is possible to analyze work performed by a working machine in more detail.

[0020] According to the display control system and the display control method of the present disclosure, it is possible to analyze work performed by a working machine more effectively. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a diagram illustrating an overall configuration of an analysis support system according to a first embodiment. Fig. Figure 2 is a diagram illustrating the structure of a working machine according to the first embodiment. Fig. Figure 3 is a diagram illustrating a configuration of a cabin of the working machine according to the first embodiment. Fig. Figure 4 is a diagram illustrating a functional configuration of a playback system according to the first embodiment. Fig. Figure 5 is a diagram illustrating a processing flow through the playback system according to the first embodiment. Fig. Figure 6 is a first diagram illustrating an example of protocol information according to the first embodiment. Fig. Figure 7 is a second diagram illustrating an example of protocol information according to the first embodiment. Fig. Figure 8 is a third diagram illustrating an example of protocol information according to the first embodiment. Fig. Figure 9 is a diagram illustrating a heatmap used to estimate work content according to the first embodiment. Fig. Figure 10 is a diagram illustrating an example of a working machine model according to the first embodiment. Fig. Figure 11 is a diagram illustrating an example of a relationship between an operator unit 3D model and a projection plane according to the first embodiment. Fig. Figure 12 is a diagram illustrating an example of a relationship between a display plane of a display unit and the projection plane according to the first embodiment. Fig. Figure 13 is a diagram illustrating an example of a relationship between the display plane of the display unit and the projection plane according to the first embodiment. Fig. Figure 14 is a diagram illustrating an example of a display image according to the first embodiment. Fig. Figure 15 is a diagram illustrating an example of a display image according to the first embodiment. Fig. Figure 16 is a diagram illustrating an example of a display image according to the first embodiment. Fig. Figure 17 is a diagram illustrating an example of a display image according to the first embodiment. Fig. Figure 18 is a diagram illustrating an example of a display image according to the first embodiment. Fig. Figure 19 is a diagram illustrating an overall configuration of an analysis support system according to a second embodiment. Fig. Figure 20 is a diagram illustrating the structure of a working machine according to the second embodiment. Fig. Figure 21 is a diagram illustrating a configuration of a cabin of the working machine according to the second embodiment. Fig. Figure 22 is a diagram illustrating a functional configuration of a playback system according to the second embodiment. Fig. Figure 23 is a diagram illustrating a process of processing by the playback system according to the second embodiment. Fig. Figure 24 is a first diagram illustrating an example of protocol information according to the second embodiment. Fig. Figure 25 is a second diagram illustrating an example of protocol information according to the second embodiment. Fig. Figure 26 is a third diagram illustrating an example of protocol information according to the second embodiment. Fig. Figure 27 is a diagram illustrating a heatmap used to estimate work content according to the second embodiment. Fig. Figure 28 is a diagram illustrating an example of a working machine model according to the second embodiment. Fig. Figure 29 is a diagram illustrating an example of a display image according to the second embodiment. Fig. Figure 30 is a diagram illustrating an example of a display image according to the second embodiment. Fig. Figure 31 is a diagram illustrating an example of a display image according to the second embodiment. Fig. Figure 32 is a diagram illustrating a functional configuration of a playback system according to a third embodiment. Fig. Figure 33 is a diagram illustrating a processing flow through the playback system according to the third embodiment. Fig. Figure 34 is a diagram illustrating an example of a display image according to the third embodiment. Fig. Figure 35 is a diagram illustrating an overall configuration of an analysis support system according to a fourth embodiment. Fig. Figure 36 is a diagram illustrating the structure of a working machine according to the fourth embodiment. Fig. Figure 37 is a diagram illustrating a configuration of a cabin of the working machine according to the fourth embodiment. Fig. Figure 38 is a diagram illustrating a functional configuration of a playback system according to the fourth embodiment. Fig. Figure 39 is a diagram illustrating a process of processing by the playback system according to the fourth embodiment. Fig. Figure 40 is a first diagram illustrating an example of protocol information according to the fourth embodiment. Fig. Figure 41 is a second diagram illustrating an example of protocol information according to the fourth embodiment. Fig. Figure 42 is a third diagram illustrating an example of protocol information according to the fourth embodiment. Fig. Figure 43 is a diagram illustrating a heatmap used to estimate work content according to the fourth embodiment. Fig. Figure 44 is a diagram illustrating an example of a working machine model according to the fourth embodiment. Fig. Figure 45 is a diagram illustrating an example of a display image according to the fourth embodiment. Fig. Figure 46 is a diagram illustrating an example of a display image according to the fourth embodiment. Fig. Figure 47 is a diagram illustrating an example of a display image according to the fourth embodiment. Fig. Figure 48 is a diagram illustrating an example of a display image according to the fourth embodiment. DESCRIPTION OF EXECUTION FORMS First embodiment

[0021] Below, a display control device and a display control method according to a first embodiment are described with reference to Fig. 1, Fig. 2, Fig. 3, Fig. 4, Fig. 5, Fig. 6, Fig. 7, Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12, Fig. 13, Fig. 14, Fig. 15, Fig. 16, Fig. 17 to Fig. 18 described in detail. Overall configuration of the analysis support system

[0022] Fig. Figure 1 is a diagram illustrating an overall configuration of an analysis support system according to the first embodiment.

[0023] An analysis support system 11 includes a playback system 110 and a data logger 120, which are installed in each of a variety of working machines 13.

[0024] The work machines 3 are the targets of the work analysis by the playback system 110. Examples of the work machines 13 include a hydraulic excavator, a wheel loader, a bulldozer, and the like. In the following description, a hydraulic excavator is described as an example of the work machines 13. Each of the work machines 13 is equipped with a variety of sensors. The data logger 120 records and collects information representing the state of the work machine 13 as detected by the sensors in time series. The information representing the state of the work machine 13 at each point in time, recorded by the data logger 120, is subsequently referred to as log information.In a case where an operating mechanism is configured to control the working machine 13 using an electrical control signal, information from the working machine 3's control signal can be recorded and collected in time series and included in the log information. The data logger 120 transmits the recorded log information to the playback system 110 via a wide area communication network at a predetermined time interval. The predetermined time interval is, for example, five minutes. The playback system 110 records the log information received from the data logger 120 onto a recording medium. The function of the playback system 110 is described below. Structure of the working machine

[0025] Fig. Figure 2 is a diagram illustrating the structure of a working machine according to the first embodiment.

[0026] The work machine 13, which is a hydraulic excavator, digs and levels earth and sand or the like at a construction site or similar location. As in Fig. As illustrated in Figure 2, the working machine 13, which is a hydraulic excavator, includes a lower undercarriage 131 for travel and an upper rotating body 132, which is rotatably installed in an upper section of the lower undercarriage 131. Additionally, the upper rotating body 132 is equipped with a cabin 132A, a working attachment 132B, and two GNSS (Global Navigation Satellite System) antennas 1G1 and 1G2.

[0027] The lower chassis 131 has a left track 1CL and a right track 1CR. The working machine 13 moves forward, turns, and moves backward by the rotation of the left track 1CL and the right track 1CR.

[0028] Cabin 132A is a location where an operator of the work machine 13 enters and operates the work machine 13. Cabin 132A is installed, for example, on a left side section of a front end section of the upper rotary body 132. The internal configuration of cabin 132A is described below.

[0029] The working device 132B includes a boom 1BM, an arm 1AR, and a bucket 1BK. The boom 1BM is attached to a front end section of the upper rotating body 132. The arm 1AR is attached to the boom 1BM. The bucket 1BK is attached to the arm 1AR. A boom cylinder 1SL1 is located between the upper rotating body 132 and the boom 1BM. By actuating the boom cylinder 1SL1, the boom 1BM can be actuated relative to the upper rotating body 132. An arm cylinder 1SL2 is located between the boom 1BM and the arm 1AR. By actuating the arm cylinder 1SL2, the arm 1AR can be actuated relative to the boom 1BM. A bucket cylinder 1SL3 is located between the arm 1AR and the bucket 1BK. By driving the bucket cylinder 1SL3, the bucket 1BK can be actuated in relation to the arm 1AR.

[0030] The upper rotating body 132, the boom 1BM, the arm 1AR and the bucket 1BK, which have been described above in the working machine 13, which is a hydraulic excavator, are aspects of moving parts of the working machine 13.

[0031] The shovel 1BK serves as a configuration example of a "working tool" according to the present disclosure. The shovel 1BK includes a cutting edge 1BKT, which is used for excavation work or the like. Cabin configuration

[0032] Fig. Figure 3 is a diagram illustrating a configuration of cabin 132A of the working machine according to the first embodiment.

[0033] As in Fig. As illustrated in Figure 3, the cabin 132A is equipped with control levers 1L1 and 1L2, foot pedals 1F1 and 1F2, and drive levers 1R1 and 1R2. The control levers 1L1 and 1L2, the foot pedals 1F1 and 1F2, and the drive levers 1R1 and 1R2 are enclosed in a control unit 132AO for operating the moving parts of the working machine 13.

[0034] The control lever 1L1 and the control lever 1L2 are located to the left and right of each seat 1ST in cabin 132A. The foot pedal 1F1 and the foot pedal 1F2 are located on the floor surface in front of seat 1ST in cabin 132A.

[0035] An example of an operating pattern that specifies a correspondence relationship between an input operation on the control levers 1L1 and 1L2, the travel levers 1R1 and 1R2 and the operation of the working machine 13, which is a hydraulic excavator, is as follows.

[0036] The control lever 1L1, located on the left side relative to the forward direction of the cabin, is an operating mechanism for implementing a rotary movement of the upper rotating body 132 and an excavation / dumping operation of the arm 1AR. Specifically, when the operator of the machine 13 tilts the control lever 1L1 forward, the arm 1AR performs the dumping operation. Furthermore, when the operator of the machine 13 tilts the control lever 1L1 backward, the arm 1AR performs the excavation operation. Additionally, the upper rotating body 132 rotates clockwise when the operator of the machine 13 tilts the control lever 1L1 to the right. Furthermore, the upper rotating body 132 rotates counterclockwise when the operator of the machine 13 tilts the control lever 1L1 to the left.The upper rotating body 132 can rotate clockwise and counterclockwise in response to tilting the control lever 1L1 in the forward-backward direction, and the arm 1AR can perform the dumping or excavation operation in response to tilting the control lever 1L1 in the left-right direction.

[0037] The control lever 1L2, located on the right side relative to the forward direction of the cab, is an operating mechanism for performing the digging / dumping operation of the bucket 1BK and a raising / lowering operation of the boom 1BM. Specifically, when the operator of the machine 13 tilts the control lever 1L2 forward, the boom 1BM is lowered. When the operator of the machine 13 tilts the control lever 1L2 backward, the boom 1BM is raised. Furthermore, when the operator of the machine 13 tilts the control lever 1L2 to the right, the bucket 1BK is dumped. Finally, when the operator of the machine 13 tilts the control lever 1L2 to the left, the bucket 1BK is excavated.

[0038] The drive levers 1R1 and 1R2 are operating mechanisms for controlling the operation of the lower undercarriage 131, that is, for controlling the movement of the work machine 13. The drive lever 1R1, located on the left side in the forward direction of the cab, corresponds to the rotary drive of the left track 1CL of the lower undercarriage 131. Specifically, when the operator of the work machine 13 tilts the drive lever 1R1 forward, the left track 1CL rotates in the forward direction. When the operator of the work machine 13 tilts the drive lever 1R1 backward, the left track 1CL rotates in the reverse direction.

[0039] The drive lever 1R2, located on the right side of the cab in the forward direction, corresponds to the rotary drive of the right track 1CR of the lower undercarriage 131. Specifically, when the operator of the machine 13 tilts the drive lever 1R2 forward, the right track 1CR rotates in the forward direction. When the operator of the machine 13 tilts the drive lever 1R2 backward, the right track 1CR rotates in the reverse direction. The foot pedals 1F1 and 1F2 are each interlocked with the drive levers 1R1 and 1R2, and the machine can also be driven using the foot pedals 1F1 and 1F2.

[0040] The operating patterns described above are merely an example, and the present invention is not limited to the above modes, which depend on the type of hydraulic excavator or the like.

[0041] It should be noted that the reference to Fig. 2 described working machine 3 may not include the GNSS antennas 1G1 and 1G2 depending on the embodiment. Functional configuration of the playback system

[0042] Fig. Figure 4 is a diagram illustrating a functional configuration of a playback system according to the first embodiment.

[0043] Functions of the playback system 110 according to the first embodiment are described below with reference to Fig. 4 described.

[0044] As in Fig. As illustrated in Figure 4, the playback system 110 includes a CPU 1100, a memory 1101, a display unit 1102, an operator control unit 1103, a communication interface 1104, and a data storage unit 1105. The CPU (central processing unit) 1100 can be a processor such as an FPGA or a GPU instead of a CPU.

[0045] The CPU 1100 is a processor that controls the overall operation of the Playback System 110. Various functions of the CPU 1100 will be described later.

[0046] Memory 1101 is a so-called main memory device. Instructions and data required for the operation of the CPU 1100 based on a program are loaded into memory 1101.

[0047] The display unit 1102 is a display device capable of displaying information in a visible manner, and is, for example, a liquid crystal display, an organic EL display, or the like.

[0048] The operating receiver unit 1103 is an input device and is, for example, a general mouse, keyboard, touch sensor or the like.

[0049] The communication interface 1104 is a communication interface for communication with the data logger 120.

[0050] The data storage device 1105 is an auxiliary storage device and is, for example, a hard disk drive (HDD), a solid-state drive (SSD), or the like. The data storage device 1105 records protocol information 1TL received from the data logger 120, the vehicle type of the work machine 13, a work machine model 1TM, which is a pre-prepared 3D model for each model, and the like. The work machine model 1TM is described below.

[0051] In addition, the data storage device 1105 also records a unit work prediction model 1PM1 and an element work prediction model 1PM2, which are trained machine learning models used to estimate the work content of the work machine 13, a heat map (1H1 and 1H2) generated during the estimation process, the estimated work content R of the work machine 13, three-dimensional point group information 1MAP representing the terrain information of a construction site, and the like. It should be noted that the unit work prediction model 1PM1, the element work prediction model 1PM2, and the heat map (1H1, 1H2) are described further below.The three-dimensional point group information 1MAP includes, for example, three-dimensional point group information measured at a work site using, for example, a drone or the like, information obtained by sequentially updating terrain information from bucket edge coordinate information of the working machine 13 in operation, and the like, and is recorded in the data storage 1105.

[0052] The functions of the CPU 1100 of the playback system 110 according to the first embodiment are described in detail. The CPU 1100 operates based on a predetermined program to perform functions as a capture unit 11000, a receiver unit 11001, an extraction unit 11002, an image generation unit 11003, an estimation unit 11004, a determination unit 11005, a synchronization unit 11006, and a display control unit 11007.

[0053] It should be noted that the predetermined program can serve to implement some of the functions to be performed by the playback system 110. For example, the program can be used to perform a function in combination with other programs already stored in the data memory 1105 or in combination with other programs implemented in other devices. In other embodiments, the playback system 110 can additionally or instead of the above configuration include a user-defined highly integrated circuit (LSI) such as a programmable logic device (PLD).

[0054] Examples of PLDs include programmable array logic (PAL), generic array logic (GAL), complex programmable logic device (CPLD), and field-programmable gate array (FPGA). In this case, some or all of the functions implemented by the processor can be realized by the integrated circuit.

[0055] The acquisition unit 11000 captures the log information 1TL to be reproduced from a multitude of individual log information 1TL recorded and collected in the data storage 1105. It is assumed that the multitude of individual log information 1TL in the data storage 1105 are recorded for respective files with different filenames. The acquisition unit 11000 captures either a single log information 1TL to be analyzed or two individual log information 1TL to be compared, for example, according to an operator's instruction. The following is an example of a case in which the acquisition unit 11000 captures two individual log information 1TLs for comparison.One of the two log pieces of information captured by the capture unit 11000 is a log piece of information designated by the operator as the playback target using a file name or similar identifier. Hereinafter, this log piece of information is referred to as designated log piece 1TL1 (first log piece). The other of the two log pieces of information is a log piece of information that is played back simultaneously with the playback based on designated log piece 1TL1. Hereinafter, this log piece of information is referred to as compared log piece 1TL2 (second log piece).The compared log information 1TL2 can be a log entry pre-selected as a "model case" of the work content for each of various work content scenarios and conditions, based on factors such as good fuel efficiency, short operating time, expert driving, and other considerations. Alternatively, the compared log information 1TL2 can be, for example, a past log entry recorded during an operation previously performed by the same operator as the designated log information 1TL1. In this case, comparing the two log entries can be used, for example, to analyze the operator's skill level.

[0056] The acquisition unit 11000 according to the present embodiment automatically searches for the compared protocol information 1TL2, which is suitable for comparison with the designated protocol information 1TL1, and acquires it based on a determination result by the determination unit 11005 described below. However, the acquisition unit 11000 according to another embodiment is not limited to the above mode, and, for example, the acquisition unit 11000 can acquire protocol information designated by the operator using a file name or the like as the compared protocol information 1TL2. Additionally, if only a single compared protocol information 1TL2 is recorded, the acquisition unit 11000 can acquire the single compared protocol information 1TL2.

[0057] The receiving unit 11001 receives a predetermined playback instruction from the operator of the playback system 110. For example, the receiving unit 11001 receives a playback instruction for the work machine 13 from the operator of the playback system 110.

[0058] The extraction unit 11002 extracts angular information, which is used for playback for the working machine 13, from the captured log information 1TL. The angular information is an example of “operational information (motion information)” according to the present disclosure.

[0059] The image generation unit 11003 generates an operating image representing the operation of the machine 13 by sequentially applying operating information, based on the protocol information 1TL, to the machine model 1TM, which is a three-dimensional model of the machine 13. As described below, in the present embodiment, the machine model 1TM includes an operator unit 3D model 1M1 (an example of the operator unit model of the present disclosure), which represents the operator unit 132AO located in the cabin 132A of the machine 13. The image generation unit 11003 sets a viewpoint 1PT to project the machine model 1TM onto a two-dimensional projection plane at a position facing the operator unit 132AO in the cabin 132A and generates the operating image.The image generation unit 11003 sets the working machine model 1TM at the viewing point which is facing the control unit 132AO in the cabin 132A, and generates the operating image.

[0060] The estimation unit 11004 estimates the work content of the work machine 13 at each point in time from the recorded protocol information 1TL.

[0061] The determination unit 11005 determines whether the compared protocol information 1TL2 is suitable for comparison with the designated protocol information 1TL1, based on the information included in the designated protocol information 1TL1.

[0062] The synchronization unit 11006 performs processing to synchronize the playback of the work machine model 1TM based on the designated protocol information 1TL1 and the playback of the work machine model 1TM based on the compared protocol information 1TL2. Specifically, the synchronization unit 11006 specifies the playback start time on the time series of the animation of the work machine model 1TM based on the work content estimated by the estimation unit 11004.

[0063] The display control unit 11007 displays the operating image generated by the image generation unit 11003 on the display unit 1102. The display control unit 11007 can also display a work content image, representing a time series of the estimated work content, on the display unit 1102. The display control unit 11007 can also display an operator input image on the display unit 1102, representing a time series of a value corresponding to the operator input of the operator unit 132AO included in the protocol information 1TL. The display control unit 11007 can also display an instantaneous fuel efficiency image on the display unit 1102, representing a time series of the instantaneous fuel efficiency values ​​included in the protocol information 1TL.The display control unit 11007 can display an image (terrain image) on the display unit 1102, which represents terrain based on the three-dimensional point group information 1MAP, for example in combination with the operating image. Processing sequence by the playback system

[0064] The following describes a specific processing sequence performed by the playback system 110, with reference to Fig. 5, Fig. 6, Fig. 7, Fig. 8, Fig. 9, Fig. 10, Fig. 11, Fig. 12, Fig. 13, Fig. 14, Fig. 15, Fig. 16, Fig. 17 to Fig. 18 described in detail.

[0065] The process of the in Fig. The processing illustrated in Figure 5 begins when the operator of playback system 110 starts a dedicated application.

[0066] When the operator performs an operation to start the dedicated application, the capture unit 11000 of the CPU 1100 loads the protocol information 1TL1, designated as the playback target, into memory 1101 and captures it (step 1S00).

[0067] Here, the protocol information 1TL (designated protocol information 1TL1, compared protocol information 1TL2) is presented with reference to Fig. 6, Fig. 7 to Fig. 8 described.

[0068] As in Fig. 6, Fig. 7 to Fig. As illustrated in Figure 8, the protocol information 1TL includes work machine identification information. The work machine identification information is specifically an individual identification number for the individual identification of work machine 13. Fig. 6, Fig. 7 to Fig. 8 It is assumed that the work machine identification information is assigned such that it corresponds to the vehicle type, model, type number, machine number, and the like of the work machine 13, indicating a hydraulic excavator, wheel loader, bulldozer, or the like. The work machine identification information may be a number or may be an alphabet, a symbol, a combination thereof, or the like.

[0069] As in Fig. As illustrated in Figure 6, the log information 1TL includes information representing the position and orientation of the machine 13 at any given time, as well as angular information of the moving parts of the machine 13. Specifically, the position of the machine 13, the roll angle of the machine 13 (the tilt of the machine body in the left-right direction), the pitch angle (the tilt of the machine body in the forward-backward direction), the rotation angle, the boom angle, the arm angle, and the blade angle are recorded in the log information 1TL at any given time. Here, the data logger 120 installed in the machine 13 specifies and records the position of the machine 13 based on position measurement information representing latitude, longitude, and altitude, which is information obtained, for example, by reception through the GNSS antennas 1G1 and 1G2.The data logger 120 calculates and records the roll angle and pitch angle of the working machine 13 based on a measurement result from an inertial measurement unit (IMU) installed in the working machine 13. The data logger 120 calculates and records the rotation angle of the upper rotating body 132 based on the position measurement information obtained from each of the GNSS antennas 1G1 and 1G2 provided in the upper rotating body 132. Furthermore, the data logger 120 calculates and records the boom angle, arm angle, and blade angle based on the extension and retraction degrees of each of the boom cylinder 1SL1, arm cylinder 1SL2, and blade cylinder 1SL3. The IMU can be mounted on the boom, arm, blade, and upper rotating body to be used, for example, to acquire the boom angle, arm angle, blade angle, and rotation angle.

[0070] The position, roll angle, and pitch angle are information required to specify the position and orientation of the working machine 13 itself. Therefore, for example, in an embodiment where only the movements of the moving parts of the working machine 13—that is, the upper rotary body 132, the boom 1BM, the arm 1AR, and the bucket 1BK—are represented as an animation, and the position and orientation of the working machine 13 itself are not represented, the position, roll angle, and pitch angle information need not be included in the protocol information.

[0071] The in Fig. The protocol information illustrated in Figure 6 1TL corresponds to “operating information” as defined in this disclosure. The operating information includes, for example, information describing the operation of the moving parts of the working machine 13, such as the working device 132B, the upper rotary body 132, and the lower drive body 131, which are operated using the operating unit 132AO of the working machine 13.

[0072] Alternatively, the ones in Fig. The protocol information illustrated in Figure 6 (TL) represents the operation of the working machine 13 as it can be visually observed from outside the working machine 13 and can also be referred to as external view operating information. That is, in the present embodiment, the operating information or external view operating information includes the position information, roll angle, pitch angle, yaw angle, boom angle, arm angle, and bucket angle of the working machine 13. In the present embodiment, the operating information (or external view operating information), as information used in estimating the work content, has the characteristic that, unlike the drive mechanism information and operator information described below, it is less likely to be affected by a difference in the vehicle rank of the working machine 13.

[0073] As in Fig. As illustrated in Figure 7, the log information 1TL includes pilot pressures (proportional pressure control pressures (PPC pressures)) which indicate the degrees of input to the control levers 1L1 and 1L2 and the like by the operator at any given time, that is, the degrees of lever tilt and the degrees of pedal pressure. Specifically, the PPC pressures of the control levers 1L1 and 1L2, the travel levers 1R1 and 1R2, or the foot pedals 1F1 and 1F2, corresponding to the respective operating modes of counterclockwise / clockwise rotation, arm extension / deployment, boom raising / lowering, bucket extension / deployment, forward / reverse movement of the right track, and forward / reverse movement of the left track by the operator, are recorded in the log information 1TL at any given time. The data in Figure 7 is further detailed in the log information 1TL. Fig. The 7 illustrated time points correspond to those in Fig. 6 illustrated time points. In the present embodiment, each in Fig. 7. Illustrated information is also referred to as operating information.

[0074] As in Fig. As illustrated in Figure 8, the log information 1TL includes information representing the status of the main drive mechanisms, such as the engine and hydraulic pumps of the working machine 13, at any given time. Specifically, the log information 1TL records the engine coolant temperature, engine power, instantaneous fuel consumption, and hydraulic pump oil temperature at any given time. The information in Fig. The eight illustrated elements are merely examples, and elements such as engine coolant temperature, hydraulic pump oil temperature, or similar parameters can be omitted. Alternatively, the fuel throttle opening and other factors influencing fuel efficiency can be included. As yet another example, in the case of an electrically powered construction machine, the information could include elements such as current power consumption and remaining battery charge. The information in Fig. The 8 illustrated time points correspond to those in Fig. 6 and Fig. 7 illustrated time points. In the present embodiment, each in Fig. 8. Illustrated information is also referred to as drive mechanism information.

[0075] Returning to Fig. 5 The estimation unit 11004 of the CPU 1100 estimates the work content of the work machine 13 at each time based on the designated protocol information 1TL1 acquired in step 1S00 (step 1S01).

[0076] Here, a procedure is described in which the estimation unit 11004 estimates the work content of the working machine 13 from the protocol information 1TL, with reference to Fig. 9 described. The estimation unit 11004 estimates the work content of the working machine 13 for both unit work and element work. Unit work is work performed to achieve a specific purpose. Element work is work that specifies a series of processes or tasks, which are elements that constitute unit work and are classified according to purpose.

[0077] Examples of unit work sections include "excavation and loading", "plowing", "sloping (from below)", "load gathering", "driving" and "stopping", which are in Fig. 9 are illustrated, as well as “trench excavation”, “backfilling”, “sloping (from above)” and the like.

[0078] "Excavation and loading" refers to work involving the excavation and removal of earth and sand or rock, and the loading of this material onto the platform of a transport vehicle. "Excavation and loading" encompasses all work performed as a single unit, including excavation, turning the loaded vehicle, unloading, turning the unloaded vehicle, waiting for unloading, and compacting the loading platform.

[0079] "Plowing" is the work of removing excess soil relief to a predetermined height. Plowing is a unit operation that includes digging and unloading, or digging, turning with a load, unloading, and turning without a load, and may include push leveling and sweeping.

[0080] "Below slope" is the work of creating a slope by the work machine 13, which is located below the target location. Below slope is a unitary work that includes compaction, excavation, loaded turning, unloading, and unloaded turning, and may include thrust grading.

[0081] "Load collection" refers to work involving the gathering of soil and sand resulting from excavation or similar processes before loading it onto a transport vehicle. Load collection is a single, unified task that includes excavation, turning the loaded vehicle, unloading, turning the unloaded vehicle, and may also include grading.

[0082] "Driving" refers to work in which the working machine 13 is moved. Driving as a unit of work is unit work that includes driving as an element of work.

[0083] "Stopping" is a state of halting for a predetermined time or longer, without any soil, sand, or rock in the shovel. Stopping as a unit of work is unit of work that includes stopping as an element of work.

[0084] "Trench excavation" is the work of digging and scraping the soil into a long and narrow trench. Trench excavation is a single operation that includes excavation, turning with a load, unloading, turning without a load, and may also include grading.

[0085] "Backfilling" is the process of filling a trench or hole already formed in the ground with soil and sand, and backfilling to level the surface. Backfilling is a single operation that includes excavation, turning with a load, unloading, compaction, and turning without a load, and may include grading and sweeping.

[0086] "Sloping" (from above) is the work of creating a slope by the work machine 13, which is located above the target location. Sloping (from above) is a unitary work that includes compaction, excavation, loaded turning, unloading and unloaded turning, and may include thrust grading.

[0087] Examples of element work sections include "excavation", "loaded turning", "waiting for unloading", "unloading", "unloaded turning", "loading platform pressing", which are in Fig. 9 are illustrated, as well as “compacting”, “thrust leveling”, “sweeping” and the like.

[0088] The term "excavation" refers to work in which earth, sand or rock is dug out and removed using the 1BK shovel.

[0089] “Loaded rotation” refers to work in which the upper rotating body 132 is rotated while the removed earth and sand or rock is held in the bucket 1BK.

[0090] “Waiting for unloading” refers to work where one waits for a transport vehicle to be loaded, while holding the removed earth and sand or rock in the bucket 1BK.

[0091] "Unloading" refers to work in which the removed earth and sand or rock from bucket 1BK is unloaded onto a transport vehicle or to a predetermined location.

[0092] “Unloaded turning” refers to work in which the upper rotating body 132 is turned without soil, sand or rock in the bucket 1BK.

[0093] The term "loading platform pressing" refers to work in which earth and sand loaded onto a loading platform of a transport vehicle are pressed from above with the 1BK bucket to smooth the earth and sand.

[0094] "Compacting" refers to work in which soil and sand are pushed into the disturbed ground with a shovel to shape and solidify the soil.

[0095] "Thrust leveling" refers to work in which earth and sand are pushed and leveled with the ground surface of the shovel 1BK.

[0096] The term "sweeping" refers to work in which earth and sand are swept and leveled using a side surface of the shovel 1BK.

[0097] The estimation unit 11004 obtains a time series of probabilities regarding unit work when the protocol information 1TL is fed into the unit work prediction model 1PM1 in chronological order. The unit work prediction model 1PM1 is a model that outputs the probability regarding unit work when the protocol information 1TL is fed into it during training using, for example, teacher data, and can be stored, for example, in data storage 1105.

[0098] The estimation unit 11004 obtains a time series of probabilities regarding element work when the protocol information 1TL is fed into the element work prediction model 1PM2 in chronological order. The element work prediction model 1PM2 is a model that outputs the probability with respect to element work when the protocol information 1TL is fed into it during training using, for example, teacher data, and can be stored, for example, in data storage 1105.

[0099] The estimation unit 11004 smooths the time series of probabilities by applying a time averaging filter to the time series of probabilities with respect to unit work and the time series of probabilities with respect to element work, and generates the unit work heatmap 1H1, which represents the smoothed time series of probabilities with respect to unit work, and the element work heatmap 1H2, which represents the smoothed time series of probabilities with respect to element work, as shown in Fig. Figure 9 illustrates this. Heatmaps 1H1 and 1H2 are maps in which colors representing the probabilities of job classifications are applied to a plane, with the vertical axis representing the job classifications and the horizontal axis representing the time based on the smoothed time series of probabilities. For example, the colors corresponding to the heatmaps might be closer to blue for a lower probability of job classification and closer to red for a higher probability. The estimation unit 11004 stores heatmaps 1H1 and 1H2 in data memory 1105.

[0100] Estimation unit 11004 specifies a period in which the probability of unit work is dominant, based on the smoothed time series of probabilities, and estimates the work content of machine 13 during that period. For example, during a period in which the probability of "excavation and loading," which is unit work, is dominant, the work content of machine 13 is estimated as "excavation and loading." Similarly, estimation unit 11004 specifies a period in which the probability of element work is dominant, based on the smoothed time series of probabilities, and estimates the work content of machine 13 during that period. For example, during a period in which the probability of "excavation," which is element work, is dominant, the work content of machine 13 is estimated as "excavation."The estimation unit 11004 stores information about the estimated work content 1R for the working machine 13 in the data storage unit 1105.

[0101] Returning to Fig. Step 5 then determines whether the acquisition unit 11000 should display information to be compared with the designated protocol information 1TL1, referred to as the playback target (compared protocol information 1TL2), based on a predetermined instruction from the operator (step 1S02A). If the information is to be displayed (step 1S02A: YES), processing proceeds to step 1S04 after processing in step 1S02B and step 1S03 has been completed. If the information is not to be displayed (step 1S02A: NO), processing proceeds to step 1S04 without executing steps 1S02B and 1S03. If the information is to be displayed (step 1S02A: YES), the acquisition unit 11000 captures the compared protocol information 1TL2, which is suitable for comparison with the designated protocol information 1TL1 captured in step 1S00 (step 1S02B).

[0102] At this point, the determination unit 11005 performs a determination processing sequence to determine whether the multitude of individual compared protocol information 1TL2, recorded in the data store 1105, are suitable for comparison with the designated protocol information 1TL1 acquired in step 1S00. In this determination processing sequence, the determination unit 11005 first selects one of the multitude of individual compared protocol information 1TL2 that is pre-recorded in the data store 1105.Then the determination unit 11005 determines whether, for example, the vehicle rank, the amount of change in the rotation angle (the extent of the swivel), the bucket height at the time of waiting to be unloaded by the working machine 13, and the like are substantially the same, based on the log information 1TL, and if it determines that they are not substantially the same, it selects a new single compared log information 1TL2 and performs the same determination processing as described above.

[0103] In addition, according to another embodiment, the determination unit 11005 can further limit the candidates for reproduction based on a work position, a working time, a work classification or the like, in addition to the determination processing described above.

[0104] Returning to Fig. 5 Next, the synchronization unit 11006 performs processing to synchronize the animation playback for the work machine model 1TM based on the designated protocol information 1TL1 and the animation playback for the work machine model 1TM based on the compared protocol information 1TL2 (step 1S03). Specifically, the synchronization unit 11006 specifies the playback start time on the time series in the animation of the work machine model 1TM based on the designated protocol information 1TL1 and the playback start time on the time series in the animation of the work machine model 1TM based on the compared protocol information 1TL2.

[0105] One method for specifying the playback start time to synchronize the two is as follows. That is, the synchronization unit 11006 extracts the unit work toggling time in the designated log information 1TL1 based on the unit work heatmap 1H1 for the designated log information 1TL1, generated by the estimating unit 11004. Next, the synchronization unit 11006 extracts the unit work toggling time in the compared log information 1TL2 based on the unit work heatmap 1H1 for the compared log information 1TL2, generated by the estimating unit 11004. The synchronization unit 11006 selects one of the extracted times and specifies the selected time as the playback start time of each animation.

[0106] The playback system 110 according to another embodiment is not limited to the mode described above. The playback system 110 according to another embodiment can compare a combination of angular information of the boom, arm, bucket, and the like in the compared protocol information 1TL2 with a combination of angular information of the boom, arm, bucket, and the like in the designated protocol information 1TL1, extract a time point at which both angles are close to each other, and specify that time point, for example, as the playback start time of each animation.

[0107] After step 1S03, or if “NO” is determined in step 1S02A, the receiving unit 11001 receives a playback instruction from the operator (step 1S04).

[0108] One aspect of the playback instruction can be an action such as pressing a play button. Additionally, the playback instruction can include information that serves as the starting point for playback, such as a time, the position of the machine 13, and various events such as the occurrence of an abnormality in the machine 13.

[0109] Next, the acquisition unit 11000 selects the work machine model 1TM corresponding to the referenced work machine identification information from the data storage 1105 and reads it, based on the work machine identification information, as the type of work machine 13 received by the receiving unit 11001 (step 1S05).

[0110] Here, the work machine model 1TM is described with reference to Fig. 10 described.

[0111] As in Fig. As illustrated in Figure 10, the machine model 1TM is information that includes the machine identification information and an external shape 3D model 1M0 of the machine 13 specified by the machine identification information, and the operator unit 3D model 1M1, which is the three-dimensional model of the operator unit 132AO. The external shape 3D model 1M0 is a 3D model representing the machine 13 and is constructed for each part of the machine 13, such as the lower chassis and the upper rotating body. For example, the external shape 3D model 1M0 represents the shape of the machine 13.For example, the outer shape 3D model 1M0 includes an outer shape model 1M01 of the lower carriage, which represents the lower carriage 131 of the working machine 13, an outer shape model 1M02 of the upper rotating body, which represents the upper rotating body 132, a boom outer shape model 1M03, which represents the boom 1BM, an arm outer shape model 1M04, which represents the arm 1AR, and a bucket outer shape model 1M05, which represents the bucket 1BK.

[0112] The control unit 3D model 1M1 is a 3D model representing the control unit 132AO and is a three-dimensional form representing the cabin 132A including the control unit 132AO, the seat 1ST in the cabin 132A, and the control unit 132AO. Among these are the models 1M11, 1M12, 1M13, 1M14, and the like, representing the control levers 1L1 and 1L2 and the drive levers 1R1 and 1R2, and the like, which are enclosed in the control unit 132AO. These models change their inclinations with respect to the viewing point 1PT when the three-dimensional model is projected onto a two-dimensional projection plane 1PS, a straight line, or a plane passing through the viewing point 1PT, based on the protocol information 1TL. The inclinations of cabin 132A, seat 1ST, and the like do not change with respect to the viewing point 1PT or a straight line or plane passing through the viewing point 1PT. Fig. In Figure 10, the projection plane 1PS is a two-dimensional plane for projecting a three-dimensional solid shape onto the two-dimensional plane. The viewing point 1PT is a position from which the three-dimensional stereoscopic shape projected onto the projection plane 1PS is viewed. In the present embodiment, the viewing point 1PT is set at a position facing the control unit 132AO in the cabin 132A. The viewing point 1PT can, for example, be set at a position corresponding to the eye position of the operator sitting in the seat 1ST. A vector 1PTD represents a direction that serves as a reference when viewing the projection plane 1PS from the viewing point 1PT and is, for example, a front direction.

[0113] Returning to Fig. 5. The extraction unit 11002 extracts information used for playback from each of the designated log information 1TL1 and the compared log information 1TL2, or from the designated log information 1TL1 alone (step 1S06). For example, the extraction unit 11002 extracts various types of angular information, such as boom angle, arm angle, and bucket angle, as well as PPC pressures of the control levers 1L1 and 1L2, the travel levers 1R1 and 1R2, or the foot pedals 1F1 and 1F2, as information used for playback. Additionally, in steps 1S00 and 1S02B, only information used for playback can be captured. The following describes a processing sequence by taking as an example a case where playback is performed using the designated log information 1TL1 and the compared log information 1TL2, which are compared with each other.If no playback is performed based on the compared protocol information 1TL2, the processing regarding the compared protocol information 1TL2 can generally be omitted.

[0114] Next, the image generation unit 11003 and the display control unit 11007 perform a simultaneous playback of an animation of the work machine model 1TM based on the designated protocol information 1TL1 and an animation playback of the work machine model 1TM based on the compared protocol information 1TL2 (step 1S07). Here, the image generation unit 11003 generates an operating image representing the operation of the work machine 13, while various types of information recorded in the designated protocol information 1TL1 are applied to the work machine model 1TM in chronological order of the timestamp, starting from the playback start time specified in the synchronous processing in step 1S03 (step 1S07a).Simultaneously with the generation of the operating image for the animation, the image generation unit 11003 generates an operating image representing the operation of the working machine 13, while various types of information recorded in the compared protocol information 1TL2 are applied to the working machine model 1TM in chronological order of the timestamps from the playback start time specified in step 1S03 of the synchronous processing (step 1S07a). In step 1S07a, the display control unit 11007 displays the operating image generated by the image generation unit 11003 on the display unit 1102.

[0115] The following section describes in detail the contents of the operational image generation processing by the image generation unit 11003 in step 1S07a. The image generation unit 11003 modifies the angle of the corresponding section of the outer shape 3D model 1M0 based on various types of angular information, such as the rotation angle and the boom angle, specified in the log information 1TL (the designated log information 1TL1 and the compared log information 1TL2). For example, the image generation unit 11003 represents the position and orientation of the bucket 1BK of the working machine 13 by tilting the bucket outer shape model 1M05 about the axis of rotation defined at a position of the connection with the arm outer shape model 1M04 to the bucket angle specified in the log information 1TL.

[0116] Similarly, the image generation unit 11003 reproduces the position and orientation of the arm 1AR of the working machine 13 by tilting the arm outer shape model 1M04 about the axis of rotation defined at a position of the connection with the boom outer shape model 1M03 to the arm angle specified in the protocol information 1TL.

[0117] Likewise, the image generation unit 11003 reproduces the position and orientation of the upper rotary body 132 of the working machine 13 by tilting the outer shape model 1M02 of the upper rotary body about the axis of rotation defined at a position of the connection with the outer shape model 1M01 of the lower driving body to the angle of rotation specified in the protocol information TL.

[0118] Similarly, the image generation unit 11003 reproduces the position of the upper rotary body 132 of the working machine 13 by tilting the outer shape model 1M01 of the lower carriage about the roll rotation axis defined in the outer shape model 1M01 of the lower carriage to the roll angle specified in the protocol information 1TL and tilting the outer shape model 1M01 of the lower carriage about the pitch rotation axis defined in the outer shape model 1M01 of the lower carriage to the pitch angle specified in the protocol information 1TL.

[0119] Furthermore, the playback system 110 according to the first embodiment can reproduce an animation of the driving movement of the working machine 13 based on the PPC presses of the forward / reverse movement of the right crawler track and the forward / reverse movement of the left crawler track at any time included in the log information 1TL.

[0120] Specifically, the outer shape 3D model 1M0 is moved forward, backward, forward left and right, and backward left and right based on the PPC pressures of the forward / backward movement of the right and left track. For example, the outer shape 3D model 1M0 moves forward based on the numerical values ​​of the PPC pressures of the forward movement of the right and left track. The movement speed can be changed based on the numerical values ​​of the PPC pressures.

[0121] The outer shape 3D model 1M0 is moved backward based on the numerical values ​​of the PPC pressures for the reverse movement of the right and left track. Furthermore, based on the difference in the numerical values ​​of the PPC pressures between the forward movement of the right and left track, the outer shape 3D model 1M0 is moved to curve in the forward left-right direction. For example, if the numerical value of the PPC pressure for the forward movement of the right track is greater than the numerical value of the PPC pressure for the forward movement of the left track, a movement to curve forward to the left is implemented. The speed of movement and the size of the curve can be changed depending on the numerical values ​​of the respective PPC pressures and the difference between these values.

[0122] Similarly, based on the difference in the numerical values ​​of the PPC pressures between the reverse movement of the right and left track, the outer shape 3D model 1M0 is moved to curve in the reverse left-right direction. For example, if the numerical value of the PPC pressure for the reverse movement of the right track is greater than the numerical value of the PPC pressure for the reverse movement of the left track, a movement to curve in the reverse left direction is implemented. The speed of movement and the size of the curve can be changed depending on the numerical values ​​of the respective PPC pressures and the difference between these values.

[0123] The movement of work machine 3 can be animated more precisely by playing back the animation using position information in addition to the PPC inputs for the forward / reverse movement of the right and left tracks. In this case, the speed and position of work machine 3's movement can be expressed more accurately using the position information. Additionally, by playing back the animation of work machine 13 based on the roll angle, pitch angle, or both, in addition to the PPC inputs for the forward / reverse movement of the right and left tracks, the tilt of work machine 13 in the left-right direction or the tilt of work machine 3 in the forward-backward direction during movement can be displayed.

[0124] The image generation unit 11003 changes the angle of the corresponding section of the outer shape 3D model 1M0 based on the PPC pressures specified in the protocol information 1TL (the designated protocol information 1TL1 and the compared protocol information 1TL2). For example, the image generation unit 11003 reproduces the position and orientation of the blade 1BK of the working machine 13 by tilting the blade outer shape model 1M05 about the axis of rotation defined at a position of the connection with the arm outer shape model 1M04 to the blade angle specified in the protocol information 1TL.

[0125] Similarly, the image generation unit 11003 reproduces the position and orientation of the arm 1AR of the working machine 13 by tilting the arm outer shape model 1M04 about the axis of rotation defined at a position of the connection with the boom outer shape model 1M03 to the arm angle specified in the protocol information 1TL.

[0126] Similarly, the image generation unit 11003 reproduces the position and orientation of the upper rotary body 132 of the working machine 13 by tilting the outer shape model 1M02 of the upper rotary body about the axis of rotation defined at a position of the connection with the outer shape model 1M01 of the lower driving body to the angle of rotation specified in the protocol information 1TL.

[0127] Similarly, the image generation unit 11003 reproduces the position of the upper rotary body 132 of the working machine 13 by tilting the outer shape model 1M01 of the lower carriage about the roll rotation axis defined in the outer shape model 1M01 of the lower carriage to the roll angle specified in the protocol information 1TL and tilting the outer shape model 1M01 of the lower carriage about the pitch rotation axis defined in the outer shape model 1M01 of the lower carriage to the pitch angle specified in the protocol information 1TL.

[0128] Similarly, the image generation unit 11003 changes the inclinations of model 1M11, representing control lever 1L1, model 1M12, representing control lever 1L2, model 1M13, representing drive lever 1R1, and model 1M14, representing drive lever 1R2, based on the PPC pushes.

[0129] As in Fig. As illustrated in Figure 11, the image-generating unit 11003 can rotate the angle of the projection plane 1PS in the direction of an arrow 1RA based on the roll angle and change the position (height) of the projection plane 1PS based on the pitch angle. That is, for example, as in Fig. As illustrated in Figure 12, the angular relationship with the projection plane 1PS to a plane 1PS1, a plane 1PS2, or the like can be changed based on the roll angle, with an area 1DS on a display screen of the display unit 1102 serving as a reference. Alternatively, for example, as shown in Fig. Figure 13 illustrates how the positional relationship (height) with the projection plane 1PS to a plane 1PS3, a plane 1PS4, or the like can be changed based on the pitch angle, with the area 1DS on the display screen of the display unit 1102 serving as a reference. In this way, by changing the angle and position of the projection plane 1PS based on the roll angle and the pitch angle, it is possible to reflect the changes in the roll angle and the pitch angle, which represent the changes in the position of the working machine 13 from an external viewpoint, on the two-dimensional image generated using the viewpoint 1PT inside the driver's seat 132A of the working machine 13.In this case, for example, the occurrence of vibrations or the like, which change the roll angle or pitch angle of the working machine 13 according to the operation of the control unit 132AO, can be reflected as a change in, for example, the operating image, the terrain image, or the like, which models 1M11 to 1M14 represent. The image generation unit 11003 can be configured to select whether the roll angle and pitch angle used when changing the angle and position of the projection plane 1PS are set, for example, based on values ​​based on the designated protocol information 1TL1 or values ​​based on the compared protocol information 1TL2 according to a predetermined operation by the operator at the time of image reproduction. According to this configuration, it is possible, for example, to compare the degree of vibration or the like by replaying images while switching the selection.

[0130] In step 1S07, the display control unit 11007 displays a terrain image on the display unit 1102 based on the three-dimensional point group information 1MAP in combination with the operating image (step 1S07b).

[0131] In step 1S07, the display control unit 11007 can display a supplementary image on the display unit 1102, which represents a time series of different types of information included in the work content estimate result or the log information 1TL (step 1S07c).

[0132] While the animation of the operation of two work machines 13 is playing, the image generation unit 11003 determines whether the animation playback should be terminated (step 1S08). For example, if a command to terminate playback is received based on the pressing of a stop button or the like, it is determined that the animation playback should be terminated. It can also be determined that the animation playback should be terminated after a predetermined time period has elapsed since the start of the animation playback. If the animation playback is not to be terminated (step 1S08; NO), the image generation unit 11003 continues the simultaneous animation playback for two work machine models 1TM. On the other hand, if the animation playback is to be terminated (step 1S08; YES), the image generation unit 11003 terminates the animation playback processing.

[0133] In the reference to Fig. In the processing sequence described in Section 5, steps 1S00, 1S01, 1S03, 1S04, 1S05, 1S06 and 1S08 are not the essential configuration of the playback system 110, and other embodiments may not include these steps. Display image of the playback system

[0134] Fig. 14, Fig. 15, Fig. 16, Fig. 17 to Fig. Figure 18 are diagrams illustrating an example of a display image of the display unit 1102 according to the first embodiment. Fig. 14, Fig. 15, Fig. 16 to Fig. Figure 17 illustrates examples of display images based on the designated protocol information 1TL1 and the compared protocol information 1TL2 of the playback system according to the first embodiment. Fig. Figure 18 illustrates an example of a display image based on the designated protocol information 1TL1 of the playback system according to the first embodiment. A display image 1D2, which is in Fig. As illustrated in Figure 15, an operating image 1D21 in a state in which a certain amount of time has elapsed since an operating image 1D11, which is enclosed in a display image 1D1, is shown in Figure 15. Fig. Figure 14 illustrates this. Operating image 1D11 and operating image 1D21 include images depicting models 1M11, 1M12, 1M03, and the like, and a terrain image 1D13 is combined. In the Fig. In the illustrated operating example 5, operating screen 1D11 and operating screen 1D21 are displayed in step 1S7a, and the terrain image 1D13 is displayed in step 1S7b. Additionally, buttons 1B1 and 1B2 for selecting the protocol information 1TL, which serves as a reference, are displayed in operating screen 1D11 and operating screen 1D21. Fig. Figure 14 illustrates a state in which the 1B1 key is selected, and in this case an image is generated with the roll angle and pitch angle set based on the designated protocol information 1TL1. Fig. Figure 15 illustrates a state in which button 1B2 is selected, and in this case, an image is generated showing the roll angle and pitch angle set based on the compared protocol information 1TL2. In the Fig. 14 and Fig. In 15 illustrated examples, changes in roll angle and pitch angle are represented, for instance, as changes in the slope or position of the terrain image D13. The state when the designated log information 1TL1 is recorded and the state when the compared log information 1TL2 is recorded can be compared by toggling the selection state of buttons 1B1 and 1B2. The image indicated by the solid line is an image based on the designated log information 1TL1. The image indicated by the dashed line is an image based on the compared log information 1TL2.

[0135] The display images 1D1 and 1D2 include supplementary images 1D12 and 1D22, which are generated in step 1S7c. Fig. 16 illustrates this in Fig. 14 illustrated supplementary image 1D12. As in Fig. As illustrated in Figure 16, the supplementary image 1D12 includes a work content image 1D121, which represents a time series of the work content estimated based on the labeled log information 1TL1, a work content image 1D122, which represents a time series of the work content estimated based on the compared log information 1TL2, a playback time symbol 1D129, and a scroll bar 1D120.

[0136] Supplementary image 1D12 includes an operator value image 1D123 and an operator value image 1D124, which represent a time series of the value corresponding to the operator value of the operating unit 132AO included in the log information 1TL. The solid line represents the operator value based on the designated log information 1TL1, and the dashed line represents the operator value based on the compared log information 1TL2. Operator value image 1D123 represents the boom lift lever pressure. Operator value image 1D124 represents the boom extension lever pressure.

[0137] Fig. Figure 17 illustrates an example of the supplementary image 1D12 as a result of scrolling the scroll bar 1D120, which is located in Fig. Figure 16 illustrates this, downwards. The supplementary image 1D12, which is in Fig. Figure 18 illustrates this, including operator image screens 1D125, 1D126, and 1D128, and an instantaneous fuel efficiency image 1D127, which represents a time series of the instantaneous fuel efficiency included in the log information 1TL. Operator image 1D125 represents the boom lift lever pressure. Operator image 1D126 represents the boom raise lever pressure. Operator image 1D128 represents the bucket dump lever pressure. The content of the supplementary image 1D12, which is shown in Fig. 16 and Fig. Figure 17 illustrates the included time series information, which is one example, and the content is not limited to this example. For instance, some of the information in Figure 17 may be included in Figure 17. Fig. 16 and Fig. The 17 illustrated elements have been omitted.

[0138] A display image 1D1A, which is in Fig. Figure 18 illustrates the operating image 1D11 and the supplementary image 1D12 based on the designated protocol information 1TL1. Operation and impacts

[0139] As described above, the playback system 110 according to the first embodiment includes the acquisition unit 11000, which acquires the protocol information 1TL of the machine 13 that is assigned to a specific time; the image generation unit 11003, which generates the operating image representing the operation of the machine 13 by sequentially applying the operating information representing the operation of the machine 13, based on the protocol information 1TL, to the machine model 1TM, which is the three-dimensional model of the machine 13; and the display control unit 11007, which displays the operating image on the display unit 1102. The machine model 1TM includes the operator unit 3D model 1M1, which represents the operator unit 132AO provided in the cabin 132A of the machine 13.The image generation unit 11003 positions the machine model 1TM at the viewing point facing the control unit 132AO in the cabin 132A and generates the operating image. According to this configuration, the movement of the control unit 132AO can be displayed, for example, with reference to the operator's viewing point. Therefore, according to the present embodiment, the movement of the machine 13 can be displayed in an easily understandable manner.

[0140] Furthermore, in the display system 110 according to the first embodiment, since the operating image includes the image representing the control unit 132AO, an analysis and comparison of the lever operation can be easily performed, for example. The display system 110 can display a supplementary image on the display unit 1102, which includes a work content image representing a time series of work content, an operating amount image representing a time series of the operating amount of the control unit, an image of the instantaneous fuel efficiency representing a time series of the instantaneous fuel efficiency, and the like, together with the operating image. According to this configuration, it is possible, for example, to capture the temporal changes in the operation or the like of the control unit 132AO in more detail for each work element than in the case without the supplementary image.Examples of temporal changes include whether the lever operation includes not only full opening and full closing, but also operation with a medium operating amount for fine adjustment, the sequence in which each lever is operated for each operating item, and the like. Modified example

[0141] The content of the protocol information 1TL according to the first embodiment ( Fig. 6, Fig. 7 to Fig. 8) is not limited to this in other embodiments. For example, if the working machine 13 is not a hydraulic excavator but another type of vehicle, log information 1TL is recorded according to the vehicle type. The other vehicle types are, for example, a wheel loader, a bulldozer, and the like.

[0142] Furthermore, the description is given under the assumption that the playback system 110 according to the first embodiment is installed at a location remote from the working machine 13 and is connected to the data logger 120 installed in the working machine 13 via the wide area communication network; however, other embodiments are not limited to this aspect.

[0143] For example, in another embodiment, part or all of the configuration of the playback system 110 can be installed inside the machine 13. In this case, the data logger 120 can transmit the log information 1TL to the playback system 110 via a network inside the machine 13 or the like, without using the wide area communication network. In this way, the operator entering the machine 13 can verify the movement of the machine 13 they are operating on-site by replaying the animation. Furthermore, the movement of the machine 13 can be replayed as a model case for the operator of the machine 13 to use as a guide.

[0144] The playback system 110 installed in the work machine 13 can acquire the protocol information 1TL of another work machine 13 via a wide area communication network or the like. In this way, the states of the work machines 13, except for the work machine 13 in which the playback system 110 is installed, can be displayed as an animation.

[0145] In addition, according to another embodiment, the playback system 110 can be installed at a location remote from the working machine 13 and can transmit video information generated by the animation playback processing to a monitor installed in the working machine 13 for display.

[0146] In another embodiment, one aspect of the playback instruction received from the operator may be, for example, a playback period. For instance, the playback period may include a playback start time and a playback end time. In this case, the playback system 110 plays back the work machine 13 during the received playback period. In another embodiment, determining the playback end time is not essential. For example, in another embodiment, only the playback start time may be received by the operator as a playback instruction, and playback may continue for a certain period of time from the playback start time, playback may continue as long as the log information is available, or playback may be stopped in accordance with the occurrence of other various events.

[0147] The protocol information to be recorded 1TL ( Fig. 6, Fig. 7 to Fig. 8) do not need to be arranged in chronological order. In this case, the image generation unit 11003 can apply information from the protocol information 1TL, which is used for playback, to the working machine model 1TM in time series.

[0148] The steps of the various types of processing of the playback system 110, as described above, are stored on a computer-readable recording medium in the form of a program, and the computer reads and executes the program to implement the various types of processing described above. The computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Furthermore, this computer program can be distributed to the computer via a communication circuit, and the computer receiving this distribution can execute the program.

[0149] The program described above can be a program for executing a section of the functions described above. Furthermore, the functions described above can be implemented in combination with a program already recorded on the computer system, namely a so-called differential file, a differential program, or the like.

[0150] Some or all functions of the playback system 110 described above can be assigned to the working machine 13. For example, some or all functions of the acquisition unit 11000, the receiving unit 11001, the extraction unit 11002, the image generation unit 11003, the estimation unit 11004, the determination unit 11005, the synchronization unit 11006, the display control unit 11007, the memory 1101, the display unit 1102, the operating receiving unit 1103, the communication interface 1104 and the data storage 1105 can be assigned to the working machine 13.

[0151] Certain embodiments of the present disclosure have been described above, but all embodiments are merely illustrative and are not intended to limit the scope of protection of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the core of the invention. These embodiments and modifications are included in the scope of protection and core of the invention and are also included in the scope of protection of the invention described in the claims and their equivalents. Additional remarks

[0152] The reproduction system 110 according to the present disclosure is understood, for example, as follows. (1) A playback system 110 according to a first viewpoint includes: the acquisition unit 11000, which is configured to acquire the protocol information 1TL of the working machine 13, which is associated with a point in time; the image generation unit 11003, which is configured to produce an operating image representing an operation of the working machine by sequentially applying operating information representing the operation of the working machine, based on the protocol information, to a three-dimensional model (working machine model 1TM) of the working machine;and the display control unit 11007, which is configured to display the operating image on the predetermined display unit 1102, wherein the three-dimensional model includes an operator unit model (operator unit 3D model 1M1) representing the operator unit 132AO provided in the cabin 132A of the working machine, and the image generation unit generates the operating image with the three-dimensional model set to the viewing point 1PT in order to project the three-dimensional model onto the two-dimensional projection plane 1PS facing the operator unit in the cabin. According to the present and the following considerations, the movement of the working machine 13 can be represented in an easily understandable manner. (2) A playback system 110 according to a second point of view is the playback system according to (1) wherein the operating image includes an image that represents the control unit. (3) A playback system 110 according to a third consideration is the playback system according to (1) or (2), which further includes the estimation unit 11004, which is configured to estimate work content of the working machine at each point in time based on the log information, and the display control unit further displays on the display unit the work content image 1D121, which represents a time series of the estimated work content. (4) A playback system 110 according to a fourth consideration is the playback system according to (1) to (3), wherein the display control unit further displays on the display unit an operator amount image representing a time series of a value corresponding to an operator amount of the operator unit included in the protocol information. (5) A playback system 110 according to a fifth consideration is the playback system according to (1) to (4), wherein the display control unit further displays on the display unit an image of an instantaneous fuel efficiency which represents a time series of the instantaneous fuel efficiency included in the log information. (6) A playback system 110 according to a sixth consideration is the playback system according to (1) to (5), wherein the image-generating unit rotates the projection plane based on a roll angle of the working machine and changes a height of the projection plane based on a pitch angle of the working machine. Second embodiment

[0153] A playback system and a playback method according to a second embodiment are described below with reference to Fig. 19, Fig. 20, Fig. 21, Fig. 22, Fig. 23, Fig. 24, Fig. 25, Fig. 26, Fig. 27, Fig. 28, Fig. 29 to Fig. 30 described in detail. Overall configuration of the analysis support system

[0154] Fig. Figure 19 is a diagram illustrating an overall configuration of an analysis support system according to the second embodiment.

[0155] An analysis support system 21 includes a playback system 210, a data logger 220 installed in each of a variety of working machines 23, and an image acquisition device 2CAM.

[0156] The work machines 23 are the targets of the work analysis by the playback system 210. Examples of the work machines 23 include a hydraulic excavator, a wheel loader, a bulldozer, and the like. In the following description, a hydraulic excavator is described as an example of the work machines 23. Each of the work machines 23 is equipped with a variety of sensors. The data logger 220 records and collects information in time series that represents the state of the work machine 23 as detected by the sensors. The information representing the state of the work machine 23 at each time point, recorded by the data logger 220, is subsequently referred to as log information. The log information is associated with a specific time point.In a case where an operating mechanism for the working machine 23 is configured to operate the working machine 23 using an electrical control signal, information from the working machine 23's control signal can be recorded and collected in time series and included in the log information. The data logger 220 transmits the recorded log information to the playback system 210 via a wide area communication network at a predetermined time interval. The predetermined time interval is, for example, five minutes. The playback system 210 records the log information received from the data logger 220 onto a recording medium. The image capture device 2CAM, for example, captures a moving image of the outside of the working machine 23 in the forward direction.The image capture device 2CAM can store moving image information representing the captured moving image in a storage device inside or outside the image capture device 2CAM. The moving image information is associated with time-time information (point in time). The image capture device 2CAM transmits the captured moving image information to the playback system 210 at a predetermined time interval via the wide area communication network or distributes the captured moving image information to the playback system 210 substantially in real time. The playback system 210 records the moving image information received from the image capture device CAM onto a recording medium or plays back the moving image information substantially in real time. The following is an example of a case in which the playback system 210 temporarily stores the moving image information.The function of the playback system 210 is described below. Structure of the working machine

[0157] Fig. Figure 20 is a diagram illustrating the structure of a working machine according to the second embodiment.

[0158] The work machine 23, which is a hydraulic excavator, excavates and levels earth, sand, or similar materials at a construction site or similar location. As in Fig. As illustrated in Figure 20, the working machine 23, which is a hydraulic excavator, includes a lower undercarriage 231 for travel and an upper rotating body 232, which is rotatably installed in an upper section of the lower undercarriage 231. Additionally, the upper rotating body 232 is equipped with a cabin 232A, a working device 232B, and two antennas 2G1 and 2G2 of a global navigation satellite system (GNSS).

[0159] The lower chassis 231 has a left track 2CL and a right track 2CR. The working machine 23 moves forward, turns, and moves backward by the rotation of the left track 2CL and the right track 2CR.

[0160] Cabin 232A is a location where an operator of the machine 23 enters and operates it. Cabin 232A is installed, for example, on a left side section of a front end section of the upper rotary body 232. Furthermore, the image acquisition device 2CAM for capturing a moving image in the forward direction of the machine 23 is mounted inside cabin 232A. However, the mounting position of the image acquisition device 2CAM on the machine 23 and the number of image acquisition devices 2CAM are not limited to this example. The configuration of the operating unit inside cabin 232A is described below.

[0161] The working device 232B includes a boom 2BM, an arm 2AR, and a bucket 2BK. The boom 2BM is attached to a front end section of the upper rotating body 232. The arm 2AR is attached to the boom 2BM. The bucket 2BK is attached to the arm 2AR. A boom cylinder 2SL1 is located between the upper rotating body 232 and the boom 2BM. By actuating the boom cylinder 2SL1, the boom 2BM can be actuated relative to the upper rotating body 232. The arm cylinder 2SL2 is located between the boom 2BM and the arm 2AR. By actuating the arm cylinder 2SL2, the arm 2AR can be actuated relative to the boom 2BM. A bucket cylinder 2SL3 is located between the arm 2AR and the bucket 2BK. By driving the bucket cylinder 2SL3, the bucket 2BK can be actuated in relation to the arm 2AR.

[0162] The upper rotating body 232, the boom 2BM, the arm 2AR and the bucket 2BK, which are described above in the working machine 23, which is a hydraulic excavator, are aspects of moving parts of the working machine 23.

[0163] It should be noted that in the present embodiment, the shovel 2BK serves as a configuration example of a “working tool” according to the present disclosure. The shovel 2BK includes a cutting edge 2BKT, which is used for digging or the like. Cabin configuration

[0164] Fig. Figure 21 is a diagram illustrating a configuration of cabin 232A of the working machine according to the second embodiment.

[0165] As in Fig. As illustrated in Figure 21, the cabin 232A is equipped with control levers 2L1 and 2L2, foot pedals 2F1 and 2F2, and drive levers 2R1 and 2R2. The control levers 2L1 and 2L2, the foot pedals 2F1 and 2F2, and the drive levers 2R1 and 2R2 are enclosed in a control unit 232AO for operating the moving parts of the working machine 23.

[0166] The control lever 2L1 and the control lever 2L2 are located to the left and right of seat 2ST in cabin 232A, respectively. The foot pedal 2F1 and the foot pedal 2F2 are located on the floor surface in front of seat 2ST in cabin 232A.

[0167] An example of an operating pattern that shows a correspondence relationship between an input operation on the control levers 2L1 and 2L2, the travel levers 2R1 and 2R2 and the operation of the working machine 23, which is a hydraulic excavator, is as follows.

[0168] The control lever 2L1, located on the left side relative to the forward direction of the cabin, is an operating mechanism for implementing a rotary movement of the upper rotating body 232 and an excavation / dumping operation of the arm 2AR. Specifically, when the operator of the machine 23 tilts the control lever 2L1 forward, the arm 2AR performs the dumping operation. When the operator of the machine 23 tilts the control lever 2L1 backward, the arm 2AR performs the excavation operation. Furthermore, the upper rotating body 232 rotates clockwise when the operator of the machine 23 tilts the control lever 2L1 to the right. Additionally, the upper rotating body 232 rotates counterclockwise when the operator of the machine 23 tilts the control lever 2L1 to the left.The upper rotating body 232 can rotate clockwise and counterclockwise in response to tilting the control lever 2L1 in the forward-backward direction, and the arm 2AR can perform the dumping or excavation operation in response to tilting the control lever 2L1 in the left-right direction.

[0169] The control lever 2L2, located on the right side relative to the forward direction of the cab, is an operating mechanism for performing the digging / dumping operation of the bucket 2BK and a raising / lowering operation of the boom 2BM. Specifically, when the operator of the machine 23 tilts the control lever 2L2 forward, the boom 2BM is lowered. When the operator of the machine 23 tilts the control lever 2L2 backward, the boom 2BM is raised. Furthermore, when the operator of the machine 23 tilts the control lever 2L2 to the right, the bucket 2BK is dumped. Additionally, when the operator of the machine 23 tilts the control lever 2L2 to the left, the bucket 2BK is excavated.

[0170] The drive levers 2R1 and 2R2 are operating mechanisms for controlling the operation of the lower undercarriage 231, that is, for controlling the movement of the work machine 23. The drive lever 2R1, located on the left side in the forward direction of the cab, corresponds to the rotary drive of the left track 2CL of the lower undercarriage 231. Specifically, when the operator of the work machine 23 tilts the drive lever 2R1 forward, the left track 2CL rotates in the forward direction. When the operator of the work machine 23 tilts the drive lever 2R1 backward, the left track 2CL rotates in the reverse direction.

[0171] The drive lever 2R2, located on the right side of the cab in the forward direction, corresponds to the rotary drive of the right track 2CR of the lower undercarriage 231. Specifically, when the operator of the machine 23 tilts the drive lever 2R2 forward, the right track 2CR rotates in the forward direction. When the operator of the machine 23 tilts the drive lever 2R2 backward, the right track 2CR rotates in the reverse direction. The foot pedals 2F1 and 2F2 are each locked to the drive levers 2R1 and 2R2, and the machine can also be driven using the foot pedals 2F1 and 2F2.

[0172] The operating patterns described above are merely an example, and the present invention is not limited to the above modes, which depend on the type of hydraulic excavator or the like.

[0173] It should be noted that the information relating to Fig. The working machine described in section 20 may not include the GNSS antennas 2G1 and 2G2, depending on the embodiment. Functional configuration of the playback system

[0174] Fig. Figure 22 is a diagram illustrating a functional configuration of a playback system according to the second embodiment.

[0175] The functions of the playback system 210 according to the second embodiment are described below with reference to Fig. 22 described.

[0176] As in Fig. As illustrated in Figure 22, the playback system 210 includes a CPU 2100, a memory 2101, a display unit 2102, an operator receiver unit 2103, a communication interface 2104, and a data storage unit 2105. The CPU (central processing unit) 2100 can be a processor such as an FPGA or a GPU instead of a CPU.

[0177] The CPU 2100 is a processor that controls the overall operation of the Playback System 210. Various functions of the CPU 2100 are described below.

[0178] Memory 2101 is a so-called main memory device. Instructions and data required for the operation of the CPU 2100 based on a program are loaded into memory 2101.

[0179] The display unit 2102 is a display device capable of displaying information in a visible manner, and is, for example, a liquid crystal display, an organic EL display, or the like.

[0180] The operating receiver unit 2103 is an input device and is, for example, a general mouse, keyboard, touch sensor or the like.

[0181] The communication interface 2104 is a communication interface for communication with the data logger 220.

[0182] The data storage device 2105 is an auxiliary storage device and is, for example, a hard disk drive (HDD), a solid-state drive (SSD), or the like. The data storage device 2105 records protocol information 2TL received from the data logger 220, the vehicle type of the work machine 23, a work machine model 2TM, which is a 3D model prepared in advance for each model, video information 2MV recorded by the work machine 23, and the like. The work machine model 2TM is described below.

[0183] In addition, the data storage device 2105 also records a unit work prediction model 2PM1 and an element work prediction model 2PM2, which are trained machine learning models used to estimate the work content 2R of the work machine 23, a heat map (2H1 and 2H2) generated during the estimation process, the estimated work content 2R of the work machine 23, three-dimensional point group information 2MAP representing the terrain information of a construction site, and the like. It should be noted that the unit work prediction model 2PM1, the element work prediction model 2PM2, and the heat map (2H1, 2H2) are described further below.The three-dimensional point group information 2MAP includes, for example, three-dimensional point group information measured at a construction site using, for example, a drone or the like, information obtained by sequentially updating terrain information from cutting edge coordinate information of the working machine 23 in operation, and the like, and is recorded in the data storage 2105.

[0184] The functions of the CPU 2100 of the playback system 210 according to the second embodiment are described in detail. The CPU 2100 operates based on a predetermined program to perform functions as a capture unit 21000, a receiver unit 21001, an extraction unit 21002, a playback unit 21003, an estimation unit 21004, a determination unit 21005, and a synchronization unit 21006.

[0185] It should be noted that the predetermined program can serve to implement some of the functions to be performed by the playback system 210. For example, the program can be used to perform a function in combination with other programs already stored in the data memory 2105 or in combination with other programs implemented in other devices. In other embodiments, the playback system 210 can additionally or instead of the above configuration include a user-defined highly integrated circuit (LSI), such as a programmable logic device (PLD).

[0186] Examples of PLDs include programmable array logic (PAL), generic array logic (GAL), complex programmable logic device (CPLD), and field-programmable gate array (FPGA). In this case, some or all of the functions implemented by the processor can be realized by the integrated circuit.

[0187] The acquisition unit 21000 captures the protocol information 2TL to be played back from a multitude of individual protocol information 2TL recorded and collected in the data storage unit 2105. The acquisition unit 21000 also captures the video information 2MV corresponding to the protocol information 2TL to be played back. It is assumed that the multitude of individual protocol information 2TL and the multitude of individual video information 2MV are recorded in the data storage unit 2105 for respective files, which are recorded with different filenames. The acquisition unit 21000 captures either a single protocol information 2TL and the video information 2MV to be analyzed, or two individual protocol information 2TL and two individual video information 2MV to be compared, for example, according to an instruction from the operator.The following example describes a case in which the 21000 capture unit acquires two log information pieces (2TL) and two video information pieces (2MV) for comparison. Since the video information pieces (2MV) are acquired in conjunction with the acquired log information pieces (2TL), the description of the video information pieces (2MV) that are acquired when the log information pieces (2TL) are acquired is omitted. One of the two log information pieces acquired by the 21000 capture unit is a log information that the operator designates as the playback target using a file name or similar identifier. Hereinafter, this log information is referred to as designated log information (2TL1) (first log information). The other of the two log information pieces is a log information that is played back simultaneously with the playback based on designated log information (2TL1).This log information is subsequently referred to as the compared log information 2TL2 (second log information). The compared log information 2TL2 can be log information pre-selected as a "model case" of the work content for each of various work content scenarios and conditions, based on factors such as good fuel efficiency, short operating time, expert driving, and other considerations. Alternatively, the compared log information 2TL2 can be, for example, past log information recorded during an operation previously performed by the same operator as the designated log information 2TL1. In this case, comparing the two log information sets can be used, for example, to analyze the operator's skill level.

[0188] The acquisition unit 21000 according to the present embodiment automatically searches for the compared protocol information 2TL2, which is suitable for comparison with the designated protocol information 2TL1, and acquires it based on a determination result by the determination unit 21005 described below. However, the acquisition unit 21000 according to another embodiment is not limited to the above mode, and, for example, the acquisition unit 21000 can acquire protocol information designated by the operator using a file name or the like as the compared protocol information 2TL2. Additionally, if only one compared protocol information 2TL2 is recorded, the acquisition unit 21000 can acquire the compared protocol information 2TL2.

[0189] The receiving unit 21001 receives a predetermined playback instruction from the operator of the playback system 210. For example, the receiving unit 21001 receives a playback instruction for the work machine 23 from the operator of the playback system 210.

[0190] The extraction unit 21002 extracts angular information, which is used for playback for the working machine 23, from the captured log information 2TL. The angular information is an example of “operational information (motion information)” according to the present disclosure.

[0191] The playback unit 21003 sequentially reproduces the operating images depicting the operation of the working machine 23 in the first display area of ​​a display screen of the display unit 2102 by sequentially applying the operating information depicting the operation of the working machine 23, based on the protocol information 2TL, to the three-dimensional model of the working machine 23. Additionally, the playback unit 21003 sequentially reproduces the captured image, based on the moving image information 2MV according to the protocol information 2TL, in synchronization with the operating image in the second display area, which differs from the first display area of ​​the display screen of the display unit 2102, together with the display of the operating image.Synchronization between the playback of the operating image (playback of the work machine model 2TM) and the playback of the recorded moving image of the same work machine 23 can be ensured based on the time information included in the protocol information 2TL and the time information included in the moving image information 2MV. The playback unit 21003 adjusts the playback timing of the operating image and the recorded moving image so that the recording time, which corresponds to the operating information included in the protocol information 2TL, and the recording time of each individual frame included in the moving image information 2MV substantially coincide.

[0192] The playback unit 21003 can display the estimation result of the work content by the estimation unit 21004 on the display unit 2102 as supplementary information, or it can combine an image (terrain image) representing terrain with the operating image based on the three-dimensional point group information 2MAP and display the combined image on the display unit 2102.

[0193] The estimation unit 21004 estimates the work content of the work machine 23 at any given time from the recorded protocol information 2TL.

[0194] The determination unit 21005 determines whether the compared protocol information 2TL2 is suitable for comparison with the designated protocol information 2TL1, based on the information included in the designated protocol information 2TL1.

[0195] The synchronization unit 21006 performs processing to synchronize the playback of the work machine model 2TM based on the designated protocol information 2TL1 (operating image playback) and the playback of the work machine model 2TM based on the compared protocol information 2TL2 (operating image playback). Specifically, the synchronization unit 21006 specifies the playback start time on the time series of the animation of the work machine model 2TM based on the work content estimated by the estimation unit 21004. Processing sequence by the playback system

[0196] The following describes a specific processing sequence performed by the playback system 210, with reference to Fig. 23, Fig. 24, Fig. 25, Fig. 26, Fig. 27, Fig. 28, Fig. 29, Fig. 30 to Fig. 31 described in detail.

[0197] The process of the in Fig. The processing illustrated in Figure 23 begins when the operator of playback system 210 starts a dedicated application.

[0198] When the operator performs an operation to start the dedicated application, the capture unit 21000 of the CPU 2100 loads the memory 2101 and captures the designated protocol information 2TL1, which is designated as the playback target (step 2S00).

[0199] Here, the protocol information 2TL (designated protocol information 2TL1, compared protocol information 2TL2) is presented with reference to Fig. 24, Fig. 25 to Fig. 26 described.

[0200] As in Fig. 24, Fig. 25 to Fig. As illustrated in Figure 26, the protocol information 2TL includes work machine identification information. The work machine identification information is specifically an individual identification number for the individual identification of the work machine 23. Fig. 24, Fig. 25 to Fig. 26 It is assumed that the work machine identification information is assigned in such a way as to correspond to the vehicle type, model, type number, machine number, and the like of the work machine 23, indicating a hydraulic excavator, wheel loader, bulldozer, or the like. The work machine identification information may be a number or may be an alphabet, a symbol, a combination thereof, or the like.

[0201] As in Fig. As illustrated in Figure 24, the log information 2TL includes information representing the position and orientation of the machine 23 at any given time, as well as angular information of the moving parts of the machine 23. Specifically, the position of the machine 23, the roll angle of the machine 23 (the tilt of the machine body in the left-right direction), the pitch angle (the tilt of the machine body in the forward-backward direction), the rotation angle, the boom angle, the arm angle, and the bucket angle are recorded in the log information TL at each point in time. Here, the data logger 20 installed in the machine 3 specifies and records the position of the machine 23 based on position measurement information representing latitude, longitude, and altitude, information obtained, for example, by reception through the GNSS antennas 2G1 and 2G2.The data logger 220 calculates and records the roll angle and pitch angle of the working machine 23 based on a measurement result from an inertial measurement unit (IMU) installed in the working machine 23. The data logger 220 calculates and records the rotation angle of the upper rotating body 232 based on the position measurement information obtained from each of the GNSS antennas 2G1 and 2G2 provided in the upper rotating body 232. Furthermore, the data logger 220 calculates and records the boom angle, arm angle, and blade angle based on the extension and retraction degrees of each of the boom cylinder 2SL1, arm cylinder 2SL2, and blade cylinder 2SL3. The IMU can be mounted on the boom, arm, blade, and upper rotating body to be used, for example, to acquire the boom angle, arm angle, blade angle, and rotation angle.

[0202] The position, roll angle, and pitch angle are pieces of information required to specify the position and orientation of the working machine 23 itself. Therefore, for example, in an embodiment where only the movements of the moving parts of the working machine 23—that is, the upper rotary body 232, the boom 2BM, the arm 2AR, and the bucket 2BK—are represented as an animation, and the position and orientation of the working machine 23 itself are not represented, the information regarding the position, roll angle, and pitch angle need not be included in the protocol information.

[0203] The in Fig. The protocol information illustrated in Figure 24 (2TL) corresponds to "operating information" as defined in this disclosure. Operating information includes, for example, information describing the operation of the moving parts of the working machine 23, such as the working device 232B, the upper rotary body 232, and the lower drive body 231, which are operated using the operating unit 232AO of the working machine 23.

[0204] Alternatively, the ones in Fig. The protocol information illustrated in Figure 24 (2TL) represents the operation of the working machine 23, which can be visually observed from outside the working machine 23 and can also be referred to as external view operating information. That is, in the present embodiment, the operating information or external view operating information includes the position information, roll angle, pitch angle, yaw angle, boom angle, arm angle, and bucket angle of the working machine 23. In the present embodiment, the operating information (or external view operating information), as information used in estimating the work content, has the characteristic that, unlike the drive mechanism information and operator information described below, it is less likely to be affected by a difference in the vehicle rank of the working machine 23.

[0205] As in Fig. As illustrated in Figure 25, the log information 2TL includes pilot pressures (proportional pressure control pressures (PPC pressures)) which indicate the degrees of input to the control levers 2L1 and 2L2 and the like by the operator at any given time, that is, the degrees of lever tilt and the degrees of pedal pressure. Specifically, the PPC pressures of the control levers 2L1 and 2L2, the travel levers 2R1 and 2R2, or the foot pedals 2F1 and 2F2, corresponding to the respective operating modes of counterclockwise / clockwise rotation, arm extension / deployment, boom raising / lowering, bucket extension / deployment, forward / reverse movement of the right track, and forward / reverse movement of the left track by the operator, are recorded in the log information 2TL at each point in time. The data in Fig. The 25 illustrated time points correspond to those in Fig. 24 illustrated time points. In the present embodiment, each in Fig. 25 illustrated information, also referred to as operating information.

[0206] As in Fig. As illustrated in Figure 26, the log information TL includes information representing the status of the main drive mechanisms, such as the engine and hydraulic pumps of the working machine 23, at any given time. Specifically, the log information 2TL records the engine coolant temperature, engine power, instantaneous fuel consumption, and hydraulic pump oil temperature for each point in time. The information in Fig. The 26 illustrated elements are merely examples, and elements such as engine coolant temperature, hydraulic pump oil temperature, or similar parameters can be omitted. Alternatively, the fuel throttle opening and other factors influencing fuel efficiency can be included. As yet another example, in the case of an electrically powered construction machine, the information can include elements such as current power consumption and remaining battery level. The information in Fig. The 26 illustrated time points correspond to those in Fig. 24 and Fig. 25 illustrated time points. In the present embodiment, each in Fig. 26 illustrated information, also referred to as drive mechanism information.

[0207] Returning to Fig. 23 The estimation unit 21004 of the CPU 2100 estimates the work content of the work machine 23 at each time based on the designated protocol information 2TL1 acquired in step 2S00 (step 2S01).

[0208] Here, a procedure is described in which the estimation unit 21004 estimates the work content of the working machine 23 from the protocol information 2TL, with reference to Fig. 27 described. The estimation unit 21004 estimates the work content of the working machine 23 for both unit work and element work. Unit work is work performed to achieve a specific purpose. Element work is work that specifies a series of operations or tasks, which are elements that constitute the unit work and are classified according to purpose.

[0209] Examples of unit work sections include "excavation and loading", "plowing", "sloping (from below)", "load gathering", "driving" and "stopping", which are in Fig. 27 are illustrated, as well as “trench excavation”, “backfilling”, “sloping (from above)” and the like.

[0210] "Excavation and loading" refers to work involving the excavation and removal of earth and sand or rock, and the loading of this material onto the platform of a transport vehicle. "Excavation and loading" encompasses all work performed as a single unit, including excavation, turning the loaded vehicle, unloading, turning the unloaded vehicle, waiting for unloading, and compacting the loading platform.

[0211] "Plowing" is the work of removing excess soil relief to a predetermined height. Plowing is a unit operation that includes digging and unloading, or digging, turning with a load, unloading, and turning without a load, and may include push leveling and sweeping.

[0212] "Below slope" is the work of creating a slope by the work machine 23, which is located below the target location. Below slope is a unitary work that includes compaction, excavation, loaded turning, unloading, and unloaded turning, and may include thrust grading.

[0213] "Load collection" refers to work involving the gathering of soil and sand resulting from excavation or similar processes before loading it onto a transport vehicle. Load collection is a single, unified task that includes excavation, turning the loaded vehicle, unloading, turning the unloaded vehicle, and may also include grading.

[0214] "Driving" refers to work in which the working machine 23 is moved. Driving as a unit of work is unit work that includes driving as an element of work.

[0215] "Stopping" is a state of halting for a predetermined time or longer, without any soil, sand, or rock in the shovel 2BK. Stopping as a unit of work is unit of work that includes stopping as an element of work.

[0216] "Trench excavation" is the work of digging and scraping the soil into a long and narrow trench. Trench excavation is a single operation that includes excavation, turning with a load, unloading, turning without a load, and may also include grading.

[0217] "Backfilling" is the process of filling a trench or hole already formed in the ground with soil and sand, and backfilling to level the surface. Backfilling is a single operation that includes excavation, turning with a load, unloading, compaction, and turning without a load, and may include grading and sweeping.

[0218] "Sloping" (from above) is the work of creating a slope by the work machine 23, which is located above the target location. Sloping (from above) is a unitary work that includes compaction, excavation, loaded turning, unloading and unloaded turning, and may include thrust grading.

[0219] Examples of element work sections include "excavation", "loaded turning", "waiting for unloading", "unloading", "unloaded turning", "loading platform pressing", which are in Fig. 27 are illustrated, as well as “compacting”, “thrust leveling”, “sweeping” and the like.

[0220] The term "excavation" refers to work in which earth, sand or rock is dug out and removed using the 2BK shovel.

[0221] “Loaded rotation” refers to work in which the rotating body 232 is rotated while the removed earth and sand or rock is held in the bucket 2BK.

[0222] “Waiting for unloading” refers to work where one waits for a transport vehicle to be loaded, while holding the removed earth and sand or rock in the 2BK bucket.

[0223] "Unloading" refers to work in which the removed earth and sand or rock from the 2BK bucket is unloaded onto a transport vehicle or to a predetermined location.

[0224] “Unloaded turning” refers to work in which the upper rotating body 232 is turned without soil, sand or rock in the bucket 2BK.

[0225] The term "loading platform pressing" refers to work in which earth and sand loaded on the loading platform of a transport vehicle are pressed from above with the 2BK bucket to smooth the earth and sand.

[0226] "Compacting" refers to work in which soil and sand are pushed into the disturbed ground with a shovel to shape and solidify the soil.

[0227] "Thrust leveling" refers to work in which earth and sand are pushed and leveled with the ground surface of the 2BK bucket.

[0228] The term "sweeping" refers to work in which earth and sand are swept and leveled using a side surface of the shovel 2BK.

[0229] The estimation unit 21004 obtains a time series of probabilities regarding unit work when the protocol information 2TL is input into the unit work prediction model 2PM1 in chronological order. The unit work prediction model 2PM1 is a model that outputs the probability regarding unit work when the protocol information 2TL is input during training using, for example, teacher data, and can be stored, for example, in data storage 2105.

[0230] The estimation unit 21004 obtains a time series of probabilities regarding element work when the protocol information 2TL is fed into the element work prediction model 2PM2 in chronological order. The element work prediction model 2PM2 is a model that outputs the probability with respect to element work when the protocol information 2TL is fed into it during training using, for example, teacher data, and can be stored, for example, in data storage 2105.

[0231] The estimation unit 21004 smooths the time series of probabilities by applying a time-averaging filter to the time series of probabilities with respect to unit work and the time series of probabilities with respect to element work, generating a unit work heatmap 2H1, which represents the smoothed time series of probabilities with respect to unit work, and an element work heatmap 2H2, which represents the smoothed time series of probabilities with respect to element work, as shown in Fig. Figure 27 illustrates this. Heatmaps 2H1 and 2H2 are maps in which colors representing the probabilities of job classifications are applied to a plane, with the vertical axis representing the job classifications and the horizontal axis representing time points based on the smoothed time series of probabilities. For example, the colors relating to the heatmaps might be closer to blue for a lower probability of job classification and closer to red for a higher probability of job classification. The estimation unit 21004 stores heatmaps 2H1 and 2H2 in data storage 2105.

[0232] The estimation unit 21004 specifies a period in which the probability of unit work is dominant, based on the smoothed time series of probabilities, and estimates the work content of the work machine 23 in that period. For example, in a period in which the probability of "excavation and loading," which is unit work, is dominant, the work content of the work machine 23 is estimated as "excavation and loading." Similarly, the estimation unit 21004 specifies a period in which the probability of element work is dominant, based on the smoothed time series of probabilities, and estimates the work content of the work machine 23 in that period. For example, in a period in which the probability of "excavation," which is element work, is dominant, the work content of the work machine 23 is estimated as "excavation."The estimation unit 21004 stores information about the estimated work content R for the working machine 23 in the data storage unit 2105.

[0233] Returning to Fig. Step 23 then determines whether the acquisition unit 21000 should display information to be compared with the designated protocol information 2TL1, which is designated as the playback target (compared protocol information 2TL2), based on a predetermined instruction from the operator (step 2S02A). If the information is to be displayed (step 2S02A: YES), processing proceeds to step 2S04 after processing in step 2S02B and step 2S03 has been performed. If the information is not to be displayed (step 2S02A: NO), processing proceeds to step 2S04 without executing steps 2S02B and 2S03. If the information is to be displayed (step 2S02A: YES), the acquisition unit 21000 captures the compared protocol information 2TL2, which is suitable for comparison with the designated protocol information 2TL1 captured in step 2S00 (step 2S02B).

[0234] At this point, the determination unit 21005 performs a determination processing sequence to determine whether the multitude of individual compared protocol information 2TL2, recorded in the data store 2105, are suitable for comparison with the designated protocol information 2TL1 acquired in step 2S00. In this determination processing sequence, the determination unit 21005 first selects one of the multitude of individual compared protocol information 2TL2 that were pre-recorded in the data store 2105.Then, the determination unit 21005 determines whether, for example, the vehicle rank, the amount of change in the rotation angle (the extent of the swivel), the bucket height at the time of waiting to be unloaded by the working machine 23, and the like are substantially the same, based on the log information 2TL, and, if it determines that they are not substantially the same, selects a new single compared log information 2TL2 and performs the same determination processing as described above.

[0235] In addition, according to another embodiment, the determination unit 21005 can further limit the candidates for reproduction based on a work position, a working time, a work classification or the like, in addition to the determination processing described above.

[0236] Returning to Fig. 23 Next, the synchronization unit 21006 performs processing to synchronize the animation playback for the work machine model 2TM based on the designated protocol information 2TL1 and the animation playback for the work machine model 2TM based on the compared protocol information 2TL2 (step 2S03). In particular, the synchronization unit 21006 specifies the playback start time on the time series in the animation of the work machine model 2TM based on the designated protocol information 2TL1 and the playback start time on the time series in the animation of the work machine model 2TM based on the compared protocol information 2TL2.

[0237] One method for specifying the playback start time to synchronize the two is as follows. That is, the synchronization unit 21006 extracts the unit work toggling time in the designated log information 2TL1 based on the unit work heatmap 2H1 for the designated log information 2TL1, generated by the estimating unit 21004. Next, the synchronization unit 21006 extracts the unit work toggling time in the compared log information 2TL2 based on the unit work heatmap 2H1 for the compared log information 2TL2, generated by the estimating unit 21004. The synchronization unit 21006 selects one of the extracted times and specifies the selected time as the playback start time of each animation.

[0238] The playback system 210 according to another embodiment is not limited to the mode described above. The playback system 210 (synchronization unit 21006) according to another embodiment can compare a combination of angular information of the boom, arm, bucket, and the like in the compared protocol information 2TL2 with a combination of angular information of the boom, arm, bucket, and the like in the designated protocol information 2TL1, extract a time point at which both angles are close to each other, and specify that time point, for example, as the playback start time of each animation.

[0239] After step 2S03, or if “NO” is determined in step 2S02A, the receiving unit 21001 receives a playback instruction from the operator (step 2S04).

[0240] One aspect of the playback instruction can be an operation such as pressing a playback button. Additionally, the playback instruction can include information that serves as the starting point for playback, such as a time, the position of the machine 23, and various events, such as the occurrence of an abnormality in the machine 23. When the receiving unit 21001 receives the playback instruction, a playback area designation unit 21007 can be used to designate the work content that is to be a generation target of the motion trajectory image.

[0241] Next, the acquisition unit 21000 selects the work machine model 2TM corresponding to the referenced work machine identification information from the data storage 2105 and reads it, based on the work machine identification information, as the type of work machine 23 received from the receiving unit 21001 (step 2S05).

[0242] Here, the 2TM work machine model is described with reference to Fig. 28 described.

[0243] As in Fig. As illustrated in Figure 28, the working machine model 2TM is information that includes the working machine identification information and an external shape 3D model 2M0 of the working machine 23 specified by the working machine identification information. The external shape 3D model 2M0 is a 3D model that represents the working machine 23 and is constructed for each part of the working machine 23, such as the lower drive body and the upper rotating body. For example, the outer shape 3D model 2M0 represents the shape of the working machine 23. For example, the outer shape 3D model 2M0 includes an outer shape model of the lower travel body 2M01, which represents the lower travel body 231 of the working machine 23, an outer shape model of the upper rotating body 2M02, which represents the upper rotating body 232, a boom outer shape model 2M03, which represents the boom 2BM, an arm outer shape model 2M04, which represents the arm 2AR, and a bucket outer shape model 2M05, which represents the bucket 2BK..

[0244] Returning to Fig. 23. Extraction unit 21002 extracts information used for playback from each of the designated protocol information 2TL1 and the compared protocol information 2TL2, or from the designated protocol information 2TL1 (step 2S06). For example, extraction unit 21002 extracts various types of angular information, such as boom angle, arm angle, and blade angle, as information used for playback. The in Fig. The 25 illustrated pilot printouts can be extracted as the information used for replay. Additionally, only information used for replay can be captured in steps 2S00 and 2S02B. The following describes a processing flow by taking as an example a case where replay is performed with the designated protocol information 2TL1 and the compared protocol information 2TL2. If replay is not performed based on the compared protocol information 2TL2, the processing with respect to the compared protocol information 2TL2 can generally be omitted.

[0245] Next, the playback unit 21003 performs a simultaneous playback of animation of the working machine model 2TM based on the designated protocol information 2TL1 and animation playback of the working machine model 2TM based on the compared protocol information 2TL2, motion picture playback based on the motion picture information 2MV according to the designated protocol information 2TL1, and motion picture playback based on the motion picture information 2MV according to the compared protocol information 2TL2 (step 2S07).Here, the playback unit 21003 displays an operating image representing the operation of the work machine 23, while various types of information recorded in the designated protocol information 2TL1 are applied to the work machine model 2TM in chronological order of the timestamp, starting from the playback start time specified in step 2S03 of the synchronous processing (step 2S07a). Simultaneously with the playback of the operating image for the animation, the playback unit 21003 generates an operating image representing the operation of the work machine 23, while various types of information recorded in the compared protocol information 2TL2 are applied to the work machine model 2TM in chronological order of the timestamp, starting from the playback start time specified in step 2S03 of the synchronous processing (step 2S07a).

[0246] The following describes in detail the contents of the operational image generation processing by the playback unit 21003. The playback unit 21003 modifies the angle of the corresponding section of the outer shape 3D model 2M0 based on various types of angular information, such as the rotation angle and the boom angle, specified in the log information 2TL (the designated log information 2TL1 and the compared log information 2TL2). For example, the playback unit 21003 reproduces the position and orientation of the bucket 2BK of the working machine 23 by tilting the bucket outer shape model 2M05 about the axis of rotation defined at a position of the connection with the arm outer shape model 2M04 to the bucket angle specified in the log information 2TL.

[0247] Similarly, the playback unit 21003 reproduces the position and orientation of the arm 2AR of the working machine 23 by tilting the arm outer shape model 2M04 about the axis of rotation defined at a position of the connection with the boom outer shape model 2M03 to the arm angle specified in the protocol information 2TL.

[0248] Similarly, the playback unit 21003 reproduces the position and orientation of the upper rotary body 232 of the working machine 23 by tilting the outer shape model 2M02 of the upper rotary body about the axis of rotation defined at a position of the connection with the outer shape model 2M01 of the lower driving body to the angle of rotation specified in the protocol information 2TL.

[0249] Similarly, the playback unit 21003 reproduces the position of the upper rotary body 232 of the working machine 23 by tilting the outer shape model 2M01 of the lower carriage about the roll rotation axis defined in the outer shape model 2M01 of the lower carriage to the roll angle specified in the protocol information 2TL and tilting the outer shape model 2M01 of the lower carriage about the pitch rotation axis defined in the outer shape model 2M01 of the lower carriage to the pitch angle specified in the protocol information 2TL.

[0250] Furthermore, according to the second embodiment, the playback system 210 can reproduce an animation of the driving movement of the working machine 23 based on the PPC presses of the forward / reverse movement of the right crawler track and the forward / reverse movement of the left crawler track at any time included in the log information 2TL.

[0251] Specifically, the outer shape 3D model 2M0 is moved forward, backward, forward left and right, and backward left and right based on the PPC pressures of the forward / backward movement of the right and left track. For example, the outer shape 3D model 2M0 moves forward based on the numerical values ​​of the PPC pressures of the forward movement of the right and left track. The movement speed can be changed based on the numerical values ​​of the PPC pressures.

[0252] The outer shape 3D model 2M0 is moved backward based on the numerical values ​​of the PPC pressures for the reverse movement of the right and left track. Furthermore, based on the difference in the numerical values ​​of the PPC pressures between the forward movement of the right and left track, the outer shape 3D model 2M0 is moved to curve in the forward left-right direction. For example, if the numerical value of the PPC pressure for the forward movement of the right track is greater than the numerical value of the PPC pressure for the forward movement of the left track, a movement to curve forward to the left is implemented. The speed of movement and the size of the curve can be changed depending on the numerical values ​​of the respective PPC pressures and the difference between these values.

[0253] Similarly, based on a difference in the numerical values ​​of the PPC pressures between the reverse movement of the right track and the reverse movement of the left track, the outer shape 3D model 2M0 is moved so that it curves in the reverse left-right direction. For example, if the numerical value of the PPC pressure for the reverse movement of the right track is greater than the numerical value of the PPC pressure for the reverse movement of the left track, a movement to curve in the reverse left direction is implemented.

[0254] The speed of movement and the size of the curve can be changed depending on the numerical values ​​of the respective PPC pressures and the difference between the numerical values ​​of the PPC pressures.

[0255] The movement of the work machine 23 can be animated more precisely by playing back the animation using position information in addition to the PPC (Power Point Control) inputs for the forward / reverse movement of the right and left tracks. In this case, the speed and position of the movement of the work machine 23 can be expressed more accurately using the position information. Additionally, by playing back the animation of the work machine 23 based on the roll angle, pitch angle, or both, in addition to the PPC inputs for the forward / reverse movement of the right and left tracks, the tilt of the work machine 23 in the left-right direction or the tilt of the work machine 23 in the forward-backward direction during movement can be displayed.

[0256] In step 2S07, the playback unit 21003 causes the acquisition unit 21000 to acquire the moving image information 2MV according to the protocol information 2TL (the designated protocol information 2TL1 and the compared protocol information 2TL2) in synchronization with the playback of the operating image in step 2S07a, and sequentially reproduces the captured image based on the moving image information 2MV acquired by the acquisition unit 21000 (step 2S07b).

[0257] In step 2S07, the playback unit 21003 displays a terrain image on the display unit 2102, which represents the terrain based on the three-dimensional point group information 2MAP, in combination with the operating image (step 2S07c).

[0258] In step 2S07, for example, the playback unit 21003 displays supplementary information, such as an estimate of the work content, on the display unit 2102 at any given time (step 2S07d).

[0259] While the animation of the operation of the two work machines 23 is playing, the playback unit 21003 determines whether the animation playback should be terminated (step 2S08). For example, if a command to terminate playback is received based on the pressing of a stop button or the like, it is determined that the animation playback should be terminated. It can also be determined that the animation playback should be terminated after a predetermined time period has elapsed since the start of the animation playback. If the animation playback is not to be terminated (step 2S08; NO), the playback unit 21003 continues the simultaneous animation playback for the two work machine models 2TM. On the other hand, if the animation playback is to be terminated (step 2S08; YES), the playback unit 21003 terminates the animation playback processing.

[0260] In the reference to Fig. In the processing sequence described in Section 23, steps 2S00, 2S01, 2S03, 2S04, 2S05, 2S06 and 2S08 are not the essential configuration of the playback system 210, and other embodiments may not include these steps. Display image of the playback system

[0261] Fig. 29 and Fig. Figure 30 illustrates examples of display images based on the designated protocol information 2TL1 and the compared protocol information 2TL2 of the playback system according to the second embodiment. Fig. Figure 31 illustrates an example of a display image based on the designated protocol information 2TL1 of the playback system according to the second embodiment.

[0262] When comparing the designated protocol information 2TL1 with the compared protocol information 2TL2, the CPU 2100 of the playback system 210, according to the second embodiment, displays display images 2D1 and 2D2 on the display unit 2102, as for example in Fig. 29 and Fig. 30 illustrated.

[0263] The in Fig. Figure 29 illustrates display image 2D1, which includes a first display area 2DR1 and a second display area 2DR2. An operating image 2D11 is displayed in the first display area 2DR1. The operating image 2D11 is an image that includes an operating image 2D111 (indicated by a solid line), based on the designated protocol information 2TL1, and an operating image 2D112 (indicated by a dashed line), based on the compared protocol information 2TL2, which are superimposed, for example, in different display colors. The operating image 2D11 includes the operating image 2D111 and a terrain image 2D113, which is generated for combination with the operating image 2D112. Additionally, a captured image 2D12 is displayed in the second display area 2DR2.The captured image 2D12 includes a captured image 2D121 based on the moving image information 2MV corresponding to the designated protocol information 2TL1, and a captured image 2D122 based on the moving image information 2MV corresponding to the compared protocol information 2TL2. In this example, the captured image 2D12 includes images depicting the dump truck's loading platform, the working attachment, and the bucket's load, as well as the loading targets of excavated objects, and the like. In this example, by comparing the captured image 2D121 and the captured image 2D122, the bucket's loading status can be determined, for instance.

[0264] The display image 2D1 includes supplementary information (image) 2D13. The supplementary information 2D13 includes a time series 2D131 of the estimated result of the work content based on the designated protocol information 2TL1, a time series 2D132 of the estimated result of the work content based on the compared protocol information 2TL2, and a playback time symbol 2D133. It should be noted that, according to another embodiment, the playback system 210 can be configured to display the two operating images 2D111 and 2D112 side by side, instead of displaying them in an overlapping (superimposed) manner. In this case, the respective images can be displayed on two adjacent displays.

[0265] The in Fig. Figure 30 illustrates display image 2D2, which includes the first display area 2DR1 and the second display area 2DR2. An operating image 2D21 is displayed in the first display area 2DR1. The operating image 2D21 is an image viewed from a different angle than the one shown in Figure 30. Fig. Figure 29 illustrates operating image 2D11. The viewing angle can be changed, for example, by performing a predefined mouse operation on operating image 2D11. Operating image 2D21 is an image that includes operating image 2D211 (indicated by a solid line), which is based on the designated protocol information 2TL1, and operating image 2D212 (indicated by a dashed line), which is based on the compared protocol information 2TL2, superimposed, for example, in different display colors. Operating image 2D21 includes operating image 2D211 and a terrain image 2D213, which is generated for combination with operating image 2D212. Additionally, a captured image 2D22 is displayed in the second display area 2DR2.The captured image 2D22 includes a captured image 2D221 based on the moving image information 2MV corresponding to the designated protocol information 2TL1, and a captured image 2D222 based on the moving image information 2MV corresponding to the compared protocol information 2TL2.

[0266] The display image 2D2 includes the supplementary information (image) 2D23. The supplementary information 2D23 includes a time series 2D231 of the work content estimate based on the designated protocol information 2TL1, a time series 2D232 of the work content estimate based on the compared protocol information 2TL2, and a playback time symbol 2D233.

[0267] A 2D1A display image that appears in Fig. Figure 31 illustrates this, encompassing the first display area 2DR1 and the second display area 2DR2. The operating image 2D11 is displayed in the first display area 2DR1. Operating image 2D11 is the operating image 2D111 based on the designated protocol information 2TL1. Operating image 2D11 includes the terrain image 2D113, which is generated to be combined with operating image 2D111. Additionally, the captured image 2D12 is displayed in the second display area 2DR2. The captured image 2D12 is the captured image 2D121 based on the motion image information 2MV according to the designated protocol information 2TL1. In this example, the captured image 2D12 includes images representing the dump truck's loading platform, the attachment and bucket load, the loading targets of excavated objects, and the like.In this example, the captured image 2D121 can be used to determine the bucket's loading status. The display image 2D1 includes the supplementary information (image) 2D13. This supplementary information includes the time series 2D131 of the estimated work content based on the labeled log information 2TL1 and the playback time symbol 2D133. Operation and impacts

[0268] As described above, the playback system 210 according to the second embodiment includes the acquisition unit 21000 and the playback unit 21003. The acquisition unit 21000 acquires the log information 2TL of the working machine 23, which is assigned to a time point in time, and the moving image information 2MV, which represents a recorded moving image that was captured by the working machine 23 and assigned to a time point in time.The playback unit 21003 sequentially reproduces the operating image 2D11, which depicts the operation of the working machine 23, in the first display area 2DR1 of the display unit 2102. It does this by sequentially applying the operating information representing the operation of the working machine 23, based on the protocol information 2TL, to the working machine model 2TM, which is a three-dimensional model of the working machine 23. It then sequentially reproduces the captured image 2D12, based on the motion image information 2MV, in synchronization with the operating image 2D11 in the second display area 2DR2, which differs from the first display area 2DR1 of the display unit 2102. According to this configuration, the work performed by the working machine 23 can be analyzed in detail by referring to the captured image taken by the working machine 23.

[0269] In the present embodiment, the acquisition unit 21000 captures individual protocol information 2TL and individual moving image information 2MV of the plurality of work machines 23, and the playback unit 21003 sequentially displays the operating images 2D111 and 2D112 of each of the work machines 23 in a superimposed manner in the first display area 2DR1 and displays each of the captured images 2D121 and 2D122 side by side in the second display area 2DR2 based on the respective individual moving image information 2MV. According to this configuration, an animation display of a plurality of 3D models based on a plurality of individual protocol information 2TL and a moving image display of a plurality of captured images can be implemented simultaneously. Third embodiment

[0270] A playback system and a playback method according to a third embodiment are described below with reference to Fig. 32, Fig. 33 to Fig. 34 described in detail. Fig. Figure 32 is a diagram illustrating a functional configuration of a playback system according to the first embodiment. Fig. Figure 33 is a diagram illustrating a processing flow through the playback system according to the third embodiment. Fig. Figure 34 is a diagram illustrating an example of a display image according to the third embodiment. A display system 210a of the third embodiment is configured, for example, to set a plurality of work machines 23 that are working (or have worked) at the same construction site at the same time as display targets, to display the work machines 23 on a bird's-eye view image representing the construction site, and to specify the work machine 23 as the display target on the bird's-eye view image, so that the work machine 23 can be switched as the display target.

[0271] The playback system 210a of the third embodiment, which is in Fig. 32, which is illustrated, differs from the playback system 210 of the second embodiment, which is shown in Fig. 22 is illustrated in the following configuration. That is, part of the content of a 21000a capture unit, which is in Fig. 32 is illustrated, differing from that of the recording unit 21000, which is shown in Fig. 22 is illustrated. The one in Fig. 32 illustrated CPU 2100 now includes the playback area determination unit 21007 and a bird's-eye view image display unit 21008.

[0272] The 21007 playback area determination unit defines a playback area by, for example, specifying a section of a construction site and a date and time of playback according to operator input. The construction site area can be defined by a coordinate range or by specifying a registered name of a pre-registered coordinate range.

[0273] The bird's-eye view image display unit 21008 generates a bird's-eye view image that includes the multitude of work machines 23 that are located at the same time in the same construction site area, which is determined by the playback area determination unit 21007, and displays the bird's-eye view image, which includes each of the work machines 23 in a selectable form, in a third display area 2DR3, which is distinct from the first display area 2DR1 and the second display area 2DR2 of the display unit 2102.

[0274] The acquisition unit 21000a acquires all log information 2TL that meets the designation condition by the playback area designation unit 21007 as the target of estimation by the estimation unit 21004, and acquires the log information 2TL and the moving image information 2MV of the working machine 23 that were selected using the bird's-eye view image displayed by the bird's-eye view image display unit 21008 as the target of playback by the playback unit 21003.

[0275] Other configurations of the CPU 2100 are the same in the second and third embodiments.

[0276] As in Fig. Figure 33 illustrates that in the playback system 210a of the third embodiment, the playback area designation unit 21007 first designates an area to be reproduced (step 2S101). Next, the acquisition unit 21000a captures all protocol information 2TL that fulfills the playback condition (step 2S102). Next, the estimation unit 21004 estimates the work content based on all captured protocol information 2TL (step 2S103). Next, the receiving unit 21001 receives a playback instruction (step 2S104). Next, the bird's-eye view image display unit 21008 displays the bird's-eye view image (step 2S105). Next, the bird's-eye view image display unit 21008 specifies the work machine 23 to be reproduced based on mouse operation or the like on the bird's-eye view image (step 2S106).In the subsequent processing in step 2S106, the machine 23 to be reproduced can be specified again or not.

[0277] Next, the acquisition unit 21000 selects and reads the work machine model 2TM from the data store 2105 based on the work machine identification information as the type of specified work machine 23 (step 2S107). Next, the extraction unit 21002 extracts the information used for playback from the log information 2TL1 (step 2S108).

[0278] Next, the playback unit 21003 performs playback processing (step 2S109). In step 2S109, operating image playback (2S109a), moving image playback (2S109b), terrain image display (2S109c), and supplementary information display (2S109d) are processed according to the operating image playback (2S07a), moving image playback (2S07b), terrain image display (2S07c), and supplementary information display (2S07d), respectively, which are in Fig. 23 are illustrated, and their descriptions are omitted.

[0279] The determination processing in step 2S110 is identical to the determination processing in step 2S08, which is described in Fig. Figure 23 illustrates this. However, in the case of step 2S110, processing returns to step 2S105 if playback is not terminated.

[0280] Fig. Figure 34 is a diagram illustrating an example of a display image according to the third embodiment. A Fig. The illustrated display image 2D3 includes the first display area 2DR1, the second display area 2DR2, and the third display area 2DR3. The first display area 2DR1 includes an operating image 2D31. In this example, the operating image 2D31 includes an operating image 2D311 of one specified work machine and a terrain image 2D313. The second display area 2DR2 includes a captured image 2D32. The third display area 2DR3 includes a bird's-eye view image 2D34. In this case, the bird's-eye view image 2D34 includes a symbol 2D341 and a symbol 2D342, which correspond to the two work machines 23. The symbols 2D341 and 2D342 indicate the image capture direction by a triangular dashed line area and indicate the direction of movement by a dashed arrow.

[0281] The display image 2D3 includes supplementary information 2D33. This supplementary information includes a time series of the estimated result of the work content based on the designated protocol information 2TL, a time series indicating a control state (for example, a state of positional accuracy) of the automatic control system, and a playback time symbol 2D333.

[0282] According to the present embodiment, the playback system 210a newly includes the bird's-eye view image display unit 21008. The bird's-eye view image display unit 21008 generates the bird's-eye view image 2D34, which includes the plurality of work machines 23 that are located in the same construction site area at the same time, and displays the bird's-eye view image 2D34, which includes each of the work machines in a selectable form, in a third display area 2DR3, which is distinct from the first display area 2DR1 and the second display area 2DR2 of the display unit 2102. The acquisition unit 21000a acquires the log information 2TL and the motion image information 2MV of the work machine 23 selected in the bird's-eye view image 2D34. According to this configuration, for example, if a plurality of work machines 23 are to be analyzed (or monitored), the work machine 23 to be displayed can simply be selected. Modified example

[0283] The content of the log information 2TL according to the preceding embodiment is not limited to this in other embodiments. For example, if the work machine 23 is not a hydraulic excavator but another type of vehicle, log information 2TL is recorded according to the vehicle type. Other vehicle types include, for example, a wheel loader, a bulldozer, and the like.

[0284] Furthermore, the description is given under the assumption that the playback system 210 according to the above embodiment is installed at a location remote from the working machine 23 and is connected to the data logger 220 installed in the working machine 23 via the wide area communication network; however, other embodiments are not limited to this aspect.

[0285] For example, in another embodiment, part or all of the configuration of the playback system 210 can be installed inside the machine 23. In this case, the data logger 220 can transmit the log information 2TL to the playback system 210 via a network inside the machine 23 or the like, without using the wide area communication network. In this way, the operator entering the machine 23 can verify the movement of the machine 23 they are operating on-site by replaying the animation. Furthermore, the movement of the machine 23 can be replayed as a model case for the operator of the machine 23 to use as a guide.

[0286] The playback system 210 installed in the work machine 23 can acquire the protocol information 2TL of another work machine 23 via a wide area communication network or the like. In this way, the states of the work machines 23, except for the work machine 23 in which the playback system 210 is installed, can be displayed as an animation.

[0287] In addition, according to another embodiment, the playback system 210 can be installed at a location remote from the working machine 23 and can transmit video information generated by the animation playback processing to a monitor installed in the working machine 23 for display.

[0288] In another embodiment, one aspect of the playback instruction received from the operator may be, for example, a playback period. For instance, the playback period may include a playback start time and a playback end time. In this case, the playback system 210 plays back the work machine 23 during the received playback period. In another embodiment, determining the playback end time is not essential. For example, in another embodiment, only the playback start time may be received by the operator as a playback instruction, and playback may be performed for a certain period of time from the playback start time, playback may continue as long as the log information is available, or playback may be stopped in accordance with the occurrence of other various events.

[0289] The protocol information 2TL to be captured does not need to be arranged in chronological order. In this case, the playback unit 21003 can apply the information from the protocol information 2TL used for playback to the work machine model 2TM in time series.

[0290] The steps of the various types of processing of the playback system 210, as described above, are stored on a computer-readable recording medium in the form of a program, and the computer reads and executes the program to implement the various types of processing described above. The computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Furthermore, this computer program can be distributed to the computer via a communication circuit, and the computer receiving this distribution can execute the program.

[0291] The program described above can be a program for executing a section of the functions described above. Furthermore, the functions described above can be implemented in combination with a program already recorded on the computer system, namely a so-called differential file, a differential program, or the like.

[0292] Some or all of the functions of the playback system 210 described above can be assigned to the working machine 23. For example, some or all of the functions of the acquisition unit 21000, the receiving unit 21001, the extraction unit 21002, the playback unit 21003, the estimating unit 21004, the determining unit 21005, the synchronization unit 21006, the memory 2101, the display unit 2102, the operating receiving unit 2103, the communication interface 2104, and the data storage unit 2105 can be assigned to the working machine 23.

[0293] Certain embodiments of the present disclosure have been described above, but all embodiments are merely illustrative and are not intended to limit the scope of protection of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the core of the invention. These embodiments and modifications are included in the scope of protection and core of the invention and are also included in the scope of protection of the invention described in the claims and their equivalents. Additional remarks

[0294] The reproduction system 210, 210a according to the present disclosure is understood, for example, as follows. (7) A playback system 210, 210a according to a seventh consideration includes: the capture unit 21000, 21000a, which is configured to capture the log information 2TL of the working machine 23, which is associated with a time point in time, and the moving image information 2MV, which represents a captured moving image that was recorded by the working machine and associated with a time point in time;and the playback unit 21003, which is configured to sequentially reproduce in the first display area 2DR1 of the predetermined display unit 2102 the operating image 2D11, 2D21, 2D31, which represents the operation of the working machine, by sequentially applying operating information representing the operation of the working machine, based on the protocol information, to a three-dimensional model (working machine model 2TM) of the working machine, and in the second display area 2DR2, which differs from the first display area of ​​the display unit, to sequentially reproduce the recorded image 2D12, 2D22, 2D32 based on the moving image information 2MV in synchronization with the operating image. According to the present and the following aspects, it is possible to analyze work performed by a working machine in more detail. (8) A playback system 210a according to an eighth consideration is the playback system according to (7), which further includes the bird's-eye view image display unit 21008, which is configured to produce the bird's-eye view image 2D34, which includes a plurality of work machines that are in the same construction site area at the same time, and to display the bird's-eye view image in the third display area 2DR3, which is different from the first display area or the second display area of ​​the display unit, wherein each of the work machines is selectable, and wherein the capture unit 21000a captures the log information and the motion image information of the work machine selected in the bird's-eye view image. (9) A playback system 210, 210a according to a ninth consideration is the playback system according to (7) or (8), wherein the acquisition unit 21000, 21000a acquires the protocol information and the moving image information from each of a plurality of working machines, and the playback unit 21003 sequentially reproduces the operating images of the respective working machines in a superimposed manner in the first display area and reproduces the respective acquired images side by side in the second display area based on each individual moving image information. Fourth embodiment

[0295] A display control system and a display control method according to a fourth embodiment are described below with reference to Fig. 35, Fig. 36, Fig. 37, Fig. 38, Fig. 39, Fig. 40, Fig. 41, Fig. 42, Fig. 43, Fig. 44, Fig. 45, Fig. 46, Fig. 47 to Fig. 48 described in detail. Overall configuration of the analysis support system

[0296] Fig. Figure 35 is a diagram illustrating an overall configuration of an analysis support system according to the fourth embodiment.

[0297] An analysis support system 31 includes a playback system 310 and a data logger 320, which are installed in each of a variety of work machines 33.

[0298] It should be noted that in the present embodiment, the playback system 310 is a configuration example of a display control system according to the present disclosure.

[0299] The work machines 33 are targets of work analysis by the playback system 310. Examples of the work machines 33 include a hydraulic excavator, a wheel loader, and a bulldozer. In the following description, a hydraulic excavator is described as an example of the work machines 33. Each of the work machines 33 is equipped with a variety of sensors. The data logger 320 records and collects information representing the state of the work machine 33 as detected by the sensors in time series. The information representing the state of the work machine 33 at each time point, recorded by the data logger 320, is subsequently referred to as log information.In a case where an operating mechanism for the working machine 33 is configured to operate the working machine 33 using an electrical control signal, information from the working machine 33's control signal can be recorded and collected in time series and included in the log information. The data logger 320 transmits the recorded log information to the playback system 310 via a wide area communication network at a predetermined time interval. The predetermined time interval is, for example, five minutes. The playback system 310 records the log information received from the data logger 320 onto a recording medium. The function of the playback system 310 is described below. Structure of the working machine

[0300] Fig. Figure 36 is a diagram illustrating the structure of a working machine according to the fourth embodiment.

[0301] The work machine 33, which is a hydraulic excavator, excavates and levels earth, sand, or similar materials at a construction site or similar location. As in Fig. As illustrated in Figure 36, the working machine 33, which is a hydraulic excavator, includes a lower drive body 331 for travel and an upper rotating body 332, which is rotatably installed in an upper section of the lower drive body 331. Additionally, the upper rotating body 332 is equipped with a cabin 332A, a working device 332B, and two antennas 3G1 and 3G2 of a global navigation satellite system (GNSS).

[0302] The lower chassis 331 has a left track 3CL and a right track 3CR. The working machine 33 moves forward, turns, and moves backward by the rotation of the left track 3CL and the right track 3CR.

[0303] Cabin 332A is a location where an operator of the work machine 33 enters and operates the work machine 33. Cabin 332A is installed, for example, on a left side section of a front end section of the upper rotary body 332. The internal configuration of cabin 332A is described below.

[0304] The working device 332B includes a boom 3BM, an arm 3AR, and a bucket 3BK. The boom 3BM is attached to a front end section of the upper rotating body 332. The arm 3AR is attached to the boom 3BM. The bucket 3BK is attached to the arm 3AR. A boom cylinder 3SL1 is located between the upper rotating body 332 and the boom 3BM. By actuating the boom cylinder 3SL1, the boom 3BM can be actuated relative to the upper rotating body 332. The arm cylinder 3SL2 is located between the boom 3BM and the arm 3AR. By actuating the arm cylinder 3SL2, the arm 3AR can be actuated relative to the boom 3BM. A bucket cylinder 3SL3 is located between the arm 3AR and the bucket 3BK. By driving the bucket cylinder 3SL3, the bucket 3BK can be actuated in relation to the arm 3AR.

[0305] The upper rotating body 332, the boom 3BM, the arm 3AR and the bucket 3BK, which are described above in the working machine 33, which is a hydraulic excavator, are aspects of moving parts of the working machine 33.

[0306] It should be noted that the 3BK bucket serves as a configuration example of a "working tool" according to the present disclosure. The 3BK bucket includes a 3BKT cutting edge, which is used for excavation work or the like. Cabin configuration

[0307] Fig. Figure 37 is a diagram illustrating a configuration of the cab of the working machine according to the fourth embodiment.

[0308] As in Fig. As illustrated in Figure 37, the cabin 332A is equipped with control levers 3L1 and 3L2, foot pedals 3F1 and 3F2 and drive levers 3R1 and 3R2.

[0309] The control lever 3L1 and the control lever 3L2 are located to the left and right of each seat 3ST in cabin 332A. The foot pedal 3F1 and the foot pedal 3F2 are located on the floor surface in front of seat 3ST in cabin 332A.

[0310] An example of an operating pattern that specifies a correspondence relationship between an input operation on control levers 3L1 and 3L2, the travel levers 3R1 and 3R2 and the operation of the working machine 33, which is a hydraulic excavator, is as follows.

[0311] The control lever 3L1, located on the left side relative to the forward direction of the cab, is an operating mechanism for implementing a rotary movement of the upper rotating body 332 and an excavation / dumping operation of the arm 3AR. Specifically, when the operator of the machine 33 tilts the control lever 3L1 forward, the arm 3AR performs the dumping operation. When the operator of the machine 33 tilts the control lever 3L1 backward, the arm 3AR performs the excavation operation. Furthermore, the upper rotating body 332 rotates clockwise when the operator of the machine 33 tilts the control lever 3L1 to the right. Additionally, the upper rotating body 332 rotates counterclockwise when the operator of the machine 33 tilts the control lever 3L1 to the left.The upper rotating body 332 can rotate clockwise and counterclockwise in response to tilting the control lever 3L1 in the forward-backward direction, and the arm 3AR can perform the dumping or excavation operation in response to tilting the control lever 3L1 in the left-right direction.

[0312] The control lever 3L2, located on the right side relative to the forward direction of the cab, is an operating mechanism for performing the digging / dumping operation of the bucket 3BK and a raising / lowering operation of the boom 3BM. Specifically, when the operator of the machine 33 tilts the control lever 3L2 forward, the boom 3BM is lowered. When the operator of the machine 33 tilts the control lever 3L2 backward, the boom 3BM is raised. Furthermore, when the operator of the machine 33 tilts the control lever 3L2 to the right, the bucket 3BK is dumped. Additionally, when the operator of the machine 33 tilts the control lever 3L2 to the left, the bucket 3BK is excavated.

[0313] The drive levers 3R1 and 3R2 are operating mechanisms for controlling the operation of the lower chassis 331, that is, for controlling the movement of the work machine 33. The drive lever 3R1, located on the left side of the cab in the forward direction, corresponds to the rotary drive of the left track 3CL of the lower chassis 331. Specifically, when the operator of the work machine 33 tilts the drive lever 3R1 forward, the left track 3CL rotates in the forward direction. When the operator of the work machine 33 tilts the drive lever 3R1 backward, the left track 3CL rotates in the reverse direction.

[0314] The drive lever 3R2, located on the right side of the cab in the forward direction, corresponds to the rotary drive of the right track 3CR of the lower undercarriage 331. Specifically, when the operator of the machine 33 tilts the drive lever 3R2 forward, the right track 3CR rotates in the forward direction. When the operator of the machine 33 tilts the drive lever 3R2 backward, the right track 3CR rotates in the reverse direction. The foot pedals 3F1 and 3F2 are each locked to the drive levers 3R1 and 3R2, respectively, and the machine can also be driven using the foot pedals 3F1 and 3F2.

[0315] The operating patterns described above are merely an example, and the present invention is not limited to the above modes, which depend on the type of hydraulic excavator or the like.

[0316] It should be noted that the information relating to Fig. The working machine described in 36 may not include the GNSS antennas 3G1 and 3G2, depending on the embodiment. Functional configuration of the playback system

[0317] Fig. Figure 38 is a diagram illustrating a functional configuration of a playback system according to the fourth embodiment.

[0318] Functions of the playback system 310 according to the fourth embodiment are described below with reference to Fig. 38 described.

[0319] As in Fig. As illustrated in Figure 38, the playback system 310 includes a CPU 3100, a memory 3101, a display unit 3102, an operator receiver 3103, a communication interface 3104, and a data storage unit 3105. The CPU (central processing unit) 3100 can be a processor, such as an FPGA or a GPU, instead of a CPU.

[0320] The CPU 3100 is a processor that controls the overall operation of the 310 playback system. Various functions of the CPU 100 are described below.

[0321] Memory 3101 is a so-called main memory device. Instructions and data required for the operation of the CPU 3100 based on a program are loaded into memory 3101.

[0322] The display unit 3102 is a display device capable of displaying information in a visible manner, and is, for example, a liquid crystal display, an organic EL display, or the like.

[0323] The 3103 operating receiver unit is an input device and is, for example, a general mouse, keyboard, touch sensor or the like.

[0324] The communication interface 3104 is a communication interface for communication with the data logger 320.

[0325] The data storage device 3105 is an auxiliary storage device and is, for example, a hard disk drive (HDD), a solid-state drive (SSD), or the like. The data storage device 3105 records protocol information 3TL received from the data logger 320, the vehicle type of the work machine 33, a work machine model 3TM, which is a pre-prepared 3D model for each model, and the like. The work machine model 3TM is described below.

[0326] In addition, the data storage device 3105 also records a unit work prediction model 3PM1 and an element work prediction model 3PM2, which are trained machine learning models used to estimate the work content of the work machine 33, a heat map (3H1 and 3H2) generated during the estimation process, the estimated work content 3R of the work machine 33, three-dimensional point group information 3MAP representing the terrain information of a work site, and the like. It should be noted that the unit work prediction model 3PM1, the element work prediction model 3PM2, and the heat map (3H1, 3H2) are described further below.The three-dimensional point group information 3MAP includes, for example, three-dimensional point group information measured at a construction site using, for example, a drone or the like, information obtained by sequentially updating terrain information from cutting edge coordinate information of the working machine 33 in operation, and the like, and is recorded in the data storage 3105.

[0327] The functions of the CPU 3100 of the playback system 310 according to the fourth embodiment are described in detail. The CPU 3100 operates based on a predetermined program to perform functions as a capture unit 31000, a receive unit 31001, an extraction unit 31002, an image generation unit 31003, an estimation unit 31004, a determination unit 31005, a synchronization unit 31006, a display control unit 31007, and a labeling unit 31008.

[0328] It should be noted that the predetermined program can serve to implement some of the functions to be performed by the playback system 310. For example, the program can be used to perform a function in combination with other programs already stored in the data memory 3105 or in combination with other programs implemented in other devices. In other embodiments, the playback system 310 can additionally or instead of the above configuration include a user-defined highly integrated circuit (LSI) such as a programmable logic device (PLD).

[0329] Examples of PLDs include programmable array logic (PAL), generic array logic (GAL), complex programmable logic device (CPLD), and field-programmable gate array (FPGA). In this case, some or all of the functions implemented by the processor can be realized by the integrated circuit.

[0330] The acquisition unit 31000 captures the log information 3TL to be played back from a multitude of individual log information 3TL recorded and collected in the data storage unit 3105. It is assumed that the multitude of individual log information 3TL recorded in the data storage unit 3105 are for specific files recorded with different filenames. For example, the acquisition unit 31000 captures two individual log information 3TLs for comparison. The following example describes a case in which the acquisition unit 31000 captures two individual log information 3TLs for comparison. One of the two log information pieces captured by the acquisition unit 31000 is a log information that the operator designates as the playback target using a filename or similar identifier.Hereinafter, this log information is referred to as designated log information 3TL1 (first log information). The other of the two log information pieces is log information that is played back simultaneously with the playback based on designated log information 3TL1. Hereinafter, this log information is referred to as compared log information 3TL2 (second log information). Compared log information 3TL2 can be log information that has been pre-selected as a "model case" of the work content for each of various work contents and conditions, based on, for example, good fuel efficiency, short working time, driving by an expert, and other considerations.Alternatively, the compared log information 3TL2 could, for example, be a past log entry recorded during an operation previously performed by the same operator as the designated log information 3TL1. In this case, comparing the two log entries could be used to analyze, for example, the operator's skill level.

[0331] The acquisition unit 31000 according to the present embodiment automatically searches for the compared protocol information 3TL2, which is suitable for comparison with the designated protocol information 3TL1, and acquires it based on a determination result by the determination unit 31005 described below. However, the acquisition unit 31000 according to another embodiment is not limited to the above mode, and, for example, the acquisition unit 31000 can acquire protocol information designated by the operator using a file name or the like as the compared protocol information 3TL2. Additionally, if only one compared protocol information 3TL2 is recorded, the acquisition unit 31000 can acquire the compared protocol information 3TL2.

[0332] The receiving unit 31001 receives a predetermined playback instruction from the operator of the playback system 310. For example, the receiving unit 31001 receives a playback instruction for the work machine 33 from the operator of the playback system 310.

[0333] The extraction unit 31002 extracts angular information, which is used for playback for the working machine 33, from the captured log information TL. The angular information is an example of “operational information (motion information)” according to the present disclosure.

[0334] The image generation unit 31003 sequentially generates an operating image representing the operation of the working machine 33 by sequentially applying operating information, based on the protocol information 3TL, to a working machine model 3TM, which is a three-dimensional model of the working machine 33. It also sequentially generates a motion trajectory image representing the motion trajectory of a predetermined part of the working machine 33 in conjunction with the operating image, based on the operating information. Additionally, the image generation unit 31003 can generate a motion trajectory image if the work content estimated by the estimating unit 31004 corresponds to the work content designated by the labeling unit 31008.It should be noted that the motion trajectory is a path which, when the operating image is displayed as an animation, starts from the position of the relevant part at a time point a predetermined time earlier than the time of the displayed operating image, proceeds through the part's positions at respective times within a predetermined time interval until the current time, and ends at the part's position at the current time. The motion trajectory image can, for example, be a multitude of images of a predetermined shape representing the respective positions, such as spherical three-dimensional images or linear or planar images connecting the positions. The predetermined part of the machine 33 for which the motion trajectory is displayed can, for example, be a blade edge 3BKT of a blade 3BK (working tool) or a predetermined position of the upper rotating body 332.Furthermore, for example, in the case of another working machine 33, the position can be a shovel edge of a shovel, a predetermined position such as a ground contact position of a crawler track or tire, or a center coordinate of the working machine 33.

[0335] The image generation unit 31003 can generate a partial image, depicting a predetermined part of the working tool 332B of the machine 33 prior to a predetermined time, in conjunction with the operating image. The partial image is a stereoscopic image representing the part at a time that is a predetermined time earlier than the time of the displayed operating image when the operating image is shown as an animation. The predetermined part could, for example, be the bucket 3BK. In this case, the partial image is a stereoscopic image representing the bucket 3BK and can be displayed as a translucent image, for example, superimposed on the operating image.

[0336] The image generation unit 31003 can change the display color of the motion trajectory image, the size and thickness of the image representing the motion trajectory, and the like, based on at least one of the work content estimated based on the protocol information 3TL, the accuracy of position information included in the protocol information 3TL, information representing whether an automatic operating control is active in the work machine 33 included in the protocol information 3TL, a work quantity of the work machine 33 based on the protocol information 3TL, and information representing a work load of the work machine 33 based on the protocol information 3TL.The accuracy of the position information can be impaired or become unstable if the number of satellites acquired by the GNSS is low or if the baseline length to the reference point is long. The display mode of the motion trajectory can be changed between cases where the accuracy of the position information is high and cases where it is low. A root mean square (RMS) value or a dilution-of-precision (DOP) value can be used as an index of the accuracy of the position information. The automatic operating control is a control system for fully or semi-automatic control of the operation of the machine 33. The display mode of the motion trajectory can be changed depending on whether the automatic operating control is active.The working quantity of the machine 33 is a value that indicates the magnitude of, for example, the excavation depth, the load during excavation, the weight of the load, and the like. The working load of the machine 3 is a value that indicates the magnitude of the load applied to the work, which is calculated, for example, from the hydraulic pressure of the cylinder, the hydraulic pressure of the pump, the motor torque, the tractive force, and the like.

[0337] The estimation unit 31004 estimates the work content of the work machine 33 at any given time from the recorded protocol information 3TL.

[0338] The determination unit 31005 determines whether the compared protocol information 3TL2 is suitable for comparison with the designated protocol information 3TL1, based on the information included in the designated protocol information 3TL1.

[0339] The synchronization unit 31006 performs processing to synchronize the playback of the work machine model 3TM based on the designated protocol information 3TL1 and the playback of the work machine model 3TM based on the compared protocol information 3TL2. Specifically, the synchronization unit 31006 specifies the playback start time on the time series of the animation of the work machine model 3TM based on the work content estimated by the estimation unit 31004.

[0340] The display control unit 31007 displays the operating image and the motion trajectory image, generated by the image generation unit 31003, superimposed on the display unit 3102. The display control unit 31007 displays the partial image and the operating image superimposed on the display unit 3102. For example, the display control unit 31007 can display an image (terrain image) on the display unit 3102 that represents terrain based on the three-dimensional point group information 3MAP.

[0341] The designation unit 31008 denotes work content for which a motion trajectory image is generated in response to an instruction from the operator. The work content can be designated, for example, by unit work, by element work, by a combination of unit work and element work, or similar methods. Processing sequence by the playback system

[0342] The following describes a specific processing sequence performed by the playback system 310, with reference to Fig. 39, Fig. 40, Fig. 41, Fig. 42, Fig. 43, Fig. 44, Fig. 45, Fig. 46, Fig. 47 to Fig. 48 described in detail.

[0343] The process of the in Fig. The processing illustrated in Figure 39 begins when the operator of the playback system 310 starts a dedicated application.

[0344] When the operator performs an operation to start the dedicated application, the capture unit 31000 of the CPU 3100 loads the memory 3101 and captures the designated protocol information 3TL1, which is designated as the playback target (step 3S00).

[0345] Here, the protocol information 3TL (designated protocol information 3TL1, compared protocol information 3TL2) is presented with reference to Fig. 40, Fig. 41 to Fig. 42 described.

[0346] As in Fig. 40, Fig. 41 to Fig. As illustrated in Figure 42, the 3TL protocol information includes work machine identification information. The work machine identification information is specifically an individual identification number for the individual identification of the work machine 33. Fig. 40, Fig. 41 to Fig. 42 It is assumed that the work machine identification information is assigned in such a way as to correspond to the vehicle type, model, type number, machine number, and the like of the work machine 33, indicating a hydraulic excavator, wheel loader, bulldozer, or the like. The work machine identification information may be a number or may be an alphabet, a symbol, a combination thereof, or the like.

[0347] As in Fig. As illustrated in Figure 40, the log information 3TL includes information representing the position and orientation of the machine 33 at any given time, as well as angular information of the moving parts of the machine 33. Specifically, the position of the machine 33, the roll angle of the machine 33 (as the tilt of the machine body in the left-right direction), the pitch angle (as the tilt of the machine body in the forward-backward direction), the rotation angle, the boom angle, the arm angle, and the bucket angle are recorded in the log information 3TL at each point in time. Here, the data logger 320 installed in the machine 33 specifies and records the position of the machine 33 based on position measurement information representing latitude, longitude, and altitude, information obtained, for example, by reception via the GNSS antennas 3G1 and 3G2.The data logger 320 calculates and records the roll angle and pitch angle of the working machine 33 based on a measurement result from an inertial measurement unit (IMU) installed in the working machine 33. The data logger 320 calculates and records the rotation angle of the upper rotating body 332 based on the position measurement information obtained from each of the GNSS antennas 3G1 and 3G2 provided in the upper rotating body 332. Furthermore, the data logger 320 calculates and records the boom angle, arm angle, and blade angle based on the extension and retraction degrees of each of the boom cylinder 3SL1, arm cylinder 3SL2, and blade cylinder 3SL3. The IMU can be mounted on the boom, arm, blade, and upper rotating body to be used, for example, to acquire the boom angle, arm angle, blade angle, and rotation angle.

[0348] The position, roll angle, and pitch angle are information required to specify the position and orientation of the working machine 33 itself. Therefore, for example, in an embodiment where only the movements of the moving parts of the working machine 33—that is, the upper rotary body 332, the boom 3BM, the arm 3AR, and the bucket 3BK—are represented as an animation, and the position and orientation of the working machine 33 itself are not represented, the position, roll angle, and pitch angle information need not be included in the protocol information.

[0349] The in Fig. The 40 illustrated protocol information 3TL corresponds to "operational information" as defined in this disclosure. Alternatively, the information contained in Fig. 40 illustrated protocol information 3TL information, which represents the operation of the working machine 33 that can be visually perceived from outside the working machine 33, and can also be referred to as external view operating information. That is, in the present embodiment, the operating information or external view operating information includes the position information, roll angle, pitch angle, yaw angle, boom angle, arm angle, and bucket angle of the working machine 33. In the present embodiment, the operating information or external view operating information has the characteristic that, as information used in estimating the work content, it is less likely to be affected by a difference in the vehicle rank of the working machine 33, in contrast to the drive mechanism information and operator information described later.

[0350] As in Fig. As illustrated in Figure 41, the 3TL log information includes the PPC pressures as a pilot pressure, indicating the degrees of input to the control levers 3L1 and 3L2 and the like by the operator at any given time, that is, the degrees of lever tilt and the degrees of pedal pressure. Specifically, the PPC pressures of the control levers 3L1 and 3L2, the travel levers 3R1 and 3R2, or the foot pedals 3F1 and 3F2, corresponding to the respective operating modes of counterclockwise / clockwise rotation, arm extension / deployment, boom raising / lowering, bucket extension / deployment, right track forward / reverse movement, and left track forward / reverse movement by the operator, are recorded in the 3TL log information at each point in time. Fig. The 41 illustrated time points correspond to those in Fig. 40 illustrated time points. In the present embodiment, each in Fig. 41 illustrated information, also referred to as operating information.

[0351] As in Fig. As illustrated in Figure 42, the log information 3TL includes information representing the status of the main drive mechanisms, such as the engine and hydraulic pumps of the working machine 33, at any given time. Specifically, the log information 3TL records the engine coolant temperature, engine power, instantaneous fuel consumption, and hydraulic pump oil temperature for each time point. The information in Fig. The 42 illustrated time points correspond to those in Fig. 40 and Fig. 41 illustrated time points. In the present embodiment, each in Fig. 42 illustrated information, also referred to as drive mechanism information.

[0352] Returning to Fig. 39 The estimation unit 31004 of the CPU 3100 estimates the work content of the work machine 33 at each time based on the designated protocol information 3TL1 acquired in step 3S00 (step 3S01).

[0353] Here, a procedure is described in which the estimation unit 31004 estimates the work content of the working machine 33 from the protocol information 3TL, with reference to Fig. 43 described. The estimation unit 31004 estimates the work content of the machine 33 for both unit work and element work. Unit work is work performed to achieve a specific purpose. Element work is work that specifies a series of operations or tasks, which are elements that constitute the unit work and are classified according to purpose.

[0354] Examples of unit work sections include "excavation and loading", "plowing", "sloping (from below)", "load gathering", "driving" and "stopping", which are in Fig. 43 are illustrated, as well as “trench excavation”, “backfilling”, “sloping (from above)” and the like.

[0355] "Excavation and loading" refers to work involving the excavation and removal of earth and sand or rock, and the loading of this material onto the platform of a transport vehicle. "Excavation and loading" encompasses all work performed as a single unit, including excavation, turning the loaded vehicle, unloading, turning the unloaded vehicle, waiting for unloading, and compacting the loading platform.

[0356] "Plowing" is the work of removing excess soil relief to a predetermined height. Plowing is a unit operation that includes digging and unloading, or digging, turning with a load, unloading, and turning without a load, and may include push leveling and sweeping.

[0357] "Below slope" is the work of creating a slope by the work machine 33, which is located below the target location. Below slope is a unitary work that includes compaction, excavation, loaded turning, unloading, and unloaded turning, and may include thrust grading.

[0358] "Load collection" refers to work involving the gathering of soil and sand resulting from excavation or similar processes before loading it onto a transport vehicle. Load collection is a single, unified task that includes excavation, turning the loaded vehicle, unloading, turning the unloaded vehicle, and may also include grading.

[0359] "Driving" refers to work in which the working machine 33 is moved. Driving as a unit of work is unit work that includes driving as an element of work.

[0360] "Stopping" is a state of halting for a predetermined time or longer, without any soil, sand, or rock in the shovel 3BK. Stopping as a unit of work is unit of work that includes stopping as an element of work.

[0361] "Trench excavation" is the work of digging and scraping the soil into a long and narrow trench. Trench excavation is a single operation that includes excavation, turning with a load, unloading, turning without a load, and may also include grading.

[0362] "Backfilling" is the process of filling a trench or hole already formed in the ground with soil and sand, and backfilling to level the surface. Backfilling is a single operation that includes excavation, turning with a load, unloading, compaction, and turning without a load, and may include grading and sweeping.

[0363] "Sloping" (from above) is the work of creating a slope by the work machine 33, which is located above the target location. Sloping (from above) is a unit operation that includes compaction, excavation, loaded turning, unloading, and unloaded turning, and may include thrust grading.

[0364] Examples of element work sections include "excavation", "loaded turning", "waiting for unloading", "unloading", "unloaded turning", "loading platform pressing", which are in Fig. 43 are illustrated, as well as “compaction”, “thrust leveling”, “sweeping” and the like.

[0365] The term "excavation" refers to work in which earth, sand or rock is dug out and removed using the 3BK shovel.

[0366] “Loaded rotation” refers to work in which the upper rotating body 332 is rotated while the removed earth and sand or rock is held in the bucket 3BK.

[0367] “Waiting for unloading” refers to work where one waits for a transport vehicle to be loaded, while holding the removed earth and sand or rock in the 3BK bucket.

[0368] "Unloading" refers to work in which the removed earth and sand or rock from the 3BK bucket is unloaded onto a transport vehicle or at a predetermined location.

[0369] “Unloaded turning” refers to work in which the rotating body 332 is turned in the bucket 3BK without soil, sand or rock.

[0370] The term "loading platform pressing" refers to work in which earth and sand loaded on the loading platform of a transport vehicle are pressed from above with the 3BK bucket to smooth the earth and sand.

[0371] "Compacting" refers to work in which soil and sand are pushed into the disturbed ground using a shovel to shape and solidify the soil.

[0372] "Leveling" refers to work in which earth and sand are swept and leveled using the bottom surface of the 3BK shovel.

[0373] "Sweeping" refers to work in which earth and sand are swept and leveled using a side surface of the 3BK shovel.

[0374] The estimation unit 31004 obtains a time series of probabilities regarding unit work when the protocol information TL is input into the unit work prediction model 3PM1 in chronological order. The unit work prediction model 3PM1 is a model that outputs the probability regarding unit work when the protocol information 3TL is input during training using, for example, teacher data, and can be stored, for example, in data store 3105.

[0375] The estimation unit 31004 obtains a time series of probabilities regarding element work when the protocol information 3TL is input into the element work prediction model 3PM2 in chronological order. The element work prediction model 3PM2 is a model that outputs the probability with respect to element work when the protocol information 3TL is input during training using, for example, teacher data, and can be stored, for example, in data storage 3105.

[0376] The estimation unit 31004 smooths the time series of probabilities by applying a time-averaging filter to the time series of probabilities with respect to unit work and the time series of probabilities with respect to element work, and generates a unit work heatmap 3H1, which represents the smoothed time series of probabilities with respect to unit work, and an element work heatmap 3H2, which represents the smoothed time series of probabilities with respect to element work, as shown in Fig. Figure 43 illustrates this. Heatmaps 3H1 and 3H2 are maps in which colors representing the probabilities of job classifications are applied to a plane, with the vertical axis representing the job classifications and the horizontal axis representing time points based on the smoothed time series of probabilities. For example, the colors relating to the heatmaps might be closer to blue for a lower probability of job classification and closer to red for a higher probability of job classification. The estimation unit 31004 stores heatmaps 3H1 and 3H2 in data memory 3105.

[0377] Estimation unit 31004 specifies a period in which the probability of unit work is dominant, based on the smoothed time series of probabilities, and estimates the work content of machine 33 during that period. For example, during a period in which the probability of "excavation and loading," which is unit work, is dominant, the work content of machine 33 is estimated as "excavation and loading." Similarly, estimation unit 31004 specifies a period in which the probability of element work is dominant, based on the smoothed time series of probabilities, and estimates the work content of machine 33 during that period. For example, during a period in which the probability of "excavation," which is element work, is dominant, the work content of machine 33 is estimated as "excavation."The estimation unit 31004 stores information about the estimated work content R for the working machine 33 in the data storage unit 3105.

[0378] Returning to Fig. 39 then captures the compared protocol information 3TL2, which is suitable for comparison with the designated protocol information TL1 captured in step 3S00 (step 3S02).

[0379] At this point, the determination unit 31005 performs a determination processing sequence to determine whether the multitude of individual compared protocol information 3TL2, recorded in the data store 3105, are suitable for comparison with the designated protocol information 3TL1 acquired in step 3S00. In this determination processing sequence, the determination unit 31005 first selects one of the multitude of individual compared protocol information 3TL2 that were pre-recorded in the data store 3105.Then the determination unit 31005 determines whether, for example, the vehicle rank, the amount of change of the rotation angle (the size of the swivel), the bucket height at the time of waiting to be unloaded by the working machine 33 and the like are substantially the same, based on the log information 3TL, and if it determines that they are not substantially the same, it selects a new compared log information 3TL2 and performs the same determination processing as described above.

[0380] In addition, according to another embodiment, the determination unit 31005 can limit the candidates for reproduction based on a work position, a working time, a work classification or the like, in addition to the determination processing described above.

[0381] Returning to Fig. 39 Next, the synchronization unit 31006 performs processing to synchronize the animation playback for the work machine model 3TM based on the designated protocol information 3TL1 and the animation playback for the work machine model 3TM based on the compared protocol information 3TL2. In particular, the synchronization unit 31006 specifies the playback start time on the time series in the animation of the work machine model 3TM based on the designated protocol information 3TL1 and the playback start time on the time series in the animation of the work machine model 3TM based on the compared protocol information 3TL2.

[0382] One method for specifying the playback start time to synchronize the two is as follows. That is, the synchronization unit 31006 extracts the unit work toggling time in the designated log information 3TL1 based on the unit work heatmap 3H1 for the designated log information 3TL1, generated by the estimating unit 31004. Next, the synchronization unit 31006 extracts the unit work toggling time in the compared log information 3TL2 based on the unit work heatmap 3H1 for the compared log information 3TL2, generated by the estimating unit 31004. The synchronization unit 31006 selects one of the extracted times and specifies the selected time as the playback start time of each animation.

[0383] The playback system 310 according to another embodiment is not limited to the above mode. The playback system 310 (the synchronization unit 31006) according to another embodiment can compare a combination of angular information of the boom, arm, bucket, and the like in the compared protocol information 3TL2 with a combination of angular information of the boom, arm, bucket, and the like in the designated protocol information 3TL1, extract a time point at which both angles are close to each other, and specify the time point, for example, as the playback start time of each animation.

[0384] Next, the receiving unit 31001 receives a playback instruction from the operator (step 3S04). One aspect of the playback instruction can be an action such as pressing a playback button. Additionally, the playback instruction can include information that serves as the starting point for the playback, such as a time, the position of the machine 33, and various events such as the occurrence of an abnormality in the machine 33. When the receiving unit 31001 receives the playback instruction, the labeling unit 31008 can be used to label the work content as the generation target of the motion trajectory image.

[0385] Next, the acquisition unit 31000 selects and reads the work machine model 3TM according to the referenced work machine identification information from the data storage 3105 based on the work machine identification information as the type of work machine 33 received from the receiving unit 31001 (step 3S05).

[0386] Here, the 3TM work machine model is described with reference to Fig. 44 described.

[0387] As in Fig. As illustrated in Figure 44, the working machine model 3TM comprises information that includes the working machine identification information and an external shape 3D model 3M0 of the working machine 33 specified by the working machine identification information. The external shape 3D model 3M0 is a 3D model representing the working machine 33 and is constructed for each part of the working machine 33, such as the lower running gear and the upper rotating gear. For example, the external shape 3D model 3M0 represents the shape of the working machine 33.For example, the outer shape 3D model 3M0 includes an outer shape model 3M01 with a lower running body representing the lower running body 331 of the working machine 33, an outer shape model 3M02 with an upper rotating body representing the upper rotating body 332, a boom outer shape model 3M03 representing the boom 3BM, an arm outer shape model 3M04 representing the arm 3AR, and a bucket outer shape model 3M05 representing the bucket 3BK.

[0388] Returning to Fig. 39 Extraction unit 31002 extracts the information used for playback from each of the designated protocol information 3TL1 and the compared protocol information 3TL2 (step 3S06). For example, extraction unit 31002 extracts various types of angle information, such as boom angle, arm angle, and blade angle, as information used for playback. The in Fig. The 41 illustrated pilot prints can be extracted as the information used for playback. Additionally, in steps 3S00 and 3S02, only information used for playback can be captured.

[0389] Next, the image generation unit 31003 and the display control unit 31007 perform a simultaneous playback of an animation of the work machine model 3TM based on the designated protocol information 3TL1 and an animation playback of the work machine model 3TM based on the compared protocol information 3TL2 (step 3S07). Here, the image generation unit 31003 generates an operating image representing the operation of the work machine 33, while various types of information recorded in the designated protocol information 3TL1 are applied to the work machine model 3TM in chronological order of the timestamps from the playback start time specified in the synchronous processing in step 3S03 (step 3S07a).Simultaneously with the generation of the operating image for the animation, the image generation unit 31003 generates an operating image representing the operation of the working machine 33, while various types of information recorded in the compared protocol information 3TL2 are applied to the working machine model 3TM in chronological order of the timestamps from the playback start time specified in step 3S03 in the synchronous processing (step 3S07a).

[0390] The following describes in detail the contents of the operational image generation processing by the image generation unit 31003. The image generation unit 31003 modifies the angle of the corresponding section of the outer shape 3D model 3M0 based on various types of angular information, such as the rotation angle and the boom angle, specified in the log information 3TL (the designated log information 3TL1 and the compared log information 3TL2). For example, the image generation unit 31003 represents the position and orientation of the bucket 3BK of the working machine 33 by tilting the bucket outer shape model 3M05 about the axis of rotation defined at a position of the connection with the arm outer shape model 3M04 to the bucket angle specified in the log information 3TL.

[0391] Similarly, the image generation unit 31003 reproduces the position and orientation of the arm 3AR of the working machine 33 by tilting the arm outer shape model 3M04 about the axis of rotation defined at a position of the connection with the boom outer shape model 3M03 to the arm angle specified in the protocol information 3TL.

[0392] Similarly, the image generation unit 31003 reproduces the position and orientation of the upper rotary body 332 of the working machine 33 by tilting the outer shape model 3M02 of the upper rotary body about the axis of rotation defined at a position of the connection with the outer shape model 3M01 of the lower driving body to the angle of rotation specified in the protocol information 3TL.

[0393] Similarly, the image generation unit 31003 reproduces the position of the lower carriage 331 of the working machine 33 by tilting the outer shape model 3M01 of the lower carriage about the roll rotation axis defined in the outer shape model 3M01 of the lower carriage to the roll angle specified in the protocol information 3TL and tilting the outer shape model 3M01 of the lower carriage about the pitch rotation axis defined in the outer shape model 3M01 of the lower carriage to the pitch angle specified in the protocol information 3TL.

[0394] Furthermore, according to the fourth embodiment, the playback system 310 can reproduce an animation of the driving movement of the working machine 33 based on the PPC presses of the forward / reverse movement of the right crawler track and the forward / reverse movement of the left crawler track at any time included in the log information 3TL.

[0395] Specifically, the outer shape 3D model 3M0 is moved forward, backward, forward left and right, and backward left and right based on the PPC pressures of the forward / backward movement of the right track and the forward / backward movement of the left track. For example, the outer shape 3D model 3M0 moves forward based on the numerical values ​​of the PPC pressures of the forward movement of the right track and the forward movement of the left track. The movement speed can be changed based on the numerical values ​​of the PPC pressures.

[0396] The outer shape 3D model 3M0 is moved backward based on the numerical values ​​of the PPC pressures for the reverse movement of the right and left track. Furthermore, based on the difference in the numerical values ​​of the PPC pressures between the forward movement of the right and left track, the outer shape 3D model 3M0 is moved to curve in the forward left-right direction. For example, if the numerical value of the PPC pressure for the forward movement of the right track is greater than the numerical value of the PPC pressure for the forward movement of the left track, a movement to curve forward to the left is implemented. The speed of movement and the size of the curve can be changed depending on the numerical values ​​of the respective PPC pressures and the difference between these values.

[0397] Similarly, based on the difference in the numerical values ​​of the PPC pressures between the reverse movement of the right and left track, the outer shape 3D model 3M0 is moved to curve in the reverse left-right direction. For example, if the numerical value of the PPC pressure for the reverse movement of the right track is greater than the numerical value of the PPC pressure for the reverse movement of the left track, a movement to curve in the reverse left direction is implemented. The speed of movement and the size of the curve can be changed depending on the numerical values ​​of the respective PPC pressures and the difference between these values.

[0398] The movement of the work machine 33 can be animated more precisely by playing back the animation using position information in addition to the PPC (Power Point Control) inputs for the forward / reverse movement of the right and left tracks. In this case, the speed and position of the movement of the work machine 33 can be expressed more accurately using the position information. Additionally, by playing back the animation of the work machine 33 based on the roll angle, pitch angle, or both, in addition to the PPC inputs for the forward / reverse movement of the right and left tracks, the tilt of the work machine 33 in the left-right direction or the tilt of the work machine 33 in the forward-backward direction during movement can be displayed.

[0399] In step 3S07, the image generation unit 31003 sequentially generates a motion trajectory image representing the motion trajectory of a predetermined part (the blade edge 3BK of the blade 3BK) of the working machine 33, in conjunction with the operating image based on the operating information (step 3S07b). The display control unit 31007 further displays the operating image and the motion trajectory image superimposed on the display unit 3102. In step 3S07, the image generation unit 31003 sequentially generates a partial image representing a predetermined part (blade 3BK) of the working tool 332B, in conjunction with the operating image (step 3S07c). In step 3S07c, the display control unit 31007 further displays the operating image, the motion trajectory image, and the partial image superimposed on the display unit 3102.In step 3S07, the display control unit 31007 displays a terrain image on the display unit 3102 based on the three-dimensional point group information 3MAP in combination with the operating image, the motion trajectory image, and the partial image (step 3S07d). Fig. Figure 45 schematically illustrates a display image 3D1, which is shown on the display unit 3102 by the display control unit 31007. The display image 3D1 includes an operating image 3D11, a motion trajectory image 3D12, a partial image 3D13, and a terrain image 3D14, based on the designated protocol information 3TL1. Fig. Figure 46 schematically illustrates a display image 3D2, which is displayed on the display unit 3102 by the display control unit 31007. The display image 3D2 includes the operating image 3D11, the motion trajectory image 3D12, the sub-image 3D13, the terrain image 3D14 based on the designated protocol information 3TL1, and also includes an operating image 3D21, a motion trajectory image 3D22, and a sub-image 3D23 based on the compared protocol information 3TL2.

[0400] While the animation of the operation of the two work machines 33 is playing, the image generation unit 31003 determines whether the animation playback should be terminated (step 3S08). For example, if a command to terminate playback is received based on the pressing of a stop button or the like, it is determined that the animation playback should be terminated. It can also be determined that the animation playback should be terminated after a predetermined time interval since the start of the animation playback. If the animation playback is not to be terminated (step 3S08; NO), the image generation unit 31003 continues the simultaneous animation playback for the two work machine models 3TM. On the other hand, if the animation playback is to be terminated (step 3S08; YES), the image generation unit 31003 terminates the animation playback processing.

[0401] In the reference to Fig. In the processing sequence described in 39, steps 3S00, 3S01, 3S03, 3S04, 3S05, 3S06 and 3S08 are not the essential configuration of the playback system 310, and other embodiments may not include these steps. More display examples

[0402] Fig. Figure 47 schematically illustrates a display image 3D3, which is shown on the display unit 3102 by the display control unit 31007. The display image 3D3 includes an operating image 3D31, a motion trajectory image 3D32, and a terrain image 3D34, based on the protocol information 3TL1. The display mode, for example, the display color of the motion trajectory image 3D32, which is shown in Fig. The illustration in section 47 is modified based on the work content. Fig. Figure 48 schematically illustrates a display image 3D4, which is shown on the display unit 3102 by the display control unit 31007. The display image 3D4 includes an operating image 3D41, a motion trajectory image 3D42, and a terrain image 3D44, based on the protocol information 3TL1. The display mode, for example, the display color of the motion trajectory image 3D42, which is shown in Fig. As illustrated in 48, the change is based on the accuracy of the position information. Operation and impacts

[0403] As described above, the playback system 310 according to the fourth embodiment includes: the acquisition unit 31000, which is configured to acquire the protocol information 3TL of the working machine 33, which is associated with a time point; the image generation unit 31003, which is configured to sequentially generate the operating image 3D11, 3D21, which represents an operation of the working machine 33, by sequentially applying operating information representing the operation of the working machine 33, based on the protocol information 3TL, to a three-dimensional model of the working machine 33, and sequentially generate the motion trajectory image 3D12, 3D22, which represents a motion trajectory of a predetermined part of the working machine 33, in conjunction with the operating image based on the operating information;and the display control unit 31007, which is configured to display the operating image and the motion trajectory image in a superimposed manner on the predetermined display unit 3102. According to this configuration, since the sequence of a series of operations can be easily recognized by the motion trajectory image, the work performed by the machine 33 can be analyzed more effectively. Modified example

[0404] The content of the 3TL protocol information according to the fourth embodiment ( Fig. 40, Fig. 41 to Fig. 42) is not limited to this in other embodiments. For example, if the working machine 33 is not a hydraulic excavator but another type of vehicle, log information 3TL is recorded according to the vehicle type. The other vehicle types are, for example, a wheel loader, a bulldozer, and the like.

[0405] Furthermore, the description is given under the assumption that the playback system 310 according to the fourth embodiment is installed at a location remote from the working machine 33 and is connected to the data logger 320 installed in the working machine 33 via the wide area communication network; however, other embodiments are not limited to this aspect.

[0406] For example, in another embodiment, part or all of the configuration of the playback system 310 can be installed inside the machine 33. In this case, the data logger 320 can transmit the log information 3TL to the playback system 310 via a network inside the machine 33 or the like, without using the wide area communication network. In this way, the operator entering the machine 33 can verify the movement of the machine 33 they are operating on-site by replaying the animation. Furthermore, the movement of the machine 33 can be replayed as a model case for the operator of the machine 33 to use as a guide.

[0407] The playback system 310 installed in the work machine 33 can acquire the protocol information 3TL of another work machine 33 via a wide area communication network or the like. In this way, the states of the work machines 33, except for the work machine 33 in which the playback system 310 is installed, can be displayed as an animation.

[0408] In addition, according to another embodiment, the playback system 310 can be installed at a location remote from the working machine 33 and can transmit video information generated by the animation playback processing to a monitor installed in the working machine 33 for display.

[0409] In another embodiment, one aspect of the playback instruction received from the operator may be, for example, a playback period. For instance, the playback period may include a playback start time and a playback end time. In this case, the playback system 310 plays back the work machine 33 during the received playback period. In another embodiment, determining the playback end time is not essential. For example, in another embodiment, only the playback start time may be received by the operator as a playback instruction, and playback may be performed for a certain period of time from the playback start time, playback may continue as long as the log information is available, or playback may be stopped in accordance with the occurrence of other various events.

[0410] The protocol information to be recorded 3TL ( Fig. 40, Fig. 41 to Fig. 42) do not need to be arranged in chronological order. In this case, the image generation unit 31003 can apply information from the protocol information 3TL, which is used for playback, to the work machine model 3TM in time series.

[0411] The steps of the various types of processing of the playback system 310, as described above, are stored on a computer-readable recording medium in the form of a program, and the computer reads and executes the program to implement the various types of processing described above. The computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Furthermore, this computer program can be distributed to the computer via a communication circuit, and the computer receiving this distribution can execute the program.

[0412] The program described above can be a program for executing a section of the functions described above. Furthermore, the functions described above can be implemented in combination with a program already recorded on the computer system, namely a so-called differential file, a differential program, or the like.

[0413] Some or all of the functions of the playback system 310 described above can be assigned to the working machine 33. For example, some or all of the functions of the acquisition unit 31000, the receiving unit 31001, the extraction unit 31002, the image generation unit 31003, the estimation unit 31004, the determination unit 31005, the synchronization unit 31006, the display control unit 31007, the labeling unit 31008, the memory 3101, the display unit 3102, the operator receiving unit 3103, the communication interface 3104, and the data storage unit 3105 can be assigned to the working machine 33.

[0414] Certain embodiments of the present disclosure have been described above, but all embodiments are merely illustrative and are not intended to limit the scope of protection of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the core of the invention. These embodiments and modifications are included in the scope of protection and core of the invention and are also included in the scope of protection of the invention described in the claims and their equivalents. Additional remarks

[0415] The display control system (playback system 310) according to the present disclosure is understood, for example, as follows. (10) A display control system (replay system 310) according to a tenth consideration includes: the acquisition unit 31000, which is configured to acquire the protocol information 3TL of the working machine 33, which is associated with a point in time; the image generation unit 31003, which is configured to sequentially generate an operating image (3D11, 3D21, 3D31, 3D41) representing an operation of the working machine by sequentially applying operating information representing the operation of the working machine, based on the protocol information, to a three-dimensional model (working machine model 3TM) of the working machine, and sequentially generate a motion trajectory image (3D12, 3D22, 3D32, 3D42) representing a motion trajectory of a predetermined part of the working machine, in conjunction with the operating image, based on the operating information;and the display control unit 31007, which is configured to display the operating image and the motion trajectory image in a superimposed manner on the predetermined display unit 3102. From this point of view and the following points, it is possible to analyze work performed by a working machine more effectively. (11) A display control system (replay system 310) according to an eleventh consideration is the display control system according to (10), which further includes: the estimating unit 31004, which is configured to estimate work content of the working machine at any time based on the log information; and the labeling unit 31008, which is configured to label the work content for which the motion trajectory image is to be generated, wherein the image generation unit 31003 generates the motion trajectory image when the estimated work content matches the labeled work content. (12) A display control system (reproduction system 310) according to a twelfth consideration is the display control system according to (10) or (11), wherein the image generation unit 31003 generates a partial image (3D13, 3D23) representing a predetermined part of a working tool of the working machine at a time earlier by a predetermined time interval, in conjunction with the operating image, and the display control unit 31007 displays the partial image superimposed on the operating image on the display unit. (13) A display control system (reproduction system 310) according to a thirteenth point is the display control system according to (10) to (12), wherein the predetermined part corresponds to a cutting edge of the working tool of the working machine. (14) A display control system (reproduction system 310) according to a fourteenth aspect is the display control system according to (10) to (12), wherein the predetermined part corresponds to a ground contact position of a track or tire of the working machine. (15) A display control system (reproduction system 310) according to a fifteenth consideration is the display control system according to (10) to (14), wherein the image generation unit changes a mode of the motion trajectory image based on at least one of the work content estimated on the basis of the protocol information, an accuracy of position information included in the protocol information, information representing whether an automatic operating control is active in the working machine included in the protocol information, a work quantity of the working machine based on the protocol information, and information representing a work load of the working machine based on the protocol information. Industrial applicability

[0416] According to the reproduction system and reproduction method of the present disclosure, it is possible to reproduce a movement of a working machine in an easily recognizable way.

[0417] According to the display control device and the display control method of the present disclosure, it is possible to analyze work performed by a working machine in more detail.

[0418] According to the display control system and the display control method of the present disclosure, it is possible to analyze work performed by a working machine more effectively. Reference symbol list

[0419] 11 Analysis Support System, 110 Playback System, 1100 CPU, 11000 Acquisition Unit, 11001 Receiving Unit, 11002 Extraction Unit, 11003 Image Generation Unit, 11004 Estimation Unit, 11005 Determination Unit, 11006 Synchronization Unit, 11007 Display Control Unit, 1101 Memory, 1102 Display Unit, 1103 Operator Receiver Unit, 1104 Communication Interface, 1105 Data Storage, 120 Data Logger, 13 Working Machine, 1H1 to 1H2 Heatmap, 1TM Working Machine Model, 1M1 Operator Unit 3D Model, 1TL Protocol Information, 1PM1 Unit Work Prediction Model, 1PM2 Element Work Prediction Model, 1R Estimated Work Content, 1D11, 1D21, 1D31 Operating Image 21 Analysis Support System, 210 Playback System, 2100 CPU, 21000 Acquisition Unit, 21001 Receiving Unit, 21002 Extraction Unit, 21003 Playback Unit, 21004 Estimation Unit, 21005 Determination Unit, 21006 Synchronization Unit, 21007 Playback Area Labeling Unit, 21008 Bird's-Eye View Image Display Unit, 2101 Memory, 2102 Display Unit, 2103 Operator Receiving Unit, 2104 Communication Interface, 2105 Data Storage, 220 Data Logger, 23 Working Machine, 2H1 to 2H2 Heatmap, 2TM Working Machine Model, 2TL Protocol Information, 2PM1 Unit Work Prediction Model, 2PM2 Element Work Prediction Model, 2R Estimated Work Content, 2D11, 2D21, 2D31 Operating image, 2D12, 2D22, 2D32 captured image 31 Analysis Support System, 310 Playback System, 3100 CPU, 31000 Acquisition Unit, 31001 Receiving Unit, 31002 Extraction Unit, 31003 Image Generation Unit, 31004 Estimation Unit, 31005 Determination Unit, 31006 Synchronization Unit, 31007 Display Control Unit, 31008 Labeling Unit, 3101 Memory, 3102 Display Unit, 3103 Operator Receiving Unit, 3104 Communication Interface, 3105 Data Storage, 320 Data Logger, 33 Working Machine, 3H1 to 3H2 Heatmap, 3TM Working Machine Model, 3TL Protocol Information, 3PM1 Unit Work Prediction Model, 3PM2 Element Work Prediction Model, 3R Estimated Work Content, 3D11, 3D21, 3D31, 3D41 Operating image, 3D12, 3D22, 3D32, 3D42; motion trajectory image, 3D13, 3D23; partial image QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 2023-203133

[0001] JP 2023-203150

[0001] JP 2023-203165

[0001] JP 2020-183615 A ​​

[0005]

Claims

[1] Playback system, comprising: a data acquisition unit configured to capture log information from a work machine that is associated with a point in time; an image generation unit configured to produce an operating image representing the operation of the working machine by sequentially applying operating information representing the operation of the working machine, based on the protocol information, to a three-dimensional model of the working machine; and a display control unit configured to display the operating image on a predetermined display unit, wherein the three-dimensional model includes an operator unit model representing an operator unit provided in a cabin of the working machine, and The image generation unit produces the operating image with the three-dimensional model, which is set to a viewing point facing the control unit in the cabin. [2] Playback system according to claim 1, wherein the operating image includes an image representing the control unit. [3] Playback system according to claim 2, further comprising an estimation unit configured to estimate work content of the working machine at each time point based on the log information, wherein The display control unit also displays a work content image on the display unit, which represents a time series of the estimated work content. [4] Playback system according to claim 3, wherein the display control unit further displays on the display unit an operator amount image representing a time series of a value corresponding to an operator amount of the operator unit included in the protocol information. [5] Playback system according to claim 4, wherein the display control unit further displays on the display unit an image of an instantaneous fuel efficiency which represents a time series of the instantaneous fuel efficiency included in the log information. [6] Reproduction system according to claim 5, wherein the image generation unit rotates a projection plane onto which the three-dimensional model is projected based on a roll angle of the working machine and changes a height of the projection plane based on a pitch angle of the working machine. [7] Reproduction methods, including: Recording log information from a working machine that is assigned to a specific point in time; Generating an operational image representing the operation of the working machine by sequentially applying operational information representing the operation of the working machine, based on the protocol information, to a three-dimensional model of the working machine; and Displaying the operating image on a predetermined display unit, wherein the three-dimensional model includes an operator unit model representing an operator unit provided in a cabin of the working machine, and The generation of the operating image includes the generation of the operating image with the three-dimensional model, which is set to a viewing point facing the control unit in the cabin. [8] Playback system, comprising: a capture unit configured to capture log information from a working machine that is associated with a point in time, and moving image information that represents a captured moving image taken by the working machine and associated with a point in time; and a playback unit configured to sequentially reproduce an operating image representing the operation of the working machine by sequentially applying operating information representing the operation of the working machine, based on the protocol information, to a three-dimensional model of the working machine, and sequentially reproducing a captured image based on the moving image information in synchronization with the operating image. [9] Playback system according to claim 8, wherein the playback unit sequentially displays the operating image in a first display area of ​​a predetermined display unit and sequentially displays the recorded image in a second display area, which differs from the first display area of ​​the display unit, in synchronization with the operating image. [10] Playback system according to claim 9, further comprising a bird's-eye view image display unit configured to produce a bird's-eye view image including a plurality of work machines located in the same construction site area at the same time, and to display the bird's-eye view image, wherein each of the work machines is selectable, in a third display area that is different from the first display area or the second display area of ​​the display unit, wherein the capture unit captures the log information and the motion image information of the work machine selected in the bird's-eye view image. [11] Playback system according to claim 9 or 10, wherein The acquisition unit captures the log information and the moving image information from each of a large number of work machines, and The playback unit sequentially displays the operating images of the respective work machines in a superimposed manner in the first display area and displays the respective recorded images side by side in the second display area based on the respective individual moving image information. [12] Reproduction methods, comprising: Capturing log information from a working machine that is assigned to a point in time, and moving image information that represents a recorded moving image captured by the working machine and assigned to a point in time; and Sequential reproduction of an operating image representing the operation of the working machine by sequentially applying operating information representing the operation of the working machine based on the protocol information to a three-dimensional model of the working machine, and sequential reproduction of a recorded image based on the moving image information in synchronization with the operating image. [13] Display control system comprising: a data acquisition unit configured to capture log information from a work machine that is associated with a point in time; an image generation unit configured to sequentially generate an operating image representing the operation of the working machine by sequentially applying operating information representing the operation of the working machine, based on the protocol information, to a three-dimensional model of the working machine, and to sequentially generate a motion trajectory image representing the motion trajectory of a predetermined part of the working machine in conjunction with the operating image, based on the operating information; and a display control unit configured to display the operating image and the motion trajectory image in a superimposed manner on a predetermined display unit. [14] Display control system according to claim 13, further comprising: an estimation unit configured to estimate the work content of the work machine at any given time based on the log information; and a labeling unit configured to label the work content for which the motion trajectory image is to be generated, wherein the image generation unit generates the motion trajectory image if the estimated work content matches the labeled work content. [15] Display control system according to claim 14, wherein The image generation unit produces a partial image, representing a predetermined part of a working tool of the working machine, at a time that is a predetermined time interval earlier, in conjunction with the operating image, and The display control unit overlays the partial image onto the operating image on the display unit. [16] Display control system according to claim 15, wherein the predetermined part corresponds to a cutting edge of the working tool of the working machine. [17] Display control system according to claim 15, wherein the predetermined part corresponds to a ground contact position of a track or tire of the working machine. [18] Display control system according to claim 16, wherein the image generation unit changes a mode of the motion trajectory image based on at least one of the work content estimated based on the protocol information, an accuracy of position information included in the protocol information, information representing whether an automatic operating control is active in the working machine included in the protocol information, a work quantity of the working machine based on the protocol information and information representing a workload of the working machine based on the protocol information. [19] Display control methods, including: Recording log information from a working machine that is assigned to a specific point in time; Sequential generation of an operating image representing the operation of the working machine by sequentially applying operating information representing the operation of the working machine, based on the protocol information, to a three-dimensional model of the working machine, and sequential generation of a motion trajectory image representing the motion trajectory of a predetermined part of the working machine, in conjunction with the operating image based on the operating information; and Displaying the operating image and the motion trajectory image in a superimposed manner on a predetermined display unit.

Images