Replay system and replay method

Abstract

To provide a replay system and a replay method that can perform detailed analyses of operations done by a work machine.SOLUTION: A replay system comprises: an acquisition unit that acquires chronically mapped log information of a work machine and moving-image information of captured moving images which have been captured by the work machine and chronically mapped; and a replay unit that sequentially replays operation images showing the operations of the work machine by sequentially applying operation information based on the log information showing the operations of the work machine to a three-dimensional model of the work machine, and also sequentially replays the captured moving images based on the moving-image information in synchronization with the operation images.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present disclosure relates to a playback system and a playback method.

Background Art

[0002] Patent Document 1 describes a playback device that enables grasping of the movements of two different construction machines (hereinafter referred to as work machines). The playback device described in Patent Document 1 reproduces the operations of the work machine by sequentially applying the angle information of the work machine included in the log information of the work machine to the 3D (three-dimensional) model of the work machine. In the playback device described in this Patent Document 1, the playback of the 3D model based on the first log information and the playback of the 3D model based on the second log information are performed synchronously.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in the analysis of the work of a work machine using a 3D model, there is a need to check, for example, the loading state of the excavation target in a working tool such as a bucket that has excavated the excavation target such as earth and sand, and the surrounding conditions such as the terrain where the work machine is located.

[0005] The present disclosure has been made in view of the above circumstances, and an object thereof is to provide a playback system and a playback method capable of analyzing in detail the work performed by a work machine.

Means for Solving the Problems

[0006] The playback system of the present disclosure includes an acquisition unit that acquires log information of a working machine associated with a time and moving image information representing a captured video that is captured by the working machine and associated with a time, and an operation of the working machine based on the log information. An operation image representing the operation of the working machine is sequentially reproduced by sequentially applying the operation information to the three-dimensional model of the working machine, and a playback unit that sequentially reproduces the captured image based on the moving image information in synchronization with the operation image.

[0007] The playback method of the present disclosure includes a step of acquiring log information of a working machine associated with a time and moving image information representing a captured video that is captured by the working machine and associated with a time, and an operation of the working machine based on the log information. An operation image representing the operation of the working machine is sequentially reproduced by sequentially applying the operation information to the three-dimensional model of the working machine, and a step of sequentially reproducing the captured image based on the moving image information in synchronization with the operation image.

Effect of the Invention

[0008] According to the display control device and the display control method of the present disclosure, the work by the working machine can be analyzed in more detail.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

[0010] <First Embodiment> Hereinafter, a playback system and a playback method according to the first embodiment will be described in detail with reference to FIGS. 1 to 12.

[0011] (Overall Configuration of Analysis Support System) FIG. 1 is a diagram showing the overall configuration of an analysis support system according to the first embodiment. The analysis support system 1 includes a playback system 10, data loggers 20 and imaging devices CAM mounted on each of a plurality of work machines 3.

[0012] The working machine 3 is the object of work analysis by the reproduction system 10. Examples of the working machine 3 include a hydraulic excavator, a wheel loader, a bulldozer, and the like. In the following description, a hydraulic excavator will be cited as an example of the working machine 3 for explanation. A plurality of sensors are provided in each working machine 3. The data logger 20 records and accumulates in time series the information indicating the state of the working machine 3 acquired by the sensors. Hereinafter, the information indicating the state of the working machine 3 at each time recorded by the data logger 20 will also be referred to as log information. This log information is associated with the time. When the operating mechanism for operating the working machine 3 is configured to operate the working machine 3 using an electrical operation signal, the information of the operation signal of the working machine 3 may be recorded and accumulated in time series and included in the log information. Further, the data logger 20 transmits the recorded log information to the reproduction system 10 via a wide area communication network at regular time intervals. Note that the regular time interval is, for example, a 5-minute interval. The reproduction system 10 records the log information received from the data logger 20 on a recording medium. The imaging device CAM images, for example, an external moving image in the forward direction of the working machine 3. The imaging device CAM may store the moving image information representing the captured moving image in a storage device inside or outside the imaging device CAM. The moving image information is associated with time information (time). Further, the imaging device CAM transmits the captured moving image information to the reproduction system 10 at regular time intervals via a wide area communication network, or distributes it to the reproduction system 10 almost in real time. The reproduction system 10 records the moving image information received from the imaging device CAM on a recording medium or reproduces it almost in real time. Hereinafter, the case where the reproduction system 10 temporarily stores the moving image information will be described as an example. Note that the functions of the reproduction system 10 will be described later.

[0013] (Structure of Working Machine) FIG. 2 is a diagram showing the structure of a working machine according to the first embodiment. The working machine 3, which is a hydraulic excavator, excavates and levels earth and sand at a work site or the like. As shown in FIG. 2, a working machine 3 which is a hydraulic excavator has a lower traveling body 31 for traveling and an upper revolving body 32 installed on the upper part of the lower traveling body 31 and capable of revolving. Further, a driver's cab 32A, a working device 32B, and two GNSS (Global Navigation Satellite System) antennas G1 and G2 are provided on the upper revolving body 32.

[0014] The lower traveling body 31 has a left crawler CL and a right crawler CR. The working machine 3 moves forward, turns, and moves backward by the rotation of the left crawler CL and the right crawler CR.

[0015] The driver's cab 32A is a place where the operator of the working machine 3 boards and performs operations. The driver's cab 32A is installed, for example, on the left side portion of the front end of the upper revolving body 32. Further, an imaging device CAM for imaging a moving image in the forward direction of the working machine 3 is attached inside the driver's cab 32A. However, the attachment position and the number of the imaging device CAM to the working machine 3 are not limited to this example. The configuration of the operation unit inside the driver's cab 32A will be described later.

[0016] The working device 32B is composed of a boom BM, an arm AR, and a bucket BK. The boom BM is attached to the front end of the upper revolving body 32. Further, the arm AR is attached to the boom BM. Further, the bucket BK is attached to the arm AR. Further, a boom cylinder SL1 is attached between the upper revolving body 32 and the boom BM. By driving the boom cylinder SL1, the boom BM can be operated with respect to the upper revolving body 32. An arm cylinder SL2 is attached between the boom BM and the arm AR. By driving the arm cylinder SL2, the arm AR can be operated with respect to the boom BM. A bucket cylinder SL3 is attached between the arm AR and the bucket BK. By driving the bucket cylinder SL3, the bucket BK can be operated with respect to the arm AR. The upper swing body 32, boom BM, arm AR, and bucket BK included in the work machine 3, which is a hydraulic excavator, are an aspect of the movable parts of the work machine 3. Also, the bucket BK is an example of a configuration of the "working tool" according to the present disclosure. The bucket BK is provided with a cutting edge BKT used for excavation and the like.

[0017] (Configuration of the driver's cab) FIG. 3 is a diagram showing the configuration of the driver's cab 32A of the work machine according to the first embodiment.

[0018] As shown in FIG. 3, the driver's cab 32A is provided with operation levers L1, L2, foot pedals F1, F2, and travel levers R1, R2. The operation levers L1, L2, foot pedals F1, F2, and travel levers R1, R2 are included in an operation unit 32AO for operating the movable parts of the work machine 3. The operation lever L1 and the operation lever L2 are arranged on the left and right sides of the seat ST in the driver's cab 32A. Also, the foot pedals F1 and F2 are arranged on the floor surface in front of the seat ST in the driver's cab 32A.

[0019] An example of the operation pattern showing the correspondence between the input operations for the operation levers L1, L2, travel levers R1, R2 and the operations of the work machine 3, which is a hydraulic excavator, is as follows.

[0020] The operation lever L1 arranged on the left side toward the front of the driver's cab is an operation mechanism for performing the swing operation of the upper swing body 32 and the excavation / dumping operation of the arm AR. Specifically, when the operator of the work machine 3 tilts the operation lever L1 forward, the arm AR performs a dumping operation. Also, when the operator of the work machine 3 tilts the operation lever L1 backward, the arm AR performs an excavation operation. Also, when the operator of the work machine 3 tilts the operation lever L1 to the right, the upper swing body 32 swings to the right. Also, when the operator of the work machine 3 tilts the operation lever L1 to the left, the upper swing body 32 swings to the left. When the operation lever L1 is tilted in the front-rear direction, the upper swing body 32 may swing to the right or left, and when the operation lever L1 is tilted in the left-right direction, the arm AR may perform a dumping operation or an excavation operation.

[0021] The operation lever L2 arranged on the right side toward the front of the driver's cab is an operating mechanism for performing the excavation / dumping operation of the bucket BK and the raising / lowering operation of the boom BM. Specifically, when the operator of the working machine 3 tilts the operation lever L2 forward, the lowering operation of the boom BM is executed. Also, when the operator of the working machine 3 tilts the operation lever L2 backward, the raising operation of the boom BM is executed. Further, when the operator of the working machine 3 tilts the operation lever L2 to the right, the dumping operation of the bucket BK is performed. Also, when the operator of the working machine 3 tilts the operation lever L2 to the left, the excavation operation of the bucket BK is performed.

[0022] Also, the travel levers R1, R2 are operating mechanisms for controlling the operation of the lower travel body 31, that is, for controlling the travel of the working machine 3. The travel lever R1 arranged on the left side toward the front of the driver's cab corresponds to the rotational drive of the left crawler CL of the lower travel body 31. Specifically, when the operator of the working machine 3 tilts the travel lever R1 forward, the left crawler CL rotates in the forward direction. Also, when the operator of the working machine 3 tilts the travel lever R1 backward, the left crawler CL rotates in the reverse direction.

[0023] The travel lever R2 arranged on the right side toward the front of the driver's cab corresponds to the rotational drive of the right crawler CR of the lower travel body 31. Specifically, when the operator of the working machine 3 tilts the travel lever R2 forward, the right crawler CR rotates in the forward direction. Also, when the operator of the working machine 3 tilts the travel lever R2 backward, the right crawler CR rotates in the reverse direction. Note that the foot pedals F1, F2 are interlocked with the travel levers R1, R2 respectively, and the travel can also be controlled by the foot pedals F1, F2.

[0024] Note that the above-described operation patterns are only examples and are not limited to the above-described modes depending on the model of the hydraulic excavator and the like.

[0025] Note that depending on the embodiment, the working machine 3 described with reference to FIG. 2 may not be provided with the GNSS antennas G1, G2.

[0026] (Functional Configuration of the Reproduction System) FIG. 4 is a diagram showing the functional configuration of the reproduction system according to the first embodiment. Hereinafter, with reference to FIG. 4, the functions of the reproduction system 10 according to the first embodiment will be described. As shown in FIG. 4, the reproduction system 10 includes a CPU 100, a memory 101, a display unit 102, an operation reception unit 103, a communication interface 104, and a storage 105. Note that the CPU (Central Processing Unit) 100 may be a processor such as an FPGA or a GPU instead of a CPU.

[0027] The CPU 100 is a processor that controls the overall operation of the reproduction system 10. Various functions of the CPU 100 will be described later.

[0028] The memory 101 is a so-called main memory device. Instructions and data necessary for the CPU 100 to operate based on a program are expanded in the memory 101.

[0029] The display unit 102 is a display device capable of visibly displaying information, and is, for example, a liquid crystal display or an organic EL display.

[0030] The operation reception unit 103 is an input device, and is, for example, a general mouse, keyboard, touch sensor, etc.

[0031] The communication interface 104 is a communication interface for communicating with the data logger 20.

[0032] The storage 105 is a so-called auxiliary storage device, and is, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. Log information TL received from the data logger 20, the vehicle type of the working machine 3, the working machine model TM which is a 3D model prepared in advance for each model, moving image information MV captured by the working machine 3, etc. are recorded in the storage 105. Note that the working machine model TM will be described later. In addition, the storage 105 also records a unit operation prediction model PM1, which is a learned machine learning model used when estimating the operation content of the working machine 3, an element operation prediction model PM2, heatmaps (H1, H2) generated during the estimation process, the estimated operation content R of the working machine 3, 3D point cloud information MAP representing the terrain information of the work site, and the like. Note that the unit operation prediction model PM1, the element operation prediction model PM2, and the heatmaps (H1, H2) will be described later. The 3D point cloud information MAP includes, for example, 3D point cloud information measured at the work site using a drone or the like, and information obtained by sequentially updating the terrain information from the cutting edge coordinate information of the working machine 3 during operation, and is assumed to be recorded in the storage 105.

[0033] The functions of the CPU 100 of the reproduction system 10 according to the first embodiment will be described in detail. By operating based on a predetermined program, the CPU 100 exhibits functions as an acquisition unit 1000, a reception unit 1001, an extraction unit 1002, a reproduction unit 1003, an estimation unit 1004, a determination unit 1005, and a synchronization unit 1006. Note that the predetermined program may be for realizing a part of the functions to be exhibited by the reproduction system 10. For example, the program may exhibit functions in combination with other programs already stored in the storage 105, or in combination with other programs implemented in other devices. In other embodiments, the reproduction system 10 may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, part or all of the functions realized by the processor may be realized by the integrated circuit.

[0034] The acquisition unit 1000 acquires log information TL to be played back from among a plurality of log information TLs recorded and stored in the storage 105. Further, the acquisition unit 1000 acquires moving image information MV corresponding to the log information TL to be played back. Here, it is assumed that the plurality of log information TLs and the plurality of moving image information MV are recorded separately for each file recorded with different file names in the storage 105. The acquisition unit 1000 acquires, for example, one log information TL and moving image information MV to be analyzed or two log information TLs and two moving image information MV to be compared in accordance with an instruction from an operator. Hereinafter, the case where the acquisition unit 1000 acquires two log information TLs and two moving image information MV to be compared will be described as an example. Note that since the moving image information MV is acquired in a form corresponding to the acquired log information TL, the description of the moving image information MV acquired when acquiring the log information TL will be omitted as appropriate. One of the two log information acquired by the acquisition unit 1000 is log information designated by an operator as a playback target by a file name or the like. Hereinafter, this log information will be referred to as designated log information TL1 (first log information). The other of the two log information is log information played back simultaneously with the playback based on the designated log information TL1. Hereinafter, this log information will be referred to as comparison log information TL2 (second log information). The comparison log information TL2 may be log information pre-selected as a "model" for each type of work content and conditions from viewpoints such as good fuel efficiency, short working hours, and driving by a skilled operator. Alternatively, the comparison log information TL2 may be, for example, past log information recorded in a driving operation performed by the same operator as the designated log information TL1 previously. In this case, by comparing the two log information, for example, the proficiency of the operator can be analyzed.

[0035] The acquisition unit 1000 according to this embodiment automatically searches for and acquires appropriate comparison log information TL2 as a comparison target for the designated log information TL1 based on the determination result by the determination unit 1005 described later. However, the acquisition unit 1000 according to other embodiments is not limited to the above mode. For example, the acquisition unit 1000 may acquire the log information designated by the operator with a file name or the like as the comparison log information TL2. Further, when there is only one piece of recorded comparison log information TL2, the acquisition unit 1000 may acquire the one piece of comparison log information TL2.

[0036] The reception unit 1001 receives a predetermined reproduction instruction from the operator of the reproduction system 10. For example, the reception unit 1001 receives a reproduction instruction for the working machine 3 from the operator of the reproduction system 10.

[0037] The extraction unit 1002 extracts the angle information used for the reproduction of the working machine 3 from the acquired log information TL. The angle information is an example of the "motion information" according to the present disclosure.

[0038] The reproduction unit 1003 sequentially applies the motion information representing the operation of the working machine 3 based on the log information TL to the three-dimensional model of the working machine 3, and sequentially reproduces the motion image representing the operation of the working machine 3 within the first display area of the display screen of the display unit 102. Further, the reproduction unit 1003 sequentially reproduces the captured image based on the moving image information MV corresponding to the log information TL in synchronization with the motion image within the second display area different from the first display area of the display screen of the display unit 102. Ensuring the synchronization between the reproduction of the motion image (reproduction of the working machine model TM) and the reproduction of the captured video of the same working machine 3 can be performed based on the time information included in the log information TL and the time information included in the moving image information MV. The reproduction unit 1003 adjusts the reproduction timing of the motion image and the captured video so that the recording time corresponding to the motion information included in the log information TL and the imaging time of each frame included in the moving image information MV substantially coincide.

[0039] Further, the playback unit 1003 may display the estimation result of the work content by the estimation unit 1004 on the display unit 102 as supplementary information, or may synthesize an image representing the terrain (terrain image) based on the three-dimensional point cloud information MAP with the operation image and display it on the display unit 102.

[0040] The estimation unit 1004 estimates the work content of the working machine 3 at each time from the acquired log information TL.

[0041] The determination unit 1005 determines whether the comparison log information TL2 is appropriate as a comparison target for the designated log information TL1 based on the information included in the designated log information TL1.

[0042] The synchronization unit 1006 performs a process of synchronizing the playback of the working machine model TM (playback of the operation image) based on the designated log information TL1 and the playback of the working machine model TM (playback of the operation image) based on the comparison log information TL2. Specifically, the synchronization unit 1006 specifies the reproduction start time in the time series of the animation of the working machine model TM based on the work content estimated by the estimation unit 1004.

[0043] (Processing flow of the playback system) Hereinafter, with reference to FIGS. 5 to 13, the specific processing flow performed by the playback system 10 will be described in detail.

[0044] The processing flow shown in FIG. 5 starts from the point when a dedicated application is launched by the operator of the playback system 10. When a dedicated application is launched by the operator's operation, the acquisition unit 1000 of the CPU 100 expands and acquires the designated log information TL1 designated as the reproduction target in the memory 101 (step S00).

[0045] Here, the log information TL (designated log information TL1, comparison log information TL2) will be described with reference to FIGS. 6 to 8.

[0046] As shown in FIGS. 6 to 8, the log information TL includes construction machine identification information. Specifically, the construction machine identification information is an individual identification number for specifically identifying each construction machine 3. In FIGS. 6 to 8, it is assumed that the construction machine identification information is allocated so as to correspond to the vehicle type, model, type, machine number, etc. of the construction machine 3 indicating a hydraulic excavator, a wheel loader, a bulldozer, etc. Note that the construction machine identification information may be a number, a combination of numbers, letters, symbols, or a combination thereof, etc. other than a number.

[0047] As shown in FIG. 6, the log information TL includes information indicating the position and posture of the construction machine 3 at each time, and angle information of the movable parts of the construction machine 3. Specifically, the log information TL records the position of the construction machine 3, the roll angle of the construction machine 3 which is the inclination in the left - right direction of the machine body, the pitch angle which is the inclination in the front - rear direction of the machine body, the turning angle, the boom angle, the arm angle, and the bucket angle for each time. Here, the data logger 20 mounted on the construction machine 3 obtains information received from the GNSS antennas G1 and G2, for example, and based on the positioning information indicating latitude, longitude, and altitude, identifies and records the position of the construction machine 3. Further, the data logger 20 calculates and records the roll angle and pitch angle of the construction machine 3 based on the measurement results of an IMU (Inertial Measurement Unit) mounted on the construction machine 3. Also, the data logger 20 calculates and records the turning angle of the upper swing body 32 based on the positioning information obtained from each of the GNSS antennas G1 and G2 provided on the upper swing body 32. Furthermore, the data logger 20 calculates and records the boom angle, arm angle, and bucket angle based on the extension / contraction degrees of the boom cylinder SL1, arm cylinder SL2, and bucket cylinder SL3, respectively. Note that the boom angle, arm angle, bucket angle, and turning angle may be obtained, for example, by attaching an IMU to the boom, arm, bucket, and upper swing body and using those IMUs.

[0048] Note that the position, roll angle, and pitch angle are information necessary to specify the position and orientation of the working machine 3 itself. Therefore, for example, in an embodiment where only the movements of the movable parts of the working machine 3, that is, the upper swing body 32, boom BM, arm AR, and bucket BK, are animated and the position and orientation of the working machine 3 itself are not reproduced, the information on the position, roll angle, and pitch angle does not need to be included in the log information.

[0049] The log information TL shown in FIG. 6 corresponds to the "operation information" according to the present disclosure. The operation information is information representing the operations of the movable parts of the working machine 3, such as the working machine 32B, upper swing body 32, and lower traveling body 31, which are operated by the operation unit of the working machine 3. Alternatively, the log information TL shown in FIG. 6 is information representing the operations of the working machine 3 that can be visually recognized from the outside of the working machine 3, and can also be referred to as appearance operation information. That is, in the present embodiment, the operation information or appearance operation information includes the position information, roll angle, pitch angle, swing angle, boom angle, arm angle, and bucket angle of the working machine 3. In the present embodiment, different from the drive mechanism information and operation information described later, the operation information (or appearance operation information) has the feature of being less affected by the difference in the vehicle class of the working machine 3 as information used when estimating the work content.

[0050] Also, as shown in FIG. 7, the log information TL includes the degree of input to the operator's operation levers L1, L2, etc. at each time, that is, the pilot hydraulic pressure (PPC (Proportional Pressure Control) pressure) indicating the degree of tilt of the lever and the degree of depression of the pedal. Specifically, the log information TL records the PPC pressures of the operation levers L1, L2, traveling levers R1, R2, or foot pedals F1, F2 corresponding to each operation type of left / right swing, arm excavation / dumping, boom raising / lowering, bucket excavation / dumping, right crawler forward / backward, and left crawler forward / backward by the operator at each time. Each time shown in FIG. 7 corresponds to each time in FIG. 6. In the present embodiment, each piece of information shown in FIG. 7 is also referred to as operation information.

[0051] Also, as shown in FIG. 8, the log information TL includes information indicating the status of the main drive mechanisms such as the engine and hydraulic pump of the work machine 3 at each time. Specifically, the log information TL records the engine coolant temperature, engine output, instantaneous fuel consumption, and oil temperature of the hydraulic pump at each time. Note that the items shown in FIG. 8 are just examples, and for example, items such as the engine coolant temperature and the oil temperature of the hydraulic pump may be omitted, or for example, the fuel throttle opening may be included as other items affecting fuel consumption. Further, as another example, in an electrified construction machine, items such as instantaneous power consumption and battery remaining amount may be included. Also, each time shown in FIG. 8 corresponds to each time in FIGS. 6 and 7. In the present embodiment, each piece of information shown in FIG. 8 is also referred to as drive mechanism information.

[0052] Returning to FIG. 5, the estimation unit 1004 of the CPU 100 estimates the work content of the work machine 3 at each time based on the specified log information TL1 acquired in step S00 (step S01).

[0053] Here, the procedure for the estimation unit 1004 to estimate the work content of the work machine 3 from the log information TL will be described with reference to FIG. 9. The estimation unit 1004 estimates the work content of the work machine 3 with respect to both unit work and elemental work. Unit work is work that accomplishes one work purpose. Elemental work is a series of operations or work that constitutes unit work and is classified by purpose.

[0054] Examples of the classification of unit work include, for example, "excavation and loading", "hoe work", "slope (from below)", "loading collection", "traveling", "stopping and parking" shown in FIG. 9, and also "ditch excavation", "backfilling", "slope (from above)", etc. Excavation and loading is work that digs, cuts, and loads the excavated earth and sand or rock onto the loading platform of a transport vehicle. Excavation and loading is a unit work composed of excavation, loading swing, soil discharge, empty swing, waiting for soil discharge, and loading platform pressing. Scraping is an operation of scraping and leveling the ground to make the extra undulations of the ground reach a predetermined height. Scraping consists of excavation and soil discharge, or excavation, loading swing, soil discharge, and empty swing, and is a unit operation that may include spreading and brooming. Slope formation (from below) is an operation of forming a slope by a working machine 3 located below the target location. Slope formation (from below) consists of compaction, excavation, loading swing, soil discharge, and empty swing, and is a unit operation that may include spreading. Loading collection is an operation of collecting the earth and sand excavated by excavation or the like before loading it onto a transport vehicle. Loading collection consists of excavation, loading swing, soil discharge, and empty swing, and is a unit operation that may include spreading. Traveling is an operation of moving the working machine 3. Traveling as a unit operation is a unit operation composed of traveling as an elemental operation. Stopping and idling is a state in which there is no earth, sand, and rock in the bucket BK and the machine has stopped for a predetermined time or more. Stopping and idling as a unit operation is a unit operation composed of stopping as an elemental operation. Trench excavation is an operation of digging and scraping the ground into a long and narrow groove. Trench excavation consists of excavation, loading swing, soil discharge, and empty swing, and is a unit operation that may include spreading. Backfilling is an operation of putting earth and sand into an existing groove or hole in the ground and filling it flat. Backfilling consists of excavation, loading swing, soil discharge, compaction, and empty swing, and is a unit operation that may include spreading and brooming. Slope formation (from above) is an operation of forming a slope by a working machine 3 located above the target location. Slope formation (from above) consists of compaction, excavation, loading swing, soil discharge, and empty swing, and is a unit operation that may include spreading.

[0055] Examples of the classification of elemental operations include "excavation", "loading swing", "waiting for soil discharge", "soil discharge", "empty swing", "holding down the loading platform" shown in FIG. 9, and also "compaction", "spreading", "brooming", etc. Excavation is an operation of digging and scraping earth and sand or rock with the bucket BK. The load swing is an operation of swinging the upper swing body 32 while holding the excavated earth and sand or rock in the bucket BK. The waiting for dumping is an operation of waiting for a transport vehicle for loading while holding the excavated earth and sand or rock in the bucket BK. The dumping is an operation of unloading the excavated earth and sand or rock from the bucket BK to a transport vehicle or a predetermined location. The empty load swing is an operation of swinging the upper swing body 32 with no earth and sand or rock in the bucket BK. The load bed pressing is an operation of pressing the earth and sand loaded on the load bed of the transport vehicle from above with the bucket BK to make it flat. The rolling compaction is an operation of pushing the earth and sand into the disturbed ground with the bucket BK to form and strengthen the ground. The leveling is an operation of leveling the earth and sand with the bottom surface of the bucket BK. The sweeping is an operation of leveling the earth and sand with the side surface of the bucket BK.

[0056] The estimation unit 1004 obtains a time series of likelihoods related to unit operations by inputting the log information TL into the unit operation prediction model PM1 in chronological order. The unit operation prediction model PM1 is a model that outputs the likelihood related to unit operations when the log information TL is input, for example, by learning using teacher data, and may be stored in the storage 105, for example.

[0057] Also, the estimation unit 1004 obtains a time series of likelihoods related to elemental operations by inputting the log information TL into the elemental operation prediction model PM2 in chronological order. The elemental operation prediction model PM2 is a model that outputs the likelihood related to elemental operations when the log information TL is input, for example, by learning using teacher data, and may be stored in the storage 105, for example.

[0058] The estimation unit 1004 smooths the time series of likelihoods related to unit operations and the time series of likelihoods related to elemental operations by applying them to a temporal average filter respectively, and generates a unit operation heat map H1 representing the smoothed time series of likelihoods related to unit operations and an elemental operation heat map H2 representing the smoothed time series of likelihoods related to elemental operations as shown in FIG. 9. The heat maps H1 and H2 are maps in which, based on the smoothed time series of likelihoods, colors representing the likelihoods of operation categories are assigned to a plane with the operation categories on the vertical axis and time on the horizontal axis. The colors related to the heat map may, for example, approach blue as the likelihood of the operation category is lower and approach red as the likelihood of the operation category is higher. The estimation unit 1004 stores the heat maps H1 and H2 in the storage 105.

[0059] Based on the smoothed time series of likelihoods, the estimation unit 1004 identifies the time period in which the likelihood of the unit operation is dominant and estimates the operation content of the work machine 3 during that time period. For example, for the time period in which the likelihood of the unit operation "excavation and loading" is dominant, the operation content of the work machine 3 is estimated to be "excavation and loading". Similarly, based on the smoothed time series of likelihoods, the estimation unit 1004 identifies the time period in which the likelihood of the elemental operation is dominant and estimates the operation content of the work machine 3 during that time period. For example, for the time period in which the likelihood of the elemental operation "excavation" is dominant, the operation content of the work machine 3 is estimated to be "excavation". The estimation unit 1004 stores the information on the operation content R estimated for the work machine 3 in the storage 105.

[0060] Returning to FIG. 5, subsequently, acquisition unit 1000 determines whether to display information to be compared (comparison log information TL2) for the designated log information TL1 designated as the reproduction target based on a predetermined instruction operation from the operator (step S02A). If it is to be displayed (step S02A: YES), after executing the processes of step S02B and step S03, it proceeds to step S04. If it is not to be displayed (step S02A: NO), it proceeds to step S04 without executing step S02B and step S03. If it is to be displayed (step S02A: YES), acquisition unit 1000 acquires appropriate comparison log information TL2 as the comparison target for the designated log information TL1 acquired in step S00 (step S02B). At this time, determination unit 1005 performs a determination processing flow for determining whether the plurality of comparison log information TL2 recorded in storage 105 is appropriate as the comparison target for the designated log information TL1 acquired in step S00. In this determination processing flow, first, determination unit 1005 selects one from the plurality of comparison log information TL2 previously recorded in storage 105. Then, based on the log information TL, determination unit 1005 determines, for example, whether the vehicle compartment, the amount of change in the turning angle (the magnitude of the swing), the bucket height when the working machine 3 is waiting for dumping, etc. are of the same degree. If it is determined that they are not of the same degree, a new comparison log information TL2 is selected, and the same determination processing as above is performed. In addition, the determination unit 1005 according to another embodiment may narrow down the reproduction candidates based on the working position, working time, work classification, etc. in addition to the above determination processing.

[0061] Returning to FIG. 5, next, synchronization unit 1006 performs a process of synchronizing the animation reproduction of the working machine model TM based on the designated log information TL1 and the animation reproduction of the working machine model TM based on the comparison log information TL2 (step S03). Specifically, synchronization unit 1006 identifies the reproduction start time in the time series in the animation of the working machine model TM according to the designated log information TL1 and the reproduction start time in the time series in the animation of the working machine model TM according to the comparison log information TL2.

[0062] The method for specifying the playback start time for synchronizing the two is as follows. That is, the synchronization unit 1006 extracts the timing of the switching of the unit operation in the designated log information TL1 based on the unit operation heat map H1 for the designated log information TL1 generated by the estimation unit 1004. Next, the synchronization unit 1006 extracts the timing of the switching of the unit operation in the comparison log information TL2 based on the unit operation heat map H1 for the comparison log information TL2 generated by the estimation unit 1004. The synchronization unit 1006 selects one of the extracted timings and specifies it as the playback start time of each animation. Note that the playback system 10 according to other embodiments is not limited to the above aspect. The playback system 10 (synchronization unit 1006) according to other embodiments may, for example, compare the combination of angle information such as boom, arm, and bucket in the comparison log information TL2 with the combination of angle information such as boom, arm, and bucket in the designated log information TL1, extract the timing when both are at similar angles, and specify this as the playback start time of each animation.

[0063] After step S03, or when the answer is "NO" in step S02A, the reception unit 1001 receives a playback instruction from the operator (step S04). One aspect of the playback instruction may be an operation such as pressing a playback button. Also, the playback instruction may include information such as time, the position of the work machine 3, and various events such as an abnormality occurring in the work machine 3 as the starting point of playback. Further, when the reception unit 1001 receives the playback instruction, the playback range specifying unit 1007 may specify the work content to be the generation target of the movement trajectory image.

[0064] Next, the acquisition unit 1000 selects and reads out the work machine model TM corresponding to the referenced work machine identification information from the storage 105 based on the work machine identification information as the type of the work machine 3 received by the reception unit 1001 (step S05).

[0065] Here, the machine tool model TM will be described with reference to FIG. 10. As shown in FIG. 10, the machine tool model TM is information including machine tool identification information and the 3D outer shape model M0 of the machine tool 3 indicated by the machine tool identification information. The 3D outer shape model M0 is a 3D model representing the machine tool 3, and is constructed for each part of the machine tool 3 such as the lower traveling body and the upper slewing body. For example, the 3D outer shape model M0 represents the shape of the machine tool 3. For example, the 3D outer shape model M0 includes a lower traveling body outer shape model M01 representing the lower traveling body 31 of the machine tool 3, an upper slewing body outer shape model M02 representing the upper slewing body 32, a boom outer shape model M03 representing the boom BM, an arm outer shape model M04 representing the arm AR, and a bucket outer shape model M05 representing the bucket BK.

[0066] Returning to FIG. 5, the extraction unit 1002 extracts information for playback from each of the specified log information TL1 and the comparison log information TL2, or from the specified log information TL1 (step S06). For example, the extraction unit 1002 extracts various angle information such as boom angle, arm angle, bucket angle, etc. as information for playback. Note that, as information for playback, the pilot hydraulic pressure shown in FIG. 7 may be extracted. Also, in step S00 and step S02B, only the information for playback may be acquired. Hereinafter, the processing flow will be described by taking as an example the case of comparing and playing back the specified log information TL1 and the comparison log information TL2. When playback using the comparison log information TL2 is not performed, it can be performed by basically omitting the processing related to the comparison log information TL2.

[0067] Next, the playback unit 1003 performs simultaneous playback of the animation of the working machine model TM based on the designated log information TL1, the animation playback of the working machine model TM based on the comparison log information TL2, video playback based on the video information MV corresponding to the designated log information TL1, and video playback based on the video information MV corresponding to the comparison log information TL2 (step S07). Here, the playback unit 1003 applies the various types of information recorded in the designated log information TL1 to the working machine model TM in descending order of the time stamp from the playback start time specified in the synchronization process of step S03, and plays back the motion images representing the operation of the working machine 3 (step S07a). Further, simultaneously with the playback of the motion images for this animation, the playback unit 1003 applies the various types of information recorded in the comparison log information TL2 to the working machine model TM in descending order of the time stamp from the playback start time specified in the synchronization process of step S03, and plays back the motion images representing the operation of the working machine 3 (step S07a).

[0068] Hereinafter, the content of the motion image generation process by the playback unit 1003 will be described in detail. The playback unit 1003 changes the angles of the corresponding parts of the outer shape 3D model M0 based on various angle information such as the turning angle and boom angle shown in the log information TL (designated log information TL1, comparison log information TL2). For example, the playback unit 1003 tilts the bucket outer shape model M05 around the rotation axis defined at the connection position with the arm outer shape model M04 to the bucket angle shown in the log information TL, thereby reproducing the position and posture of the bucket BK of the working machine 3.

[0069] Similarly, the playback unit 1003 tilts the arm outer shape model M04 around the rotation axis defined at the connection position with the boom outer shape model M03 to the arm angle shown in the log information TL, thereby reproducing the position and posture of the arm AR of the working machine 3.

[0070] Similarly, the playback unit 1003 reproduces the position and orientation of the upper swing body 32 of the working machine 3 by tilting the outer shape model M02 of the upper swing body around the rotation axis defined at the connection position with the outer shape model M01 of the lower traveling body to the swing angle shown in the log information TL.

[0071] Similarly, the playback unit 1003 reproduces the orientation of the upper swing body 32 of the working machine 3 by tilting the outer shape model M01 of the lower traveling body around the roll rotation axis defined in the outer shape model M01 to the roll angle shown in the log information TL, and tilting it around the pitch rotation axis defined in the outer shape model M01 of the lower traveling body to the pitch angle shown in the log information TL.

[0072] In addition, the playback system 10 according to the first embodiment can reproduce the traveling animation of the working machine 3 based on the PPC pressures of the right crawler forward / backward and the left crawler forward / backward at each time included in the log information TL.

[0073] Specifically, based on the PPC pressures of the right crawler forward / backward and the left crawler forward / backward, the outer shape 3D model M0 moves forward, backward, left and right forward, and left and right backward. For example, based on the numerical values of the PPC pressures of the right crawler forward and the left crawler forward, the outer shape 3D model M0 is moved in the forward direction. The moving speed may be changed based on the numerical values of the PPC pressures.

[0074] Also, based on the numerical values of the PPC pressures of the right crawler backward and the left crawler backward, the outer shape 3D model M0 is moved in the backward direction. Also, based on the difference in the numerical values of the PPC pressures of the right crawler forward and the left crawler forward, the outer shape 3D model M0 is moved so as to curve in the front left and right direction. For example, when the numerical value of the PPC pressure of the right crawler forward is larger than the numerical value of the PPC pressure of the left crawler forward, it is moved so as to curve in the front left direction. The moving speed and the curvature can be changed according to the respective numerical values of the PPC pressures and the difference in the numerical values of the PPC pressures.

[0075] Similarly, based on the difference in the numerical values of the PPC pressures of the right crawler reverse and the left crawler reverse, the outer shape 3D model M0 is moved so as to curve in the left - right backward direction. For example, when the numerical value of the PPC pressure of the right crawler reverse is larger than the numerical value of the PPC pressure of the left crawler reverse, it is moved so as to curve in the left - backward direction. The speed of movement and the size of the curve may be changed according to the numerical values of the respective PPC pressures and the difference in the numerical values of the PPC pressures.

[0076] In addition to the PPC pressures of the right crawler forward / backward and the left crawler forward / backward, by using the position information for playback, the running of the working machine 3 can be animated more accurately. In this case, by using the position information, the speed and position of the movement of the working machine 3 can be expressed more accurately. Also, in addition to the PPC pressures of the right crawler forward / backward and the left crawler forward / backward, by animating the working machine 3 based on the roll angle, or the pitch angle, or both the roll angle and the pitch angle, the left - right inclination of the working machine 3 during running or the front - rear inclination of the working machine 3 can be reproduced.

[0077] Also, in step S07, the playback unit 1003, in synchronization with the playback of the operation image in step S07a, acquires the moving image information MV corresponding to the log information TL (designated log information TL1, comparison log information TL2) by the acquisition unit 1000, and sequentially plays back the captured images based on the moving image information MV acquired by the acquisition unit 1000 (step S07b).

[0078] Also, in step S07, the playback unit 1003 synthesizes a terrain image representing the terrain based on the three - dimensional point cloud information MAP with the operation image and displays it on the display unit 102 (step S07c).

[0079] Also, in step S07, the playback unit 1003 displays supplementary information such as the estimated result of the work content at each time on the display unit 102 (step S07d).

[0080] During the animation playback of the operations of the two machine tools 3, the playback unit 1003 determines whether to end the animation playback (step S08). For example, when receiving an instruction to end playback based on pressing a stop button or the like, it is determined to end the animation playback. After the start of the animation playback, it may be determined to end the animation playback after a predetermined period has elapsed. If the animation playback does not end (step S08; NO), the playback unit 1003 continues the simultaneous animation playback of the two machine tool models TM. On the other hand, if ending the animation playback (step S08; YES), the playback unit 1003 ends the animation playback process.

[0081] Among the processing flows described with reference to FIG. 5, steps S00, S01, S03, S04, S05, S06, and S08 are not essential components of the playback system 10, and in other embodiments, such steps may not be provided.

[0082] (Display image of the playback system) FIGS. 11 and 12 are diagrams showing examples of display images based on the designated log information TL1 and the comparison log information TL2 of the playback system according to the first embodiment. FIG. 13 is a diagram showing an example of a display image based on the designated log information TL1 of the playback system according to the first embodiment. When comparing the designated log information TL1 and the comparison log information TL2, the CPU 100 of the playback system 10 according to the first embodiment causes the display unit 102 to display display images D1 and D2 as shown in FIGS. 11 and 12, for example.

[0083] The display image D1 shown in FIG. 11 includes a first display area DR1 and a second display area DR2. In the first display area DR1, an operation image D11 is displayed. The operation image D11 is an image that includes, for example, an operation image D111 (displayed in solid lines) based on the specified log information TL1 and an operation image D112 (displayed in dashed lines) based on the comparison log information TL2, which are superimposed in different display colors. Further, the operation image D11 includes a terrain image D113 generated so as to be synthesized with the operation image D111 and the operation image D112. In the second display area DR2, an imaging image D12 is displayed. The imaging image D12 includes an imaging image D121 based on the moving image information MV corresponding to the specified log information TL1 and an imaging image D122 based on the moving image information MV corresponding to the comparison log information TL2. In this example, the imaging image D12 includes an image representing the load bed of a dump truck, a working machine, and the load of a bucket, which are objects to be loaded such as excavated materials. In this example, by comparing the imaging image D121 and the imaging image D122, for example, the state of the load of the bucket can be grasped.

[0084] Further, the display image D1 includes supplementary information (image) D13. The supplementary information D13 includes a time series D131 of the estimated result of the work content based on the specified log information TL1, a time series D132 of the estimated result of the work content based on the comparison log information TL2, and a playback time icon D133. Note that the playback system 10 according to another embodiment may be configured to display the two operation images D111 and D112 side by side instead of superimposing them. In this case, they may be displayed on two displays arranged side by side.

[0085] The display image D2 shown in FIG. 12 includes a first display area DR1 and a second display area DR2. An operation image D21 is displayed in the first display area DR1. The operation image D21 is an image with a different viewpoint from the operation image D11 shown in FIG. 11. For example, the viewpoint can be changed by performing a predetermined mouse operation on the operation image D11. The operation image D21 is an image that includes, for example, an operation image D211 (displayed in solid lines) based on the designated log information TL1 and an operation image D212 (displayed in dashed lines) based on the comparison log information TL2 superimposed in different display colors. Further, the operation image D21 includes a terrain image D213 generated so as to be synthesized with the operation image D211 and the operation image D212. An imaging image D22 is displayed in the second display area DR2. The imaging image D22 includes an imaging image D221 based on the moving image information MV corresponding to the designated log information TL1 and an imaging image D222 based on the moving image information MV corresponding to the comparison log information TL2.

[0086] Further, the display image D2 includes supplementary information (image) D23. The supplementary information D23 includes a time series D231 of the estimated result of the work content based on the designated log information TL1, a time series D232 of the estimated result of the work content based on the comparison log information TL2, and a playback time icon D233.

[0087] The display image D1A shown in FIG. 13 includes a first display area DR1 and a second display area DR2. In the first display area DR1, an operation image D11 is displayed. The operation image D11 is an operation image D111 based on the specified log information TL1. Also, the operation image D11 includes a terrain image D113 generated to be synthesized with the operation image D111. In the second display area DR2, an imaging image D12 is displayed. The imaging image D12 is an imaging image D121 based on the moving image information MV corresponding to the specified log information TL1. In this example, the imaging image D12 includes images representing the loading platform of a dump truck that is a loading target such as excavated materials, a working machine, and the load in a bucket. In this example, the state of the load in the bucket, for example, can be grasped from the imaging image D121. Also, the display image D1 includes supplementary information (image) D13. The supplementary information D13 includes a time series D131 of the estimation result of the work content based on the specified log information TL1 and a playback time icon D133.

[0088] (Function, effect) As described above, the playback system 10 according to the first embodiment includes an acquisition unit 1000 and a playback unit 1003. The acquisition unit 1000 acquires the log information TL of the working machine 3 associated with the time and the moving image information MV representing the imaging video imaged by the working machine 3 and associated with the time. The playback unit 1003 sequentially applies the operation information representing the operation of the working machine 3 based on the log information TL to the working machine model TM, which is a three-dimensional model of the working machine 3, to sequentially reproduce the operation image D11 representing the operation of the working machine 3 within the first display area DR1 of the display unit 102, and synchronizes with the operation image D11 to sequentially reproduce the imaging image D12 based on the moving image information MV within a second display area DR2 different from the first display area DR1 of the display unit 102. According to this configuration, by referring to the imaging image captured by the working machine 3, the work performed by the working machine 3 can be analyzed in detail.

[0089] Also, in this embodiment, the acquisition unit 1000 acquires the log information TL and the moving image information MV of each of the plurality of working machines 3, and the playback unit 1003 superimposes the operation images D111 and D112 of each working machine 3 and sequentially plays them in the first display area DR1, and arranges and plays the captured images D121 and D122 based on the moving image information MV in the second display area DR2. According to this configuration, it is possible to simultaneously perform the animation display of a plurality of 3D models based on the plurality of log information TL and the video display of the plurality of captured images.

[0090] <Second Embodiment> Hereinafter, the playback system and the playback method according to the second embodiment will be described in detail with reference to FIGS. 14 to 16. FIG. 14 is a diagram showing the functional configuration of the playback system according to the second embodiment. FIG. 15 is a diagram showing the processing flow of the playback system according to the second embodiment. FIG. 16 is a diagram showing an example of a display image according to the second embodiment. The playback system 10a according to the second embodiment uses, for example, a plurality of working machines 3 that work (worked) at the same time at the same work site as the playback targets, displays each working machine 3 on an overhead image representing the work site, and identifies the working machine 3 to be played back on the overhead image, so that the working machine 3 to be played back can be switched.

[0091] The playback system 10a according to the second embodiment shown in FIG. 14 has the following different configuration from the playback system 10 according to the first embodiment shown in FIG. 4. That is, a part of the content of the acquisition unit 1000a shown in FIG. 14 is different from the acquisition unit 1000 shown in FIG. 4. Further, the CPU 100 shown in FIG. 14 newly includes a playback range specifying unit 1007 and an overhead image display unit 1008.

[0092] The playback range specifying unit 1007 specifies the range to be played back, for example, by specifying the area of the work site to be played back and the date and time to be played back according to the operator's operation. The area of the work site can be specified by specifying a coordinate range or by specifying the registration name of a previously registered coordinate range.

[0093] The bird's-eye view image display unit 1008 generates a bird's-eye view image including a plurality of working machines 3 located at the same time within the same work site area specified by the reproduction range specifying unit 1007, and displays the bird's-eye view image including the working machines 3 in a selectable form within a third display area DR3 different from the first display area DR1 and the second display area DR2 of the display unit 102.

[0094] The acquisition unit 1000a acquires all the log information TL that matches the specification conditions by the reproduction range specifying unit 1007 and is the estimation target by the estimation unit 1004, and also acquires the log information TL and the moving image information MV of the working machine 3 selected using the bird's-eye view image displayed by the bird's-eye view image display unit 1008 as the reproduction target by the reproduction unit 1003.

[0095] Regarding other configurations within the CPU 100, they are the same in the first embodiment and the second embodiment.

[0096] As shown in FIG. 15, in the reproduction system 10a of the second embodiment, first, the reproduction range specifying unit 1007 specifies the range to be the reproduction target (step S101). Next, the acquisition unit 1000a acquires all the log information TL that satisfies the reproduction conditions (step S102). Next, the estimation unit 1004 estimates the work content based on all the acquired log information TL (step S103). Next, the reception unit 1001 receives a reproduction instruction (step S104). Next, the bird's-eye view image display unit 1008 displays a bird's-eye view image (step S105). Next, the bird's-eye view image display unit 1008 identifies the working machine 3 to be reproduced based on a mouse operation or the like on the bird's-eye view image (step S106). In the subsequent processing of step S106, there may be cases where the working machine 3 to be reproduced is re-identified or not re-identified.

[0097] Next, the acquisition unit 1000 selects and reads out the working machine model TM corresponding to the referred working machine identification information from the storage 105 based on the working machine identification information as the type of the identified working machine 3 (step S107). Next, the extraction unit 1002 extracts the information to be used for reproduction from the log information TL1 (step S108).

[0098] Next, the playback unit 1003 performs playback processing (step S109). In step S109, the operation image playback (S109a), moving image playback (S109b), terrain image display (S109c), and supplementary information display (S109d) are respectively processes corresponding to the operation image playback (S07a), moving image playback (S07b), terrain image display (S07c), and supplementary information display (S07d) shown in FIG. 5, and the description thereof is omitted.

[0099] Also, the determination process in step S110 is the same as the determination process in step S08 shown in FIG. 5. However, in the case of step S110, the return destination when playback is not terminated is step S105.

[0100] FIG. 16 shows an example of a display image according to the second embodiment. The display image D3 shown in FIG. 16 includes a first display area DR1, a second display area DR2, and a third display area DR3. The first display area DR1 includes an operation image D31. In this example, the operation image D31 includes an operation image D311 of a specific work machine and a terrain image D313. The second display area DR2 includes a captured image D32. The third display area DR3 includes an aerial view image D34. In this case, the aerial view image D34 includes icons D341 and D342 corresponding to two work machines 3. The icons D341 and D342 indicate the imaging direction by a triangular broken line area and the moving direction by a broken line arrow.

[0101] Also, the display image D3 includes supplementary information D33. The supplementary information D33 includes a time series of the estimated result of the work content based on the designated log information TL, a time series indicating the control state of the automatic control (for example, the state of the position accuracy), and a playback time icon D333.

[0102] According to this embodiment, the playback system 10a newly includes an overhead image display unit 1008. The overhead image display unit 1008 generates an overhead image D34 including a plurality of working machines 3 located at the same time in the same work site area, and displays the overhead image D34 including the working machines in a selectable form in a third display area DR3 different from the first display area DR1 and the second display area DR2 of the display unit 102. Further, the acquisition unit 1000a acquires the log information TL and the moving image information MV of the working machine 3 selected in the overhead image D34. According to this configuration, for example, when a plurality of working machines 3 are to be analyzed (or monitored), the working machine 3 to be played back can be easily selected.

[0103] (Modification example) The content of the log information TL according to the above embodiment is not limited to this in other embodiments. For example, when the working machine 3 is not a hydraulic excavator but another vehicle type, the log information TL corresponding to the vehicle type is recorded. Other vehicle types are, for example, wheel loaders, bulldozers, etc.

[0104] Further, the playback system 10 according to the above embodiment has been described as being installed at a location away from the working machine 3 and connected to the data logger 20 mounted on the working machine 3 via a wide area communication network, but is not limited to this mode in other embodiments.

[0105] For example, in the playback system 10 according to other embodiments, part or all of the configuration of the playback system 10 may be installed inside the working machine 3. In this case, the data logger 20 may transmit the log information TL to the playback system 10 via a network inside the working machine 3 without passing through a wide area communication network. By doing so, the operator boarding the working machine 3 can play back and confirm the movement of the working machine 3 operated by the operator himself on the spot. Further, by playing back the movement of the model working machine 3 for the operator of the working machine 3, it can be used as guidance.

[0106] Note that the playback system 10 installed inside the working machine 3 may acquire the log information TL of other working machines 3 via a wide area communication network or the like. By doing so, the state of the working machine 3 other than the working machine 3 equipped with the playback system 10 can be animated and played back.

[0107] Further, the playback system 10 according to another embodiment may be configured to transmit the video information generated by the animation playback process to the monitor mounted on the working machine 3 for display while being installed at a location away from the working machine 3.

[0108] Also, in another embodiment, as one aspect of the playback instruction received from the operator, for example, the playback period may be specified. For example, the playback period may be the playback start time and the playback end time. In this case, the playback system 10 performs the playback of the working machine 3 for the received playback period. Also, in another embodiment, the specification of the playback end time is not essential. For example, in another embodiment, as the playback instruction from the operator, only the playback start time may be received and the playback may be performed for a certain period of time from the playback start time, or the playback may continue as long as the log information exists, or the playback may be stopped in accordance with the occurrence of various other events.

[0109] Note that the log information TL to be acquired does not have to be arranged in chronological order. In this case, the playback unit 1003 may apply the information used for playback among the log information TL to the working machine model TM in chronological order.

[0110] Note that the processes of the various processes of the playback system 10 described above are stored in a computer-readable recording medium in the form of a program, and the various processes are performed by the computer reading and executing this program. Also, the computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, etc. Further, this computer program may be distributed to the computer via a communication line, and the computer that has received this distribution may execute the program.

[0111] The above program may be for realizing a part of the functions described above. Further, it may be a so-called difference file, difference program, etc. that can be realized in combination with a program already recorded in the computer system for realizing the functions described above.

[0112] A part or all of the functions of the above-described reproduction system 10 may be assigned to the working machine 3. For example, some or all of the functions of the acquisition unit 1000, reception unit 1001, extraction unit 1002, reproduction unit 1003, estimation unit 1004, determination unit 1005, synchronization unit 1006, memory 101, display unit 102, operation reception unit 103, communication interface 104, and storage 105 may be assigned to the working machine 3.

[0113] As described above, some embodiments of the present disclosure have been described. However, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalent scope thereof.

[0114] (Supplementary Note) The reproduction systems 10 and 10a according to the present disclosure are grasped as follows, for example.

[0115] (1) The playback systems 10 and 10a according to the first aspect acquire log information TL of the working machine 3 associated with time, and moving image information MV representing a captured video captured by the working machine and associated with time. The acquisition units 1000 and 1000a sequentially apply operation information representing the operation of the working machine based on the log information to a three-dimensional model (working machine model TM) of the working machine to sequentially reproduce operation images D11, D21, and D31 representing the operation of the working machine within a first display area DR1 of a predetermined display unit 102. At the same time, a playback unit 1003 sequentially reproduces captured images D12, D22, and D32 based on the moving image information MV in synchronization with the operation images within a second display area DR2 different from the first display area of the display unit. According to this aspect and the following aspects, the work by the working machine can be analyzed in more detail.

[0116] (2) The playback system 10a according to the second aspect is the playback system of (1), and generates an overhead view image D34 including a plurality of working machines located at the same time within the same work site area, and further includes an overhead view image display unit 1008 that displays the overhead view image including each of the working machines in a selectable form within a third display area DR3 different from the first and second display areas of the display unit. The acquisition unit 1000a acquires the log information and the moving image information of the working machine selected in the overhead view image.

[0117] (3) The playback systems 10 and 10a according to the third aspect are the playback systems of (1) or (2), and the acquisition units 1000 and 1000a acquire the log information and the moving image information of each of the plurality of working machines. The playback unit 1003 sequentially reproduces the operation images of each of the working machines by superimposing them within the first display area, and arranges and reproduces the captured images based on each of the moving image information within the second display area.

Explanation of Reference Numerals

[0118] 1 Analysis support system, 10 Reproduction system, 100 CPU, 1000 Acquisition unit, 1001 Reception unit, 1002 Extraction unit, 1003 Reproduction unit, 1004 Estimation unit, 1005 Judgment unit, 1006 Synchronization unit, 1007 Reproduction range designation unit, 1008 Overhead image display unit, 101 Memory, 102 Display unit, 103 Operation reception unit, 104 Communication interface, 105 Storage, 20 Data logger, 3 Working machine, H1 to H2 Heat map, TM Working machine model, TL Log information, PM1 Unit work prediction model, PM2 Element work prediction model, R Estimated work content, D11, D21, D31 Motion images, D12, D22, D32 Captured images

Claims

1. An acquisition unit that acquires log information of a working machine associated with a time and moving image information representing a captured video captured by the working machine and associated with a time; A playback unit that sequentially plays back operation images representing the operation of the working machine by sequentially applying operation information representing the operation of the working machine based on the log information to a three-dimensional model of the working machine, and sequentially plays back captured images based on the moving image information in synchronization with the operation images; A playback system comprising the above.

2. The playback unit sequentially plays back the operation images within a first display area of a predetermined display unit, and sequentially plays back the captured images in synchronization with the operation images within a second display area different from the first display area of the display unit. The playback system according to Claim 1.

3. A bird's-eye view image display unit that generates a bird's-eye view image including a plurality of working machines located at the same time within the same work site area, and displays the bird's-eye view image including each of the working machines in a selectable form within a third display area different from the first and second display areas of the display unit; The acquisition unit acquires the log information and the moving image information of the working machine selected in the bird's-eye view image. The playback system according to Claim 2.

4. The acquisition unit acquires the log information and the moving image information of each of the plurality of working machines; The playback unit sequentially plays back the operation images of each of the working machines by superimposing them within the first display area, and plays back the captured images based on each of the moving image information side by side within the second display area. The playback system according to Claim 2 or 3.

5. A step of acquiring log information of a working machine associated with a time and moving image information representing a captured video captured by the working machine and associated with a time; A step of sequentially playing back operation images representing the operation of the working machine by sequentially applying operation information representing the operation of the working machine based on the log information to a three-dimensional model of the working machine, and sequentially playing back captured images based on the moving image information in synchronization with the operation images; A playback method including the above.

Images