Monitoring system and monitoring method for industrial machinery

Abstract

To monitor state of a working machine during the working thereof.SOLUTION: The monitoring system of a working machine comprises: a first imaging apparatus which is provided in a working machine, has a first viewing angle and images a working implement of the working machine; a second imaging apparatus which is provided in the working machine, has a second viewing angle wider than the first viewing angle, and images the ground on which the working machine runs; a defect determination unit that determines the existence of defects on a working implement based on image data captured by the first imaging apparatus; and a determination unit that determines the existence of a bounding stone on the ground based on image data captured by the second imaging apparatus.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present disclosure relates to a monitoring system for a working machine and a method for monitoring a working machine.

Background Art

[0002] In the technical field related to working machines, a working vehicle as disclosed in Patent Document 1 is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] At least a part of the working machine may be damaged depending on the work content or the working environment. By monitoring the condition of the working machine during work, it may be possible to prevent damage to the working machine or to recognize damage to the working machine at an early stage.

[0005] The present disclosure aims to monitor the condition of a working machine during work.

Means for Solving the Problems

[0006] According to the present disclosure, there is provided a monitoring system for a working machine, including: a first imaging device provided on the working machine and having a first imaging angle for imaging a working tool of the working machine; a second imaging device provided on the working machine and having a second imaging angle wider than the first imaging angle for imaging the ground on which the working machine travels; a defect determination unit that determines the presence or absence of a defect of the working tool based on the image data captured by the first imaging device; and a boulder determination unit that determines the presence or absence of boulders on the ground based on the image data captured by the second imaging device.

Effects of the Invention

[0007] According to this disclosure, the status of the work machine can be monitored during operation. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a side view showing a work machine according to an embodiment. [Figure 2] Figure 2 is a top view showing a work machine according to an embodiment. [Figure 3] Figure 3 is a front view showing a work machine according to an embodiment. [Figure 4] Figure 4 shows the driver's cab according to the embodiment of this design. [Figure 5] Figure 5 is an enlarged front view of a part of the work machine according to the embodiment. [Figure 6] Figure 6 shows the drive system of a work machine according to an embodiment. [Figure 7] Figure 7 is a diagram illustrating the operation of the work machine according to the embodiment. [Figure 8] Figure 8 is a diagram illustrating the operation of the work machine according to the embodiment. [Figure 9] Figure 9 is a side view showing the work machine that performs the loading operation according to the embodiment. [Figure 10] Figure 10 is a top view showing the work machine that performs the loading operation according to the embodiment. [Figure 11] Figure 11 shows an example of imaging data captured by the first imaging device on the left according to this embodiment. [Figure 12] Figure 12 shows an example of imaging data captured by the second imaging device on the left according to this embodiment. [Figure 13] Figure 13 is a functional block diagram showing a monitoring system according to an embodiment. [Figure 14] Figure 14 illustrates a defect detection method using a defect detection unit according to an embodiment. [Figure 15] Figure 15 illustrates a defect detection method using a defect detection unit according to an embodiment of this example. [Figure 16]FIG. 16 is a diagram for explaining a defect determination method by a defect determination unit according to an embodiment. [Figure 17] FIG. 17 is a diagram for explaining a defect determination method by a defect determination unit according to an embodiment. [Figure 18] FIG. 18 is a diagram for explaining a defect determination method by a defect determination unit according to an embodiment. [Figure 19] FIG. 19 is a diagram showing an example of a soundness determination frame according to an embodiment. [Figure 20] FIG. 20 is a diagram showing an example of a defect determination frame according to an embodiment. [Figure 21] FIG. 21 is a diagram showing an example of a symbol indicating a bucket tooth displayed on a display device according to an embodiment. [Figure 22] FIG. 22 is a diagram showing the relationship between a defect determination frame and a warning form according to an embodiment. [Figure 23] FIG. 23 is a diagram for explaining a determination criterion according to an embodiment. [Figure 24] FIG. 24 is a diagram for explaining a warning criterion according to an embodiment. [Figure 25] FIG. 25 is a diagram showing an example of a setting screen for a determination criterion and a warning criterion according to an embodiment. [Figure 26] FIG. 26 is a flowchart showing a defect monitoring method for a work machine according to an embodiment. [Figure 27] FIG. 27 is a flowchart showing a boulder monitoring method for the ground according to an embodiment. [Figure 28] FIG. 28 is a diagram showing an example of an output device according to an embodiment. [Figure 29] FIG. 29 is a diagram showing an example of an output device according to an embodiment. [Figure 30] FIG. 30 is a block diagram showing a computer system according to an embodiment. [Figure 31] FIG. 31 is a diagram for explaining the operation of a work machine according to another embodiment. [Figure 32] FIG. 32 is a diagram showing a monitoring system according to another embodiment. [Modes for carrying out the invention]

[0009] The embodiments of this disclosure will be described below with reference to the drawings, but this disclosure is not limited to these embodiments. The components of the embodiments described below can be combined as appropriate. In addition, some components may not be used.

[0010] [Working machinery] Figure 1 is a side view showing the work machine 1 according to the embodiment. Figure 2 is a top view showing the work machine 1 according to the embodiment. Figure 3 is a front view showing the work machine 1 according to the embodiment.

[0011] The work machine 1 performs work at the work site. In this embodiment, the work machine 1 is a wheel loader, which is a type of articulated work machine. The work machine 1 performs excavation work to excavate the target to be excavated, and loading work to load the excavated material onto the loading target.

[0012] As shown in Figures 1, 2, and 3, the work machine 1 comprises a vehicle body 2, a driver's cab 3, a running gear 4, a front fender 7, a support member 8, a housing 9, and a work implement 10.

[0013] The vehicle body 2 includes a front vehicle body 2F and a rear vehicle body 2R. The front vehicle body 2F and the rear vehicle body 2R are connected via an articulation mechanism 2A.

[0014] The driver's cab 3 is supported by the vehicle body 2. A driver's compartment is provided in the driver's cab 3. A driver's seat is provided in the driver's compartment. The work machine 1 is operated by a driver seated in the driver's cab 3.

[0015] The running gear 4 supports the vehicle body 2 and travels on the ground 200 of the work site. The running gear 4 has wheels 5 and tires 6. The tires 6 are mounted on the wheels 5. The wheels 5 include front wheels 5F supported by the front vehicle body 2F and rear wheels 5R supported by the rear vehicle body 2R. The tires 6 include front tires 6F mounted on the front wheels 5F and rear tires 6R mounted on the rear wheels 5R.

[0016] The front wheel 5F and front tire 6F can rotate around axis FX. The rear wheel 5R and rear tire 6R can rotate around axis RX.

[0017] In this embodiment, the direction parallel to the rotation axis FX of the front wheel 5F is appropriately referred to as the vehicle width direction. The direction perpendicular to the contact surface of the front tire 6F that contacts the ground 200 is appropriately referred to as the vertical direction. The direction perpendicular to both the vehicle width direction and the vertical direction is appropriately referred to as the longitudinal direction. When the vehicle body 2 of the work machine 1 is traveling in a straight line, the rotation axis FX and the rotation axis RX are parallel.

[0018] Furthermore, in this embodiment, a position or direction close to the center CL of the work machine 1 in the vehicle width direction is appropriately referred to as the "inside in the vehicle width direction," and a position or direction far from the center CL is appropriately referred to as the "outside in the vehicle width direction."

[0019] In the vehicle width direction, one side from the center CL of the work machine 1 is the left, and the opposite side of the left is the right. In the front-rear direction, the position or direction closest to the work machine 10, relative to the driver's seat of the driver's cab 3, is the front, and the opposite side of the front is the rear. In the up-down direction, the position or direction closest to the contact surface of the front tire 6F is the bottom, and the opposite side of the bottom is the top.

[0020] The rear body 2R is positioned behind the front body 2F. The front body 2F bends to the left and right relative to the rear body 2R. The rear wheel 5R is positioned behind the front wheel 5F. The rear tire 6R is positioned behind the front tire 6F. The front wheel 5F and front tire 6F are positioned to the left and right, respectively, of the center CL of the work machine 1 in the vehicle width direction. The rear wheel 5R and rear tire 6R are positioned to the left and right, respectively, of the center CL of the work machine 1 in the vehicle width direction.

[0021] The tire 6 is a rotating member that rotates while in contact with the ground 200. The rotation of the tire 6 causes the work machine 1 to travel along the ground 200. The front tire 6F is a front rotating member that rotates while in contact with the ground 200. The rear tire 6R is positioned behind the front tire 6F and is a rear rotating member that rotates while in contact with the ground 200.

[0022] The front fender 7 prevents soil and sand scattered from the ground 200 from hitting the vehicle body 2 and the driver's cab 3 during the movement of the work machine 1. Part of the front fender 7 is positioned above the front tire 6F. Part of the front fender 7 is positioned behind the front tire 6F. The front fender 7 is positioned to the left and right of the center CL of the work machine 1 in the vehicle width direction. The left front fender 7 is attached to the left side of the front vehicle body 2F. The right front fender 7 is attached to the right side of the front vehicle body 2F.

[0023] The support member 8 is a rod-shaped member. The lower end of the support member 8 is fixed to the front body 2F. The support member 8 is inclined upward toward the front and outward in the vehicle width direction. The support members 8 are positioned to the left and right of the center CL of the work machine 1 in the vehicle width direction of the work machine 1. The lower end of the left support member 8 is fixed to the left side of the front body 2F. The left support member 8 is inclined upward toward the front and left. The lower end of the right support member 8 is fixed to the right side of the front body 2F. The right support member 8 is inclined upward toward the front and right.

[0024] The housing 9 is fixed to the upper end of the support member 8. The housing 9 is supported by the front body 2F via the support member 8. In the longitudinal direction, the housing 9 is positioned between the articulation mechanism 2A and the front end of the front body 2F. In the vertical direction, the housing 9 is positioned between the upper end of the driver's cab 3 and the upper end of the front body 2F. The housings 9 are positioned to the left and right of the center CL of the work machine 1 in the vehicle width direction of the work machine 1. The left housing 9 is fixed to the upper end of the left support member 8. The right housing 9 is fixed to the upper end of the right support member 8. The left housing 9 is positioned above the left front fender 7. The right housing 9 is positioned above the right front fender 7.

[0025] The work implement 10 operates during excavation and loading operations. The work implement 10 is connected to the front vehicle body 2F. At least a portion of the work implement 10 is positioned in front of the front wheels 5F. The work implement 10 has a boom 11, a bucket 12, a bell crank 15, and a bucket link 16.

[0026] The boom 11 is rotatably connected to the front body 2F. The boom 11 connects the front body 2F to the bucket 12. The boom 11 operates by the driving force generated by the boom cylinder 13. The boom cylinder 13 is a hydraulic cylinder. One end of the boom cylinder 13 is connected to the front body 2F. The other end of the boom cylinder 13 is connected to the boom 11. The boom cylinder 13 is positioned to the left and right of the center CL of the work machine 1 in the vehicle width direction of the work machine 1.

[0027] The bucket 12 excavates the target to be excavated. The bucket 12 is rotatably connected to the tip of the boom 11. The bucket 12 is positioned in front of the front wheel 5F. The bucket 12 is operated by the driving force generated by the bucket cylinder 14. The bucket cylinder 14 is a hydraulic cylinder. The central part of the bell crank 15 is rotatably connected to the boom 11. One end of the bucket cylinder 14 is connected to the front body 2F. The other end of the bucket cylinder 14 is connected to one end of the bell crank 15. The other end of the bell crank 15 is connected to the bucket 12 via the bucket link 16. One bucket cylinder 14 is positioned in the center in the width direction of the vehicle.

[0028] The bucket 12 comprises a bucket body 17, bucket teeth 18, and tooth protectors 19.

[0029] The bucket body 17 holds the excavated material. The bucket body 17 includes a bottom plate 17A, an upper plate 17B, a left plate 17C, and a right plate 17D. The bottom plate 17A, upper plate 17B, left plate 17C, and right plate 17D form a holding space in which the excavated material is contained. The tip of the bottom plate 17A and the tip of the upper plate 17B each extend in the vehicle width direction. The tip of the left plate 17C and the tip of the right plate 17D each extend in the vertical or longitudinal direction. The tip of the bottom plate 17A, the tip of the upper plate 17B, the tip of the left plate 17C, and the tip of the right plate 17D define the opening 12M of the bucket 12. The excavated material can enter the holding space of the bucket 12 through the opening 12M.

[0030] As shown in Figures 2 and 3, the ends 12E on both sides of the bucket body 17 in the vehicle width direction are positioned outside the tires 6 in the vehicle width direction. The distance in the vehicle width direction between the left end 12E and the right end 12E is greater than the distance in the vehicle width direction between the left side of the left tire 6 and the right side of the right tire 6.

[0031] The bucket teeth 18 constitute the cutting edge of the bucket 12. The bucket teeth 18 are attached to the bucket body 17. The bucket teeth 18 are attached to the tip of the bottom plate portion 17A. Multiple bucket teeth 18 are attached to the bucket body 17. The multiple bucket teeth 18 are arranged at intervals in the vehicle width direction.

[0032] The bucket teeth 18 are replaceable components that are interchangeably attached to the bucket body 17. The bucket teeth 18 are fixed to the bucket body 17, for example, by bolts. By releasing the bolts, the bucket teeth 18 can be separated from the bucket body 17.

[0033] In this embodiment, eight bucket teeth 18 are arranged at intervals in the vehicle width direction. The bucket teeth 18 include a bucket tooth 181 positioned on the far left, a bucket tooth 182 positioned to the left of bucket tooth 181, a bucket tooth 183 positioned to the left of bucket tooth 182, a bucket tooth 184 positioned to the left of bucket tooth 183, a bucket tooth 185 positioned to the left of bucket tooth 184, a bucket tooth 186 positioned to the left of bucket tooth 185, a bucket tooth 187 positioned to the left of bucket tooth 186, and a bucket tooth 188 positioned on the far right.

[0034] The tooth-tooth protector 19 protects the leading edge of the bottom plate portion 17A. The tooth-tooth protector 19 is attached to the bucket body 17. The tooth-tooth protector 19 is positioned between a pair of adjacent bucket teeth 18. The tooth-tooth protector 19 is attached to the leading edge of the bottom plate portion 17A. Multiple tooth-tooth protectors 19 are attached to the bucket body 17. Multiple tooth-tooth protectors 19 are arranged with spacing in the vehicle width direction.

[0035] The tooth-tooth protector 19 is a replaceable component that is removably attached to the bucket body 17. The tooth-tooth protector 19 is fixed to the bucket body 17, for example, by bolts. By releasing the bolt fixation, the tooth-tooth protector 19 can be separated from the bucket body 17.

[0036] In this embodiment, seven tooth-tooth protectors 19 are arranged at intervals in the vehicle width direction. The tooth-tooth protectors 19 include a tooth-tooth protector 191 positioned on the far left, a tooth-tooth protector 192 positioned to the left of tooth-tooth protector 192, a tooth-tooth protector 193 positioned to the left of tooth-tooth protector 193, a tooth-tooth protector 194 positioned to the left of tooth-tooth protector 194, a tooth-tooth protector 196 positioned to the left of tooth-tooth protector 195, and a tooth-tooth protector 197 positioned on the far right.

[0037] [Driver's cab] Figure 4 is a diagram showing the driver's cab of the driver's seat 3 according to the embodiment. Figure 4 is a schematic diagram showing the view from the perspective of a driver seated in the driver's seat of the driver's cab.

[0038] The driver's cab is equipped with a driving control device 20, an operation panel 21, a monitor device 22, a rear-view monitor device 23, an output device 24, and an input device 25. The driving control device 20 is operated by the driver. The driving control device 20 includes an accelerator pedal 20A and a pair of brake pedals 20B. Although not shown, the driving control device 20 also includes a steering lever, a shift lever, a forward / reverse switch, and a work lever. When the accelerator pedal 20A is operated, the travel speed of the work machine 1 increases. When the brake pedal 20B is operated, the travel speed of the work machine 1 decreases or the work machine 1 stops. When the steering lever is operated, the work machine 1 turns. When the shift lever is operated, the speed gear of the work machine 1 is changed. When the forward / reverse switch is operated, the direction of travel of the work machine 1 is switched between forward and reverse. When the work lever is operated, the work machine 10 is operated.

[0039] The output device 24 provides output data to the driver. The output data includes warnings. The output device 24 includes a display device 24A and a sound-emitting device 24B.

[0040] The display device 24A provides display data to the driver as output data. Examples of the display device 24A include flat panel displays such as liquid crystal displays (LCDs) or organic electroluminescence displays (OELDs).

[0041] The sound-producing device 24B provides voice data to the driver as output data. Examples of the sound-producing device 24B include a buzzer or a speaker.

[0042] The input device 25 generates input data when operated by the driver. Examples of the input device 25 include buttons or a keyboard. The input device 25 may also include, for example, a touch panel.

[0043] A windshield is provided at the front of the driver's cab. The windshield is attached to the pillar 27 of the driver's cab. In this embodiment, the output device 24 is supported by the pillar 27.

[0044] [Measuring device] Figure 5 is an enlarged front view of a part of the work machine 1 according to the embodiment. As shown in Figures 3 and 5, the work machine 1 has a headlight 28, a turn signal lamp 29, a first measuring device 31, and a second measuring device 32.

[0045] The headlights 28 emit illumination light forward to illuminate the area in front of the front body 2F. The headlights 28 are held in a housing 9. The headlights 28 are positioned to the left and right of the center CL of the work machine 1 in the vehicle width direction of the work machine 1. The left headlight 28 is held in the left housing 9. The right headlight 28 is held in the right housing 9.

[0046] The turn signal lamps 29 indicate the turning direction of the work machine 1 by illuminating or flashing. The turn signal lamps 29 are held in a housing 9. The turn signal lamps 29 are positioned to the left and right of the center CL of the work machine 1 in the vehicle width direction. The left turn signal lamp 29 is held in the left housing 9. The right turn signal lamp 29 is held in the right housing 9. In the vehicle width direction of the work machine 1, the turn signal lamps 29 are positioned outside the headlights 28.

[0047] The first measuring device 31 measures the work machine 10. The first measuring device 31 acquires measurement data of the work machine 10. In this embodiment, the first measuring device 31 is an imaging device that images the work machine 10. The measurement data of the work machine 10 includes image data of the work machine 10. In the following description, the first measuring device 31 will be referred to as the first imaging device 31 as appropriate.

[0048] The second measuring device 32 measures the ground 200 on which the work machine 1 travels. The second measuring device 32 acquires measurement data of the ground 200. In this embodiment, the second measuring device 32 is an imaging device that images the ground 200. The measurement data of the ground 200 includes image data of the ground 200. In the following description, the second measuring device 32 will be referred to as the second imaging device 32 as appropriate.

[0049] Each of the first imaging device 31 and the second imaging device 32 includes an optical system and an image sensor into which light that has passed through the optical system is incident. Examples of image sensors include a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0050] The first imaging device 31 and the second imaging device 32 are each located in the housing 9. In the vehicle width direction of the work machine 1, the first imaging device 31 and the second imaging device 32 are each located outside the headlights 28. In the housing 9, the second imaging device 32 is located above the first imaging device 31.

[0051] The first imaging device 31 is positioned to the left and right of the center CL of the work machine 1 in the vehicle width direction of the work machine 1. The first imaging device 31 is positioned in the left housing 9 and the right housing 9, respectively. The first imaging device 31 includes a first imaging device 31L positioned to the left of the center CL and a first imaging device 31R positioned to the right of the center CL.

[0052] The second imaging device 32 is positioned to the left and right of the center CL of the work machine 1 in the vehicle width direction of the work machine 1. The second imaging device 32 is positioned in the left housing 9 and the right housing 9, respectively. The second imaging device 32 includes a second imaging device 32L positioned to the left of the center CL and a second imaging device 32R positioned to the right of the center CL.

[0053] [Drive System] Figure 6 shows the drive system 40 of the work machine 1 according to the embodiment. The drive system 40 includes an engine 41, a fuel injector 42, a power take-off 43 (PTO), a transmission 44, a front axle 45F, a rear axle 45R, a hydraulic pump 46, a control valve 47, and a drive controller 48.

[0054] Engine 41 is, for example, a diesel engine. Fuel injector 42 injects fuel into the cylinders of engine 41. The driving force of engine 41 is adjusted by adjusting the amount of fuel injected from fuel injector 42 into engine 41.

[0055] The power takeoff 43 distributes the driving force of the engine 41 to the transmission 44 and the hydraulic pump 46. The driving force of the engine 41 is transmitted to the transmission 44 and the hydraulic pump 46 respectively via the power takeoff 43.

[0056] The transmission 44 has an input shaft to which the driving force from the engine 41 is input, and an output shaft that shifts the driving force input to the input shaft and outputs it. The input shaft of the transmission 44 is connected to the power takeoff 43. The output shaft of the transmission 44 is connected to the front axle 45F and the rear axle 45R, respectively. The driving force from the engine 41 is transmitted to the front axle 45F and the rear axle 45R, respectively, via the power takeoff 43 and the transmission 44.

[0057] The transmission 44 switches the direction of travel of the work machine 1 between forward and reverse. The transmission 44 has a forward gear 44F and a reverse gear 44R. When the forward gear 44F is engaged, the direction of travel of the work machine 1 is set to forward. When the reverse gear 44R is engaged, the direction of travel of the work machine 1 is set to reverse.

[0058] The front axle 45F transmits the driving force from the transmission 44 to the front wheels 5F. The front wheels 5F rotate based on the driving force transmitted from the front axle 45F.

[0059] The rear axle 45R transmits the driving force from the transmission 44 to the rear wheel 5R. The rear wheel 5R rotates based on the driving force transmitted from the rear axle 45R.

[0060] When the forward gear 44F is engaged, the work machine 1 becomes capable of moving forward. With the forward gear 44F engaged, when the accelerator pedal 20A is operated, the driving force of the engine 41 is transmitted via the transmission 44 to the front axle 45F and the rear axle 45R, causing the work machine 1 to move forward. However, even when the forward gear 44F is engaged, the work machine 1 will not move forward if, for example, the brake pedal 20B is operated.

[0061] When the reverse gear 44R is engaged, the work machine 1 becomes capable of moving in reverse. With the reverse gear 44R engaged, when the accelerator pedal 20A is operated, the driving force of the engine 41 is transmitted to the front axle 45F and the rear axle 45R via the transmission 44, causing the work machine 1 to move in reverse. However, even when the reverse gear 44R is engaged, the work machine 1 will not move in reverse if, for example, the brake pedal 20B is operated.

[0062] The hydraulic pump 46 discharges hydraulic fluid. The hydraulic pump 46 is a variable displacement hydraulic pump. The hydraulic pump 46 is driven based on the driving force of the engine 41. The hydraulic fluid discharged from the hydraulic pump 46 is supplied to at least one of the boom cylinder 13 and the bucket cylinder 14 via the control valve 47.

[0063] The control valve 47 controls the flow rate and direction of the hydraulic fluid supplied to the boom cylinder 13 and the bucket cylinder 14, respectively. The work machine 10 is operated by the hydraulic fluid supplied from the hydraulic pump 46 via the control valve 47.

[0064] The drive controller 48 controls the work machine 1 based on the operation signals output from the operating device 20. The drive controller 48 includes a computer system.

[0065] [Operation of the work machine] Figure 7 is a diagram illustrating the operation of the work machine 10 according to the embodiment. In this embodiment, the work machine 10 is a front-loading type work machine in which the opening 12M of the bucket 12 faces forward during excavation work.

[0066] The boom cylinder 13 extends and retracts, causing the boom 11 to move up or down.

[0067] The raising motion of the boom 11 refers to the movement of the boom 11 so that its tip is separated from the ground 200. The boom 11 is raised as the boom cylinder 13 extends.

[0068] The lowering motion of the boom 11 refers to the movement of the boom 11 so that its tip approaches the ground 200. The boom 11 lowers as the boom cylinder 13 retracts.

[0069] As the bucket cylinder 14 extends and retracts, the bucket 12 tilts or dumps.

[0070] The tilting motion of the bucket 12 refers to the movement of the bucket 12 so that the tip of the bucket 12 is lifted away from the ground 200 while the opening 12M of the bucket 12 is facing upward. As the bucket cylinder 14 extends, the bell crank 15 rotates so that the upper end of the bell crank 15 moves forward and the lower end of the bell crank 15 moves backward. When the lower end of the bell crank 15 moves backward, the bucket 12 is pulled backward by the bucket link 16 and tilts. As the bucket 12 tilts, the material to be excavated is scooped up by the bucket 12 and the excavated material is held in the bucket 12.

[0071] The dumping operation of the bucket 12 refers to the movement of the bucket 12 so that its tip approaches the ground 200 with the opening 12M of the bucket 12 facing downwards. As the bucket cylinder 14 retracts, the bell crank 15 rotates so that its upper end moves backward and its lower end moves forward. When the lower end of the bell crank 15 moves forward, the bucket 12 is pushed forward by the bucket link 16, causing it to dump. As the bucket 12 dumps, the excavated material held in the bucket 12 is discharged from the bucket 12.

[0072] [Operation of the work machine] Figure 8 is a diagram illustrating the operation of the work machine 1 according to this embodiment. The work machine 1 performs excavation and loading operations at the work site.

[0073] Excavation work is the work of excavating the target to be excavated. Examples of the target to be excavated include natural ground or rocky mountains. Natural ground is a mountain composed of soil and sand placed on the ground 200. Rocky mountains are a mountain composed of rocks or stones placed on the ground 200. In this embodiment, the target to be excavated is natural ground 210. The excavated material is a portion of the natural ground 210 that has been excavated and held in the bucket 12.

[0074] The loading operation is the process of loading the excavated material obtained from the excavation operation onto a loading target. Examples of loading targets include transport vehicles, hoppers, or belt conveyors. In one embodiment, the loading target is the dump body 230 of a dump truck 220, which is a type of transport vehicle.

[0075] The work machine 1 is operated by the operator to perform a cyclical operation that repeats a series of work modes. The cyclical operation consists of multiple interrelated work modes. The cyclical operation includes the movement of the travel device 4 and the operation of the work machine 10.

[0076] In one embodiment, the cycle operation consists of six work modes. In one embodiment, the series of work modes of the work machine 1 includes an empty forward mode, an excavation mode, a loaded reverse mode, a loaded forward mode, a loading mode, and an empty reverse mode.

[0077] The sequence of operations is predetermined. After the empty forward mode is performed, the excavation mode is performed. After the excavation mode is performed, the loaded reverse mode is performed. After the loaded reverse mode is performed, the loaded forward mode is performed. After the loaded forward mode is performed, the loading mode is performed. After the loading mode is performed, the empty reverse mode is performed.

[0078] The empty-load forward mode is a working mode in which the work machine 1 moves forward to approach the excavation target when no excavated material is held in the bucket 12. In the empty-load forward mode, the work machine 1 moves forward to approach the ground 210 as shown by arrow M1 in Figure 8, with no excavated material held in the bucket 12.

[0079] The excavation mode is a work mode in which the bucket 12 of the work machine 10 excavates the target to be excavated. Excavation work is performed in excavation mode. With at least a part of the bucket 12 inserted into the ground 210, the bucket 12 tilts, causing the ground 210 to be excavated by the bucket 12, and the excavated material is held in the bucket 12.

[0080] The load reverse mode is a working mode in which the work machine 1 reverses to move away from the excavation target while the excavated material is held in the bucket 12. In the load reverse mode, the work machine 1 reverses to move away from the ground 210, as shown by arrow M2 in Figure 8, while the excavated material is held in the bucket 12.

[0081] The load advance mode is a working mode in which the work machine 1 moves forward to approach the loading target while the excavated material is held in the bucket 12. In the load advance mode, the work machine 1 moves forward while turning toward the dump truck 220, as shown by arrow M3 in Figure 8, with the excavated material held in the bucket 12. While the work machine 1 is moving forward toward the dump truck 220, the boom 11 moves upward so that the bucket 12 is positioned above the dump body 230.

[0082] The loading mode is a work mode in which the excavated material held in the bucket 12 of the work machine 10 is loaded onto the loading target. In loading mode, the loading operation is performed. After the bucket 12 is positioned above the dump body 230, the bucket 12 performs a dumping operation, causing the excavated material held in the bucket 12 to be discharged from the bucket 12 and loaded onto the dump body 230.

[0083] The empty reverse mode is a working mode in which the work machine 1 reverses to move away from the loading target when there is no excavated material held in the bucket 12. With no excavated material held in the bucket 12, the work machine 1 reverses while turning to move away from the dump truck 220, as shown by arrow M4 in Figure 8.

[0084] The work machine 1 repeats a cycle of operations including empty forward mode, excavation mode, loaded reverse mode, loaded forward mode, loading mode, and empty reverse mode until the excavated material is loaded into the dump body 230 to the target load capacity.

[0085] [Measurement range] Figure 9 is a side view showing the work machine 1 that performs the loading operation according to the embodiment. Figure 10 is a top view showing the work machine 1 that performs the loading operation according to the embodiment.

[0086] As shown in Figures 9 and 10, when loading excavated material into the dump body 230 of the dump truck 220, the working machine 1 raises the boom 11 so that the bucket 12 is positioned above the dump body 230. After the boom 11 is raised and the bucket 12 is positioned above the dump body 230, the working machine 1 dumps the bucket 12. With the boom 11 raised, the bucket 12 dumps so that the excavated material held in the bucket 12 is discharged into the dump body 230 of the dump truck 220. The first imaging device 31 images the bucket 12 as it dumps to discharge the excavated material into the dump body 230. The dumping action of the bucket 12 causes the excavated material held in the bucket 12 to be discharged from the bucket 12 and loaded into the dump body 230.

[0087] The measurement range Ra of the first imaging device 31 includes the imaging range of the first imaging device 31. The measurement range Ra is determined based on a first field of view α, which indicates the field of view of the optical system of the first imaging device 31. The first imaging device 31 images the bucket 12 as it performs a dumping operation to discharge the excavated material into the dump body 230. The first imaging device 31 is fixed to the housing 9 such that the bucket teeth 18 are positioned within the measurement range Ra when the boom 11 is raised and the bucket 12 is dumping.

[0088] The measurement range Rb of the second imaging device 32 includes the imaging range of the second imaging device 32. The measurement range Rb is determined based on a second field of view β, which indicates the field of view of the optical system of the second imaging device 32. The second imaging device 32 images the ground 200 in the direction of travel of at least the tire 6. In the embodiment, when the work machine 1 moves forward, the second imaging device 32 images the ground 200 in front of the front tire 6F. The second imaging device 32 is fixed to the housing 9 such that when the work machine 1 moves forward, at least the ground 200 in front of the front tire 6F is located within the measurement range Rb. In the embodiment, the second imaging device 32 is fixed to the housing 9 such that the front end of the front tire 6F, the ground 200 in front of the front tire 6F, and the ground 200 outside the front tire 6F in the vehicle width direction are each located within the measurement range Rb. The ground 200 in front of the front tire 6F includes the ground 200 between the front end of the front tire 6F and the rear end of the bucket 12 in the longitudinal direction.

[0089] In this embodiment, the second field of view β of the second imaging device 32 is wider than the first field of view α of the first imaging device 31.

[0090] As shown in Figure 10, the first imaging device 31L located to the left of the center CL images at least the left portion of the bucket 12 of the work machine 10. The first imaging device 31R located to the right of the center CL images at least the right portion of the bucket 12 of the work machine 10. The left portion of the bucket 12 is the part of the bucket 12 between the left end 12E of the bucket 12 and the center CL. The right portion of the bucket 12 is the part of the bucket 12 between the right end 12E of the bucket 12 and the center CL.

[0091] In the example shown in Figure 10, the first imaging device 31L to the left of the center CL is fixed to the left housing 9 such that when the boom 11 is raised and the bucket 12 is dumped, the bucket teeth 181, 182, 183, 184, and 185 are positioned within the measurement range Ra. The first imaging device 31R to the right of the center CL is fixed to the right housing 9 such that when the boom 11 is raised and the bucket 12 is dumped, the bucket teeth 184, 185, 186, 187, and 188 are positioned within the measurement range Ra.

[0092] As shown in Figure 10, the second imaging device 32L, located to the left of the center CL, images the ground 200 in front of the front tire 6F to the left of the center CL. The second imaging device 32R, located to the right of the center CL, images the ground 200 in front of the front tire 6F to the right of the center CL.

[0093] In the example shown in Figure 10, the second imaging device 32L to the left of the center CL is fixed to the left housing 9 such that the front end of the left front tire 6F, the ground 200 in front of the left front tire 6F, and the ground 200 to the left of the left front tire 6F are positioned within the measurement range Rb. The second imaging device 32R to the right of the center CL is fixed to the right housing 9 such that the front end of the right front tire 6F, the ground 200 in front of the right front tire 6F, and the ground 200 to the right of the right front tire 6F are positioned within the measurement range Rb.

[0094] Figure 11 shows an example of image data captured by the first imaging device 31L on the left side according to this embodiment. Figure 12 shows an example of image data captured by the second imaging device 32R on the left side according to this embodiment.

[0095] As shown in Figure 11, the first imaging device 31L images the bucket 12 as it performs a dumping operation so that the excavated material is discharged into the dump body 230, looking up from below. As a result, as shown in Figure 11, the first imaging device 31L can simultaneously image multiple bucket teeth 18. In the example shown in Figure 11, bucket teeth 181, 182, 183, 184, 185, and 186 are positioned within the measurement range Ra of the first imaging device 31.

[0096] As shown in Figure 12, the second imaging device 32L images the ground 200 in front of the front tire 6F and the ground 200 to the left of the front tire 6F from above. This allows the second imaging device 32L to image the ground 200 around the front tire 6F. For example, if there are boulders 240 on the ground 200 around the front tire 6F, the second imaging device 32L can image the boulders 240 on the ground 200.

[0097] [Monitoring System] Figure 13 is a functional block diagram showing a monitoring system 30 according to an embodiment. The work machine 1 has the monitoring system 30. The monitoring system 30 monitors the work machine 10 and the ground 200 on which the work machine 1 travels.

[0098] The monitoring system 30 includes a first imaging device 31, a second imaging device 32, an output device 24, an input device 25, and a monitoring controller 33. The first imaging device 31 includes a first imaging device 31L positioned to the left of the center CL and a first imaging device 31R positioned to the right of the center CL. The second imaging device 32 includes a second imaging device 32L positioned to the left of the center CL and a second imaging device 32R positioned to the right of the center CL.

[0099] The first imaging device 31, the second imaging device 32, the output device 24, and the input device 25 are each connected to the monitoring controller 33. Image data captured by the first imaging device 31 is input to the monitoring controller 33. Image data captured by the second imaging device 32 is input to the monitoring controller 33. Input data generated by the input device 25 is input to the monitoring controller 33.

[0100] In this embodiment, each of the first imaging device 31 and the second imaging device 32 acquires measurement data at a predetermined sampling rate (SPS: samples per second). The measurement data includes image data. The sampling rate includes the frame rate (FPS: frames per second). Each of the first imaging device 31 and the second imaging device 32 captures image data at a predetermined frame rate. The first imaging device 31 captures image data at a predetermined frame rate, thereby recording a video. The second imaging device 32 captures image data at a predetermined frame rate, thereby recording a video.

[0101] The monitoring controller 33 includes a missing object monitoring unit 34 and a boulder monitoring unit 35.

[0102] The defect monitoring unit 34 recognizes whether or not there is a defect in the work implement 10 based on the image data captured by the first imaging device 31. If a defect in the work implement 10 is detected, the defect monitoring unit 34 outputs a warning from the output device 24 indicating that there is a defect in the work implement 10. A defect in the work implement 10 includes a defect in the bucket 12. A defect in the bucket 12 includes a defect in the bucket tooth 18, which is a replacement part. A defect in the bucket tooth 18 includes the bucket tooth 18 falling off the bucket body 17.

[0103] The boulder monitoring unit 35 recognizes the presence or absence of boulders 240 on the ground 200 on which the work machine 1 is traveling, based on the imaging data captured by the second imaging device 32. If boulders 240 are detected on the ground 200, the boulder monitoring unit 35 outputs a warning from the output device 24 indicating that there are boulders 240 on the ground 200. The boulders 240 on the ground 200 include the boulders 240 on the ground 200 in the direction of travel of the front tires 6F. The boulders 240 on the ground 200 in the direction of travel of the front tires 6F include the boulders 240 on the ground 200 in front of the front tires 6F when the work machine 1 is moving forward.

[0104] <Defect Monitoring Department> The data loss monitoring unit 34 includes a first measurement data acquisition unit 34A, a data loss determination unit 34B, a counting unit 34C, and a first warning control unit 34D.

[0105] The first measurement data acquisition unit 34A acquires measurement data of the work machine 10 from the first imaging device 31 at a predetermined sampling rate. In this embodiment, the measurement data of the work machine 10 includes image data of the work machine 10 captured by the first imaging device 31. The sampling rate includes the frame rate. The first measurement data acquisition unit 34A acquires image data of the work machine 10 from the first imaging device 31 at a predetermined frame rate.

[0106] The defect determination unit 34B determines whether or not there is a defect in the work implement 10 based on the image data acquired by the first measurement data acquisition unit 34A. The defect determination unit 34B determines whether or not there is a defect in the work implement 10 based on the image data. In this embodiment, the defect determination unit 34B determines whether or not the bucket teeth 18 have fallen off from the bucket body 17 based on the image data.

[0107] Figures 14, 15, 16, 17, and 18 illustrate the defect detection method by the defect detection unit 34B according to the embodiment. Below, as an example, a method for determining the presence or absence of defects in the bucket tooth 18 based on image data captured by the first imaging device 31L located to the left of the center CL will be described.

[0108] As shown in Figure 14, the first imaging device 31L images the bucket 12 as it performs a dumping operation to discharge the excavated material into the dump body 230. The first measurement data acquisition unit 34A acquires image data of the bucket 12 from the first imaging device 31L. Multiple bucket teeth 18 (181, 182, 183, 184, 185) are arranged within the recognition range 50 of the image data. Each of the multiple bucket teeth 18 is attached to the bucket body 17 so as to protrude from the bucket body 17.

[0109] The missing data detection unit 34B recognizes each of the multiple bucket teeth 18 within the recognition range 50 of the image data acquired by the first measurement data acquisition unit 34A.

[0110] After recognizing multiple bucket teeth 18, as shown in Figure 15, the missing tooth detection unit 34B estimates the position of the root 51 and the cutting edge 52 of each of the multiple bucket teeth 18. The root 51 is the base end of the bucket tooth 18. The root 51 includes the boundary between the bucket tooth 18 and the bucket body 17. The cutting edge 52 is the tip of the bucket tooth 18. In this embodiment, the missing tooth detection unit 34B estimates the root 51 and the cutting edge 52 using artificial intelligence (AI), which analyzes the input data using an algorithm and outputs output data. The missing tooth detection unit 34B may use artificial intelligence that outputs the probability of the root 51 and the cutting edge 52 based on the input data, and estimate the center where the probability is above a predetermined threshold as the position of the root 51 and the cutting edge 52. Alternatively, the missing tooth detection unit 34B may use artificial intelligence that directly estimates the position of the root 51 and the cutting edge 52 based on the input data.

[0111] After estimating the positions of the root 51 and cutting edge 52 of at least four bucket teeth 181, 182, 183, and 184, the defect determination unit 34B calculates the distance G1 between the root 51 of bucket tooth 181 and the root 51 of bucket tooth 182, the distance G2 between the root 51 of bucket tooth 182 and the root 51 of bucket tooth 183, and the distance G3 between the root 51 of bucket tooth 183 and the root 51 of bucket tooth 184, as shown in Figure 16. After calculating the distances G1, G2, and G3, the defect determination unit 34B calculates the reference tooth distance G based on the distances G1, G2, and G3. In this embodiment, the reference tooth distance G is the median of the distances G1, G2, and G3.

[0112] After calculating the reference tooth distance G, the defect detection unit 34B searches for the cutting edge 52 corresponding to the root 51, as shown in Figure 17. The root 51 and the cutting edge 52 corresponding to the root 51 constitute the base end and tip end of a single bucket tooth 18. In this embodiment, the defect detection unit 34B sets a search range 53 based on the root 51 and searches for the cutting edge 52 within the search range 53. The search range 53 is set as a rectangle in the image data. As shown in Figure 17, the dimensions of the vertical search range 53 in the image data are set to [γ×G], and the dimensions of the horizontal search range 53 in the image data are set to [δ×G]. γ and δ are predetermined fixed values.

[0113] Furthermore, when the distance between the first imaging device 31L and the bucket 12 is short, the bucket 12 appears large in the image data captured by the first imaging device 31L. When the distance between the first imaging device 31L and the bucket 12 is long, the bucket 12 appears small in the image data captured by the first imaging device 31L. If the size of the bucket 12 differs in the image data, the defect detection unit 34B may not be able to find the cutting edge 52 corresponding to the root 51. In this embodiment, the search range 53 is set based on the reference tooth distance G. As a result, the defect detection unit 34B can find the cutting edge 52 corresponding to the root 51 regardless of the distance between the first imaging device 31L and the bucket 12.

[0114] After searching for the cutting edge 52 corresponding to each of the multiple roots 51, the defect determination unit 34B calculates the distance L1 between the root 51 and the cutting edge 52 of bucket tooth 181, the distance L2 between the root 51 and the cutting edge 52 of bucket tooth 182, the distance L3 between the root 51 and the cutting edge 52 of bucket tooth 183, and the distance L4 between the root 51 and the cutting edge 52 of bucket tooth 184, as shown in Figure 18. Distance L1 corresponds to the amount of protrusion of bucket tooth 181 from the bucket body 17. Distance L2 corresponds to the amount of protrusion of bucket tooth 182 from the bucket body 17. Distance L3 corresponds to the amount of protrusion of bucket tooth 183 from the bucket body 17. Distance L4 corresponds to the amount of protrusion of bucket tooth 184 from the bucket body 17. After calculating distances L1, L2, L3, and L4, the defect determination unit 34B calculates the reference tooth length L based on distances L1, L2, L3, and L4. In this embodiment, the reference tooth length L is the median of the distances L1, L2, L3, and L4. Alternatively, the defect detection unit 34B may calculate the distances L5, L6, L7, and L8 corresponding to the four bucket teeth 185, 186, 187, and 188 respectively, based on the image data captured by the first imaging device 31R to the right of the center CL, using the same method, and then calculate the reference tooth length L based on the distances L1, L2, L3, L4, L5, L6, L7, and L8.

[0115] After calculating the reference tooth length L, the defect determination unit 34B determines a dimensional threshold related to the amount of protrusion of the bucket teeth 18 from the bucket body 17 based on the reference tooth length L. In this embodiment, the dimensional threshold is set to [ε × L], where ε is a predetermined fixed value. Thus, in this embodiment, the defect determination unit 34B determines the dimensional threshold based on the amount of protrusion (L1, L2, L3, L4) of the multiple bucket teeth 18 (181, 182, 183, 184) from the bucket body 17.

[0116] The defect determination unit 34B determines whether or not there is a defect in the bucket teeth 18 (181, 182, 183, 184) based on a dimensional threshold. The defect determination unit 34B determines that there is a defect in the bucket teeth 18 if the amount of protrusion of the bucket teeth 18 from the bucket body 17 is less than the dimensional threshold. The defect determination unit 34B determines that there is no defect in the bucket teeth 18 if the amount of protrusion of the bucket teeth 18 from the bucket body 17 is greater than or equal to the dimensional threshold. In this embodiment, the defect determination unit 34B determines that the bucket teeth 18 have fallen off the bucket body 17 if the amount of protrusion of the bucket teeth 18 is less than the dimensional threshold, and determines that the bucket teeth 18 have not fallen off the bucket body 17 if the amount of protrusion of the bucket teeth 18 is greater than or equal to the dimensional threshold.

[0117] Furthermore, when the distance between the first imaging device 31L and the bucket 12 is short, the bucket 12 appears large in the image data captured by the first imaging device 31L. When the distance between the first imaging device 31L and the bucket 12 is long, the bucket 12 appears small in the image data captured by the first imaging device 31L. If the size of the bucket 12 differs in the image data, the defect determination unit 34B may not be able to correctly determine whether or not there is a defect in the bucket tooth 18. In this embodiment, a dimensional threshold is determined based on the reference tooth length L. As a result, the defect determination unit 34B can determine whether or not there is a defect in the bucket tooth 18 regardless of the distance between the first imaging device 31L and the bucket 12.

[0118] The counting unit 34C classifies the measurement data into multiple samples based on the determination of the missing data determination unit 34B. The counting unit 34C classifies measurement data that has been determined to have no missing data into healthy samples. The counting unit 34C classifies measurement data that has been determined to have missing data into missing data samples. The counting unit 34C classifies measurement data in which the missing data determination unit 34B could not recognize the bucket tooth 18 into undetermined samples. The counting unit 34C counts the number of healthy samples that represent measurement data that has been determined to have no missing data. The counting unit 34C counts the number of missing data samples that represent measurement data that has been determined to have missing data. The counting unit 34C counts the number of undetermined samples that represent measurement data in which the bucket tooth 18 could not be recognized. In this embodiment, the measurement data is image data, and the samples classified by the counting unit 34C are frames. In the following explanation, healthy samples will be referred to as healthy frames, missing samples as missing frames, and unclassified samples as unclassified frames.

[0119] Figure 19 shows an example of a healthy determination frame according to the embodiment. As shown in Figure 19, the distance L1 indicating the protrusion amount of bucket tooth 181, the distance L2 indicating the protrusion amount of bucket tooth 182, the distance L3 indicating the protrusion amount of bucket tooth 183, and the distance L4 indicating the protrusion amount of bucket tooth 184 are all greater than or equal to the dimensional threshold. In this case, the defect determination unit 34B determines that bucket tooth 18 is not missing. The counting unit 34C classifies the image data that has been determined to have no defects into a healthy determination frame.

[0120] Figure 20 shows an example of a missing data detection frame according to the embodiment. In the example shown in Figure 20, the distance L2, which indicates the amount of protrusion of the bucket tooth 182, is less than the dimensional threshold. In this case, the missing data detection unit 34B determines that the bucket tooth 182 is missing. The counting unit 34C classifies the image data that has been determined to have a missing data into a missing data detection frame.

[0121] The above describes a method for determining whether or not a bucket tooth 18 has fallen off based on image data captured by the first imaging device 31L located to the left of the center CL. The method for determining whether or not a bucket tooth 18 has fallen off based on image data captured by the first imaging device 31R located to the right of the center CL is similar. The defect determination unit 34B determines whether or not the four bucket teeth 185, 186, 187, and 188 have fallen off based on the image data acquired by the first measurement data acquisition unit 34A from the first imaging device 31R.

[0122] The first warning control unit 34D causes the display device 24A to display a symbol 60 indicating a bucket tooth 18. The first warning control unit 34D changes the display format of the symbol 60 based on whether or not there is a missing bucket tooth 18.

[0123] Figure 21 shows an example of a symbol 60 indicating a bucket tooth 18 displayed on the display device 24A according to the embodiment. As shown in Figure 21, if the defect detection unit 34B cannot recognize a bucket tooth 18 in the image data, the first warning control unit 34D causes the display device 24A to display a symbol 60 indicating the unrecognized bucket tooth 18 in a first display mode. If the defect detection unit 34B determines that there is no defect in the bucket tooth 18 in the image data, the first warning control unit 34D causes the display device 24A to display a symbol 60 indicating the bucket tooth 18 that was determined to be free of defects in a second display mode. If the defect detection unit 34B determines that there is a defect in the bucket tooth 18 in the image data, the first warning control unit 34D causes the display device 24A to display a symbol 60 indicating the bucket tooth 18 that was determined to be missing in a third display mode.

[0124] In this embodiment, the first warning control unit 34D displays a symbol 60 in a first color that indicates a bucket tooth 18 that the defect detection unit 34B cannot recognize. The first color is, for example, gray. The first warning control unit 34D displays a symbol 60 in a second color that indicates a bucket tooth 18 that has been determined to have no defects. The second color is, for example, green. The first warning control unit 34D displays a symbol in a third color that indicates a bucket tooth 18 that has been determined to have defects. The third color is, for example, red.

[0125] The first warning control unit 34D, if missing frames are found, causes the output device 24 to output a warning indicating the presence of missing frames. The first warning control unit 34D also changes the form of the warning output from the output device 24 based on the number of missing frames counted by the counting unit 34C.

[0126] The first warning control unit 34D outputs a warning in the first form from the output device 24 if the number of missing frames is less than the sample threshold, and outputs a warning in the second form from the output device 24 if the number of missing frames is equal to or greater than the sample threshold. The sample threshold is a predetermined value.

[0127] The output device 24 includes a display device 24A that displays a symbol 60 representing a bucket tooth 18, and a sound-generating device 24B that generates a warning sound. The first warning control unit 34D may cause the display device 24A to display display data in the first form if the number of missing frames is less than the sample threshold, and may cause the display device 24A to display display data in the second form if the number of missing frames is equal to or greater than the sample threshold. The first warning control unit 34D may generate a warning sound in the first form using the sound-generating device 24B if the number of missing frames is less than the sample threshold, and may generate a warning sound in the second form using the sound-generating device 24B if the number of missing frames is equal to or greater than the sample threshold.

[0128] Figure 22 is a diagram showing the relationship between missing frame detection frames and warning forms according to the embodiment. The presence or absence of missing frames is determined at a predetermined frame rate. In the example shown in Figure 22, the presence or absence of missing frames is determined at 0.2 [sec.] intervals. In Figure 22, "normal" frames represent healthy frames or undetermined frames. In Figure 22, "missing" frames include missing frames. In the example shown in Figure 22, the sample threshold is 2.

[0129] As shown in Figure 22, if the video captured by the first imaging device 31 consists of either or both undetermined frames and healthy determined frames, no warning sound is output from the sound-producing device 24B. If the video captured by the first imaging device 31 consists only of undetermined frames, the display device 24A displays a symbol 60 indicating an unrecognizable bucket tooth 18 in a first display mode. If the video captured by the first imaging device 31 consists only of healthy determined frames, the display device 24A displays a symbol 60 indicating a bucket tooth 18 determined to have no defects in a second display mode.

[0130] Multiple bucket teeth 18 are attached to the bucket body 17. In this embodiment, eight bucket teeth 18 are arranged at intervals in the vehicle width direction. The display device 24A displays multiple symbols 60 based on the respective positions of the multiple bucket teeth 18. Eight symbols 60 are displayed at intervals on the display device 24A, corresponding to the positions of the multiple bucket teeth 181, 182, 183, 184, 185, 186, 187, and 188. If the video consists only of unjudged frames, all eight symbols 60 are displayed in the first display mode. If the video consists only of healthy judgment frames, all eight symbols 60 are displayed in the second display mode.

[0131] As shown in Figure 22, if there is one frame detected as missing in the video, the first warning control unit 34D outputs a first warning sound from the sound-generating device 24B. If there are two frames detected as missing in the video captured by the first imaging device 31, the first warning control unit 34D outputs a second warning sound from the sound-generating device 24B.

[0132] In this embodiment, the first warning control unit 34D outputs a warning in the first form from the output device 24 if it determines that each of the image data acquired consecutively by the first measurement data acquisition unit 34A for a number of less than the sample threshold is a missing data frame, and outputs a warning in the second form from the output device 24 if it determines that each of the image data acquired consecutively by the first measurement data acquisition unit 34A for a number of more than or equal to the sample threshold is a missing data frame.

[0133] For example, if the sample threshold is 2, and each of the less than 2 image data frames acquired consecutively by the first measurement data acquisition unit 34A is determined to be a missing data frame, the first warning control unit 34D will output a first warning sound from the sound-generating device 24B. In other words, if the image data acquired before and after the missing data frame is a healthy data frame or an undetermined data frame, the first warning control unit 34D will output a first warning sound from the sound-generating device 24B. If each of the 2 or more image data frames acquired consecutively by the first measurement data acquisition unit 34A is determined to be a missing data frame, the first warning control unit 34D will output a second warning sound from the sound-generating device 24B. In other words, if there are at least two consecutive missing data frames, the first warning control unit 34D will output a second warning sound from the sound-generating device 24B.

[0134] Furthermore, even if there are not two consecutive frames where a missing frame is detected, if there are at least two frames where a missing frame is detected in the video captured per unit time by the first imaging device 31, the first warning control unit 34D may output a second warning sound from the sound generator 24B.

[0135] The sample threshold is arbitrary. For example, it can be any number greater than or equal to 3.

[0136] The form of the warning includes the volume of the warning sound. Outputting a warning in the first form includes generating a warning sound at a first volume from the sound-generating device 24B. Outputting a warning in the second form includes generating a warning sound at a second volume that is louder than the first volume from the sound-generating device 24B. In the example shown in Figure 22, the volume of the second warning sound is louder than the volume of the first warning sound.

[0137] Furthermore, if there are missing frames in the video captured by the first imaging device 31, the display device 24A displays a symbol 60 indicating the bucket tooth 18 that was determined to be missing in a third display mode. The first warning control unit 34D displays the symbol 60 indicating the bucket tooth 18 that was determined to be missing and the symbol 60 indicating the bucket tooth 18 that was determined not to be missing in different display modes. In the example shown in Figure 22, if bucket tooth 184 is determined to be missing, and bucket teeth 181, 182, 183, 185, 186, 187, and 188 are determined not to be missing, the first warning control unit 34D displays the symbol indicating the bucket tooth 184 that was determined to be missing in a third display mode, and displays the symbols indicating the bucket teeth 181, 182, 183, 185, 186, 187, and 188 that were determined not to be missing in a second display mode.

[0138] <Rockfall Monitoring Department> The boulder monitoring unit 35 includes a second measurement data acquisition unit 35A, a judgment criterion setting unit 35B, a warning criterion setting unit 35C, a standard value storage unit 35D, an input data acquisition unit 35E, an operating status acquisition unit 35F, a boulder judgment unit 35G, and a second warning control unit 35H.

[0139] The second measurement data acquisition unit 35A acquires measurement data of the ground 200 from the second imaging device 32 at a predetermined sampling rate. As described above, the measurement data of the ground 200 includes image data of the ground 200 captured by the second imaging device 32. The sampling rate includes the frame rate. The second measurement data acquisition unit 35A acquires image data of the ground 200 from the second imaging device 32 at a predetermined frame rate. The second measurement data acquisition unit 35A acquires image data of the ground 200 in front of the front tire 6F.

[0140] The judgment criterion setting unit 35B sets the judgment criteria for determining the presence or absence of boulders 240 on the ground 200 in the image data.

[0141] Figure 23 is a diagram illustrating the judgment criteria according to an embodiment. In this embodiment, the judgment criteria include the dimensions of the boulder 240 in the image data. The dimensions of the boulder 240 include the number of pixels of the boulder 240 in the image data. A threshold value is set in the image data for the dimensions of the boulder 240. The threshold value for the dimensions of the boulder 240 includes a threshold value Px for the horizontal dimensions of the boulder 240 and a threshold value Py for the vertical dimensions of the boulder 240. The standard value storage unit 35D stores standard values ​​(initial values) related to the judgment criteria. The standard values ​​are recommended values ​​for the threshold values ​​related to the dimensions of the boulder 240. The judgment criterion setting unit 35B sets the judgment criteria based on the standard values. In this embodiment, the judgment criterion setting unit 35B sets the threshold value for the dimensions of the boulder 240 based on the standard values. The threshold value is the standard value.

[0142] The warning criteria setting unit 35C sets the warning criteria for when a warning is output from the output device 24.

[0143] Figure 24 is a diagram illustrating a warning criterion according to an embodiment. In this embodiment, the warning criterion includes a warning area 36 that is set in at least a portion of the ground 200.

[0144] Setting the warning criteria includes setting the dimensions of the warning area 36. The dimensions of the warning area 36 include the number of pixels of the warning area in the image data. As shown in Figure 24, the horizontal dimension Qx of the warning area 36 and the vertical dimension Qy of the warning area 36 are set in the image data.

[0145] Furthermore, setting the warning area 36 includes setting the location of the warning area 36. Setting the location of the warning area 36 includes setting the location of the warning area 36 in the image data. The warning standard setting unit 35C sets the warning area 36 in a part of the ground 200 in front of the front tire 6F. The standard value storage unit 35D stores standard values ​​(initial values) related to the warning standard. The standard values ​​are recommended values ​​related to the dimensions and location of the warning area 36. The warning standard setting unit 35C sets the warning standard based on the standard values. In this embodiment, the warning standard setting unit 35C sets the dimensions and location of the warning area 36 in the image data based on the standard values.

[0146] The input data acquisition unit 35E acquires input data from the input device 25. The driver can change the setting of the judgment criteria or the warning criteria by operating the input device 25. The judgment criteria setting unit 35B sets the judgment criteria based on the input data. The warning criteria setting unit 35C sets the warning criteria based on the input data.

[0147] Figure 25 shows an example of the setting screens for the judgment criteria and warning criteria according to the embodiment. As shown in Figure 25, the judgment criteria setting unit 35B displays the judgment criteria setting screen on the display device 24A. The warning criteria setting unit 35C displays the warning criteria setting screen on the display device 24A. The driver can change the judgment criteria setting and change the warning criteria setting by operating the input device 25 while viewing the setting screen.

[0148] The operating status acquisition unit 35F acquires status data indicating the status of the drive system 40. The drive controller 48 inputs the status data indicating the status of the drive system 40 to the monitoring controller 33. The operating status acquisition unit 35F determines the status of the drive system 40 based on the status data from the drive controller 48.

[0149] In this embodiment, the state of the drive system 40 includes the operating state of the transmission 44. The transmission 44 switches the direction of travel of the work machine 1 between the forward direction and the reverse direction. The transmission 44 includes a forward gear 44F that operates to move the work machine 1 forward and a reverse gear 44R that operates to move the work machine 1 backward.

[0150] The boulder detection unit 35G determines the presence or absence of boulders 240 on the ground 200 based on the image data acquired by the second measurement data acquisition unit 35A. The boulder detection unit 35G determines the presence or absence of boulders 240 on the ground 200 in the image data. In this embodiment, the boulder detection unit 35G determines the presence or absence of boulders 240 on the ground 200 in front of the front tire 6F in the image data.

[0151] The boulder detection unit 35G may determine the presence or absence of boulders 240 by, for example, using artificial intelligence (AI) which analyzes input data using an algorithm and outputs output data. The boulder detection unit 35G may also determine the presence or absence of boulders 240 by, for example, using a pattern matching method.

[0152] In this embodiment, the boulder detection unit 35G determines the presence or absence of boulders 240 on the ground 200 in the image data based on the determination criteria. The boulder detection unit 35G determines that boulders 240 are present if the dimensions of the boulders 240 exceed a threshold (Px, Py), and determines that boulders 240 are not present if the dimensions of the boulders 240 are less than or equal to the threshold (Px, Py). The boulder detection unit 35G may also determine that boulders 240 are present if the dimensions of the boulders 240 exceed either the threshold Px or the threshold Py. The boulder detection unit 35G may also determine that boulders 240 are present if the dimensions of the boulders 240 exceed both the threshold Px and the threshold Py.

[0153] In this embodiment, the rock detection unit 35G determines the presence or absence of rocks 240 in both the state in which the boom 11 is being raised and the state in which it is being lowered. The rock detection unit 35G constantly determines the presence or absence of rocks 240 during the cycle operation described with reference to Figure 8.

[0154] The second warning control unit 35H outputs a warning from the output device 24 based on the presence or absence of a boulder 240 on the ground 200 in front of the front tire 6F. The second warning control unit 35H outputs a warning from the output device 24 based on the boulder detection unit 35G's determination of the presence or absence of a boulder 240 and the relationship between the boulder 240 and the warning criteria. The warning criteria include a warning area 36 set in at least a part of the ground 200. The warning area 36 is set in a part of the ground 200 in front of the front tire 6F. If the second warning control unit 35H determines that a boulder 240 is present, it outputs a warning from the output device 24 based on the positional relationship between the boulder 240 and the warning area 36. If the second warning control unit 35H determines that a boulder 240 is in the warning area 36, ​​it outputs a warning from the output device 24 indicating the presence of a boulder 240 in the warning area 36.

[0155] In this embodiment, the second warning control unit 35H controls the output device 24 based on state data indicating the state of the drive system 40 acquired by the operating state acquisition unit 35F. The state of the drive system 40 includes the operating state of the transmission 44. The transmission 44 switches the direction of travel of the work machine 1 between the forward direction and the reverse direction. The second warning control unit 35H outputs a warning if it determines that there is a boulder 240 on the ground 200 in the direction of travel of the work machine 1 determined by the transmission 44. For example, if the transmission 44 is in neutral or the reverse gear 44R is engaged, even if it is determined that there is a boulder 240 on the ground 200 in front of the front tires 6F, the second warning control unit 35H does not output a warning indicating the presence of the boulder 240. When it is determined that there is a boulder 240 on the ground 200 in front of the front tire 6F, for example, when the transmission 44 changes from a neutral state to a state where the forward gear 44F is engaged, the output of a warning indicating the presence of the boulder 240 is initiated. In this embodiment, when it is determined that there is a boulder 240 on the ground 200 in front of the front tire 6F and the forward gear 44F is engaged, the second warning control unit 35H outputs a warning indicating the presence of the boulder 240, even if the work machine 1 is not moving.

[0156] [Method for monitoring missing data] Figure 26 is a flowchart showing a defect monitoring method for the work machine 10 according to an embodiment.

[0157] The first measurement data acquisition unit 34A acquires image data of the work machine 10 from the first imaging device 31 (step SA1).

[0158] As explained with reference to Figure 15, the defect detection unit 34B recognizes the bucket teeth 18 in the image data acquired by the first measurement data acquisition unit 34A, and then estimates the base 51 and cutting edge 52 of each of the multiple bucket teeth 18 (step SA2).

[0159] In step SA2, if the bucket tooth 18 is recognized and the root 51 and cutting edge 52 are estimated (step SA2: Yes), the missing tooth detection unit 34B calculates the distances G1, G2, and G3 between a pair of adjacent bucket teeth 18, as explained with reference to Figure 16. The missing tooth detection unit 34B determines whether the distances G1, G2, and G3 are equally spaced (step SA3).

[0160] In step SA3, if it is determined that distances G1, G2, and G3 are equally spaced (step SA3: Yes), the missing part detection unit 34B sets the search range 53 and searches for the cutting edge 52 corresponding to the root 51, as explained with reference to Figure 17. The missing part detection unit 34B determines whether or not it was able to find the cutting edge 52 (step SA4).

[0161] In step SA4, if it is determined that the cutting edge 52 has been found (step SA4: Yes), the defect determination unit 34B determines dimensional thresholds based on distances L1, L2, L3, and L4, as explained with reference to Figure 18. Based on the dimensional thresholds, the defect determination unit 34B determines whether or not there is a defect in the bucket tooth 18 (step SA5).

[0162] In step SA5, if it is determined that there is a missing bucket tooth 18 in the image data (step SA5: Yes), the counting unit 34C classifies the image data that was determined to have a missing portion into a missing portion determination frame (step SA6).

[0163] In step SA5, if it is determined that there are no missing bucket teeth 18 in the image data (step SA5: No), the count unit 34C classifies the image data determined to have no missing data into a healthy frame (step SA7).

[0164] In step SA2, if the bucket tooth 18 cannot be recognized in the image data (step SA2: No), in step SA3, if it is determined that the distances G1, G2, and G3 in the image data are not equally spaced (step SA3: No), and in step SA4, if it is determined that the cutting edge 52 cannot be searched in the image data (step SA5: No), the count unit 34C classifies those image data into undetermined frames (step SA8).

[0165] The counting unit 34C processes the classified image data in a time series (step SA9).

[0166] The counting unit 34C determines whether or not there are missing frames (step SA10).

[0167] If it is determined in step SA10 that there are no missing frames (step SA10: No), no warning is output from output device 24 (step SA11).

[0168] In step SA10, if it is determined that there are missing frames (step SA10: Yes), the counting unit 34C determines whether the number of missing frames is greater than or equal to the sample threshold. In this embodiment, the counting unit 34C determines whether there are at least two consecutive missing frames (step SA12).

[0169] In step SA12, if it is determined that there are not two consecutive missing frames (step SA12: No), the first warning control unit 34D outputs a warning in the first form. As explained with reference to Figure 22, in this embodiment, the first warning control unit 34D outputs a first warning sound of first volume from the sound generator 24B (step SA13).

[0170] In step SA12, if it is determined that there are at least two consecutive missing frames (step SA12: Yes), the first warning control unit 34D outputs a warning in the second form. As described with reference to Figure 22, in this embodiment, the first warning control unit 34D outputs a second warning sound from the sound generator 24B at a second volume that is louder than the first volume (step SA14).

[0171] The data loss monitoring unit 34 determines whether or not to terminate the data loss monitoring process (step SA15).

[0172] If it is determined in step SA15 to continue the missing data monitoring process (step SA15: No), the process returns to step SA1.

[0173] If it is determined in step SA15 that the missing data monitoring process has finished (step SA15: Yes), the missing data monitoring process will terminate.

[0174] The processing from step SA1 to step SA15 is performed at a predetermined frame rate.

[0175] [Method for monitoring boulders] Figure 27 is a flowchart showing a method for monitoring boulders on the ground 200 according to an embodiment.

[0176] The judgment criterion setting unit 35B sets the judgment criteria for determining the presence or absence of boulders 240 on the ground 200. Based on the standard values ​​stored in the standard value storage unit 35D, the judgment criterion setting unit 35B sets threshold Px and threshold Py related to the dimensions of the boulders 240. When the driver operates the input device 25, the judgment criterion setting unit 35B sets threshold Px and threshold Py based on the input data (step SB1).

[0177] The warning criteria setting unit 35C sets the warning criteria for when a warning is output from the output device 24. The warning criteria setting unit 35C sets the warning area 36 based on the standard values ​​stored in the standard value storage unit 35D. Setting the warning area 36 includes setting dimensions Qx and Qy, and setting the position of the warning area 36. When the driver operates the input device 25, the warning criteria setting unit 35C sets the warning area 36 based on the input data (step SB2).

[0178] The second measurement data acquisition unit 35A acquires image data of the ground 200 from the second imaging device 32. The second measurement data acquisition unit 35A acquires image data of the ground 200 in front of the front tire 6F (step SB3).

[0179] The driving status acquisition unit 35F acquires status data of the transmission 44 (step SB4).

[0180] The rock detection unit 35G determines whether or not there are rocks 240 on the ground 200 in the direction of travel of the work machine 1 as determined by the transmission 44. When the forward gear 44F is engaged, the rock detection unit 35G determines whether or not there are rocks 240 on the ground 200 in front of the front tires 6F. The rock detection unit 35G determines whether or not there are rocks 240 on the ground 200 in front of the front tires 6F while the transmission 44 is operating to move the work machine 1 forward (step SB5).

[0181] In step SB5, if it is determined that there is a boulder 240 on the ground 200 in front of the front tire 6F (step SB5: Yes), the second warning control unit 35H outputs a warning from the output device 24. The second warning control unit 35H outputs a warning from the output device 24 if it is determined that there is a boulder 240 on the ground 200 in front of the front tire 6F while the transmission 44 is operating to move the work machine 1 forward (step SB6).

[0182] In step SB5, if it is determined that there are no boulders 240 on the ground 200 in front of the front tire 6F (step SB5: Yes), no warning is output from the output device 24.

[0183] The rock-rolling monitoring unit 35 determines whether or not to terminate the rock-rolling monitoring process (step SB7).

[0184] If it is determined in step SB7 to continue the boulder monitoring process (step SB7: No), the process returns to step SB3.

[0185] If it is determined in step SB7 that the rock-rolling monitoring process has finished (step SB7: Yes), the rock-rolling monitoring process ends.

[0186] The processing from step SB3 to step SB7 is performed at a predetermined frame rate.

[0187] [Display device] Figure 28 shows an example of a display device 24A according to the embodiment. As shown in Figure 28, the second warning control unit 35H causes the display device 24A to display a target image 37 representing the ground 200, which is generated based on image data acquired by the second measurement data acquisition unit 35A. The target image 37 includes a target image 37L showing image data captured by the second imaging device 32L on the left, and a target image 37R showing image data captured by the second imaging device 32R on the right. The target image 37L and the target image 37R are displayed side by side on the display screen of the display device 24A. The target image 37L and the target image 37R are displayed simultaneously on the display screen of the display device 24A. The target image 37L is displayed in the left area of ​​the display screen of the display device 24A. The target image 37R is displayed in the right area of ​​the display screen of the display device 24A.

[0188] The second warning control unit 35H causes the display device 24A to display an area image 360 ​​indicating the warning area 36. The second warning control unit 35H superimposes the area image 360 ​​onto the target image 37 and displays it on the display device 24A. The area image 360 ​​indicating the warning area 36, ​​which is set to include the ground 200 in front of the left front tire 6F, is displayed on the display device 24A superimposed on the target image 37L. The area image 360 ​​indicating the warning area 36, ​​which is set to include the ground 200 in front of the right front tire 6F, is displayed on the display device 24A superimposed on the target image 37R.

[0189] The second warning control unit 35H causes the display device 24A to display a symbol 70 indicating the front tire 6F. The symbol 70 includes a symbol 70L indicating the left front tire 6F and a symbol 70R indicating the right front tire 6F.

[0190] The second warning control unit 35H causes the output device 24 to output a warning indicating that there is a boulder 240 on the ground 200 in front of the front tire 6F. Outputting a warning to the output device 24 includes displaying a warning image on the display device 24A indicating that there is a boulder 240 on the ground 200 in front of the front tire 6F. Outputting a warning to the output device 24 also includes generating a warning sound from the sound-generating device 24B indicating that there is a boulder 240 on the ground 200 in front of the front tire 6F.

[0191] In this embodiment, the warning image includes a frame image 80 displayed so as to surround the boulder 240 in the target image 37, a frame image 81 displayed on the periphery of the target image 37, and a background image 82 of the symbol 70.

[0192] The second warning control unit 35H causes the frame image 80 to be displayed on the display device 24A so that the boulder 240 is highlighted.

[0193] In the example shown in Figure 28, the boulder 240 in the target image 37L is located outside the warning area 36. When the boulder detection unit 35G determines that the boulder 240 is located outside the warning area 36, ​​the second warning control unit 35H displays the frame image 80L of the first form on the display device 24A.

[0194] In the example shown in Figure 28, at least a portion of the boulder 240 in the target image 37R is located inside the warning area 36. When the boulder determination unit 35G determines that the boulder 240 is located inside the warning area 36, ​​the second warning control unit 35H causes the display device 24A to display a frame image 80R of a second form, which is different from the first form.

[0195] The second warning control unit 35H causes the display device 24A to display frame image 80L and frame image 80R, respectively, so that the boulders 240 located inside the warning area 36 are emphasized more than the boulders 240 located outside the warning area 36. In the example shown in Figure 28, frame image 80R is displayed as a solid line and frame image 80L is displayed as a dotted line. Alternatively, frame image 80R may be displayed in a first color and frame image 80L in a second color. Red is an example of the first color. Green is an example of the second color.

[0196] The second warning control unit 35H causes the display device 24A to display a frame image 81 on the display device 24A such that the target image 37 in which at least a portion of the boulder 240 is located inside the warning area 36 is highlighted.

[0197] In the example shown in Figure 28, at least a portion of the boulders 240 displayed in the target image 37R are located inside the warning area 36. The boulders 240 displayed in the target image 37L are located outside the warning area 36. The second warning control unit 35H causes the display device 24A to display the frame image 81 so that the target image 37R is emphasized more than the target image 37L.

[0198] The second warning control unit 35H causes the display device 24A to display a background image 82 so that the front tire 6F, which is likely to pass over the boulder 240, is highlighted.

[0199] In the example shown in Figure 28, the right front tire 6F is likely to pass over the boulder 240. That is, as the work machine 1 moves forward, the right front tire 6F is likely to run over the boulder 240. The second warning control unit 35H displays the background image 82 of symbol 70R on the display device 24A so that the right front tire 6F, which is likely to pass over the boulder 240, is highlighted.

[0200] In the example shown in Figure 28, if there is a high probability that the left front tire 6F will pass over the boulder 240, the second warning control unit 35H can display the background image 82 of the symbol 70L on the display device 24A so that the left front tire 6F, which is likely to pass over the boulder 240, is highlighted.

[0201] The first warning control unit 34D causes the display device 24A to display a symbol 60 representing a bucket tooth 18. The first warning control unit 34D displays multiple symbols 60 based on the respective positions of the multiple bucket teeth 18. The symbols 60 include a symbol 61 representing a bucket tooth 181, a symbol 62 representing a bucket tooth 182, a symbol 63 representing a bucket tooth 183, a symbol 64 representing a bucket tooth 184, a symbol 65 representing a bucket tooth 185, a symbol 66 representing a bucket tooth 186, a symbol 67 representing a bucket tooth 187, and a symbol 68 representing a bucket tooth 188.

[0202] Multiple symbols 60 are displayed in the upper area of ​​the display screen of the display device 24A. Of the multiple symbols 60, symbol 61 is displayed on the far left, symbol 62 is positioned to the left of symbol 61, symbol 63 is positioned to the left of symbol 62, symbol 64 is positioned to the left of symbol 63, symbol 65 is positioned to the left of symbol 64, symbol 66 is positioned to the left of symbol 65, symbol 67 is positioned to the left of symbol 66, and symbol 68 is positioned on the far right.

[0203] Figure 28 shows an example where the missing element detection unit 34B fails to recognize eight bucket teeth 18, and eight symbols 60 are displayed in the first display mode.

[0204] Figure 29 shows an example of a display device 24A according to the embodiment. Note that Figure 29 is an extracted view of the upper region of the display device 24A.

[0205] Figure 29 shows an example of the symbol 60 when the defect detection unit 34B determines whether each of the eight bucket teeth 18 is missing. As shown in Figure 29, the first warning control unit 34D changes the display form of the symbol 60 based on whether or not the bucket teeth 18 are missing. The first warning control unit 34D displays the symbol 60 indicating the bucket tooth 18 that has been determined to be missing and the symbol 60 indicating the bucket tooth 18 that has been determined to be missing in different display forms. As shown in Figure 29, if it is determined that bucket tooth 184 is missing and bucket teeth 181, 182, 183, 185, 186, 187, and 188 are not missing, the first warning control unit 34D displays symbol 64, which is the bucket tooth 184 that was determined to be missing, in a third display mode, and displays symbols 61, 62, 63, 65, 66, 67, and 68, which are the bucket teeth 181, 182, 183, 185, 186, 187, and 188 that were determined to be not missing, in a second display mode. The first warning control unit 34D displays symbol 64 in a third color on the display device 24A, and displays symbols 61, 62, 63, 65, 66, 67, and 68 in a second color on the display device 24A. The third color is, for example, red. The second color is, for example, green.

[0206] [Computer System] Figure 30 is a block diagram showing a computer system 1000 according to an embodiment. Each of the monitoring controller 33 and drive controller 48 described above includes the computer system 1000. The computer system 1000 has a processor 1001 such as a CPU (Central Processing Unit), a main memory 1002 including non-volatile memory such as ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory), a storage 1003, and an interface 1004 including input / output circuits. The respective functions of the monitoring controller 33 and drive controller 48 described above are stored in the storage 1003 as computer programs. The processor 1001 reads the computer program from the storage 1003, loads it into the main memory 1002, and executes the above-mentioned processing according to the program. The computer program may be distributed to the computer system 1000 via a network.

[0207] The computer program or computer system 1000 can perform the following actions in accordance with the above-described embodiment: to image the work equipment 10 of the work equipment 1 with a first imaging device 31 having a first field of view α provided on the work equipment 1; to image the ground 200 on which the work equipment 1 travels with a second imaging device 32 having a second field of view β wider than the first field of view α provided on the work equipment 1; to determine whether or not there are any defects in the work equipment 10 based on the image data captured by the first imaging device 31; and to determine whether or not there are any boulders 240 on the ground 200 based on the image data captured by the second imaging device 32.

[0208] [effect] As described above, according to the embodiment, the monitoring system 30 includes a first imaging device 31 with a first field of view α that is provided on the work machine 1 and images the work machine 10 of the work machine 1, a second imaging device 32 with a second field of view β that is wider than the first field of view α and images the ground 200 on which the work machine 1 travels, a defect determination unit 34B that determines whether or not there is a defect in the work machine 10 based on the image data captured by the first imaging device 31, and a boulder determination unit 35G that determines whether or not there is a boulder 240 on the ground 200 based on the image data captured by the second imaging device 32. The monitoring system 30 can monitor the status of the work machine 1 during operation using the first imaging device 31 and the second imaging device 32. The monitoring system 30 can monitor for defects in the work machine 10 during operation using the first imaging device 31. The monitoring system 30 can recognize defects in the work machine 10 early based on the image data of the work machine 10 captured by the first imaging device 31. The monitoring system 30, using the second imaging device 32, can monitor boulders 240 on the ground 200 while the work machine 1 is operating. Based on the image data of the ground 200 captured by the second imaging device 32, the monitoring system 30 can take measures to prevent the tires 6 from running over the boulders 240. This reduces damage to the tires 6 and prevents disruptions to the operation of the work machine 1.

[0209] The first field of view α of the first imaging device 31 is narrower than the second field of view β. The first imaging device 31 can image the work machine 10 at a magnified view. As a result, the defect detection unit 34B can properly determine whether or not there is a defect in the work machine 10 based on the image data of the work machine 10. The second field of view β of the second imaging device 32 is wider than the first field of view α. The second imaging device 32 can image a wide area of ​​the ground 200. As a result, the boulder detection unit 35H can properly determine whether or not there is a boulder 240 on the ground 200 based on the image data of the ground 200.

[0210] The second imaging device 32 is positioned above the first imaging device 31. The first imaging device 31 can image the bucket 12 of the work machine 10 from below, looking upwards. This allows the first imaging device 31 to image multiple bucket teeth 18. The second imaging device 32 can image the ground 200 in front of the front tire 6F from above, looking downwards. This allows the second imaging device 32 to image the ground 200 around the front tire 6F.

[0211] The first imaging device 31 and the second imaging device 32 are each housed in the housing 9. This allows the first imaging device 31 to properly capture image data of the work machine 10, and the second imaging device 32 to properly capture image data of the ground 200.

[0212] In the vehicle width direction of the work machine 1, the first imaging device 31 and the second imaging device 32 are each positioned outside the headlight 28. This ensures that the illumination function of the headlight 28, the imaging function of the first imaging device 31, and the imaging function of the second imaging device 32 are all properly performed.

[0213] The first imaging devices 31 are positioned to the left and right of the center CL of the work machine 1 in the vehicle width direction of the work machine 1. The left first imaging device 31L captures the left portion of the work machine 10, and the right first imaging device 31R captures the right portion of the work machine 10. This suppresses the occurrence of blind spots and ensures that image data of the work machine 10 is properly acquired.

[0214] The first imaging device 31 is installed on the work machine 1 so as to be able to image at least the bucket 12 that is performing the dumping operation. As a result, the first imaging device 31 can image the bucket teeth 18.

[0215] The second imaging devices 32 are positioned to the left and right of the center CL of the work machine 1 in the vehicle width direction of the work machine 1. The left second imaging device 32L captures images of the ground 200 in the direction of travel of the left front tire 6F. The right second imaging device 32R captures images of the ground 200 in the direction of travel of the right front tire 6F. This suppresses the occurrence of blind spots and ensures that image data of the ground 200 is properly acquired.

[0216] [Another embodiment] In the embodiment described above, the first imaging device 31 and the second imaging device 32 are each positioned in the housing 9 that holds the headlight 28. At least one of the first imaging device 31 and the second imaging device 32 may be positioned in a different part of the work machine 1 from the housing 9.

[0217] In the above-described embodiment, the second imaging device 32 is positioned above the first imaging device 31. In the vertical direction, at least a portion of the second imaging device 32 may be positioned at the same location as the first imaging device 31. The second imaging device 32 may be positioned below the first imaging device 31.

[0218] In the above-described embodiment, the defect monitoring unit 34 determines whether or not there is a defect in the bucket 12 based on the image data of the bucket 12 performing a dumping operation in the cycle operation described with reference to Figure 8. The first imaging device 31 may also image the bucket 12 performing a dumping operation in an operation other than the cycle operation. The defect monitoring unit 34 may also determine whether or not there is a defect in the bucket 12 based on the image data of the bucket 12 performing a dumping operation in an operation other than the cycle operation. Furthermore, the image data of the bucket 12 captured by the first imaging device 31 does not have to be image data of the bucket 12 performing a dumping operation; it is sufficient if the defect monitoring unit 34 can recognize a defect in the bucket 12.

[0219] In the above embodiment, the loss of the bucket teeth 18 was defined as the bucket teeth 18 falling off the bucket body 17. The loss of the bucket teeth 18 may also be due to wear of the bucket teeth 18 or breakage of the bucket teeth 18.

[0220] In the embodiment described above, the defect monitoring unit 34 monitors for defects in the bucket teeth 18 as replacement parts. The defect monitoring unit 34 may also monitor for defects in the tooth-tooth protectors 19 as replacement parts. Defects in the tooth-tooth protectors 19 include detachment of the tooth-tooth protectors 19 from the bucket body 17, wear of the tooth-tooth protectors 19, and breakage of the tooth-tooth protectors 19.

[0221] In the above-described embodiment, the setting of the warning criteria includes setting the dimensions of the warning area 36 and setting the position of the warning area 36. The setting of the warning criteria may be either setting the dimensions of the warning area 36 or setting the position of the warning area 36. For example, the position of the warning area 36 may be a predetermined fixed value, and only the dimensions of the warning area 36 may be set. Alternatively, the dimensions of the warning area 36 may be a predetermined fixed value, and only the position of the warning area 36 may be set.

[0222] In the above embodiment, the boulder monitoring unit 35 determines the presence or absence of boulders 240 on the ground 200 based on the determination criteria, and outputs a warning from the output device 24 based on the relationship between the boulders 240 and the warning criteria. The boulder monitoring unit 35 may determine the presence of boulders 240 if both the determination criteria and the warning criteria are met. In this case, the boulder monitoring unit 35 may use the warning criteria as the determination criteria. Alternatively, the boulder monitoring unit 35 may determine the presence or absence of boulders 240 on the ground 200 regardless of the determination criteria, and output a warning if both the determination criteria and the warning criteria are met. In this case, the boulder monitoring unit 35 may use the determination criteria as the warning criteria.

[0223] In the above-described embodiment, the second warning control unit 35H displays an area image 360 ​​indicating the warning area 36 on the display device 24A. The second warning control unit 35H does not have to display the area image 360 ​​indicating the warning area 36 on the display device 24A. The second warning control unit 35H may display the area image 360 ​​indicating the warning area 36 on the display device 24A when it is determined that there is a boulder 240. The second warning control unit 35H may display the area image 360 ​​indicating the warning area 36 on the display device 24A when it is determined that there is a boulder 240 in the warning area 36.

[0224] In the above embodiment, the second warning control unit 35H is configured to output a warning from the output device 24 if it determines that there is a boulder 240 on the ground 200 in front of the front tire 6F while the transmission 44 is operating to move the work machine 1 forward. Furthermore, even if the work machine 1 has not started moving forward due to the forward gear 44F being engaged while it is determined that there is a boulder 240 on the ground 200 in front of the front tire 6F, the output device 24 will still output a warning. The second warning control unit 35H may also output a warning from the output device 24 if it is determined that there is a boulder 240 on the ground 200 in the direction of travel of the front tire 6F and the work machine 1 is moving in the direction of travel, regardless of the state of the transmission 44. For example, if the work machine 1 is provided with a rotation sensor that detects the rotation direction of the tire 6, the operating state acquisition unit 35F can determine whether or not the work machine 1 is moving in the direction of travel based on the detection data of the rotation sensor. The second warning control unit 35H may output a warning from the output device 24 if the operating state acquisition unit 35F determines that the work machine 1 is moving in the direction of travel, and the boulder detection unit 35G determines that there is a boulder 240 on the ground 200 in the direction of travel. For example, if the work machine 1 is on a downhill slope, even if the transmission 44 is in neutral or the reverse gear 44R is engaged, the work machine 1 may move forward due to the action of gravity, for example. The second warning control unit 35H may output a warning from the output device 24 if the rotation sensor detects that the work machine 1 is moving forward while it has been determined that there is a boulder 240 on the ground 200 in front of the front tires 6F.

[0225] In the above embodiment, if the work machine 1 is in a state where it can move forward or is moving forward, and it is determined that there is a boulder 240 on the ground 200 in front of the front tire 6F, the second warning control unit 35H will output a warning from the output device 24. If the work machine 1 is in a state where it can move backward or is moving backward, and it is determined that there is a boulder 240 on the ground 200 behind the rear tire 6R, the second warning control unit 35H may output a warning from the output device 24.

[0226] Figure 31 shows the operation of the work machine 1 according to another embodiment. As shown in Figure 31, the work machine 1 has a second imaging device 32 that images the ground 200 behind the rear tire 6R. The second measurement data acquisition unit 35A acquires image data of the ground 200 behind the rear tire 6R from the second imaging device 32. The second warning control unit 35H may output a warning from the output device 24 if it determines that there is a boulder 240 on the ground 200 behind the rear tire 6R while the transmission 44 is operating to move the work machine 1 in reverse. Alternatively, the second warning control unit 35H may output a warning from the output device 24 if it determines that the work machine 1 is moving in reverse, regardless of the state of the transmission 44, and it determines that there is a boulder 240 on the ground 200 behind the rear tire 6R.

[0227] In the embodiment described above, the output device 24 is located in the operator's cab of the work machine 1. The output device 24 may also be located outside the work machine 1.

[0228] Figure 32 shows a monitoring system 300 according to another embodiment. In the example shown in Figure 32, the work machine 1 is remotely controlled by an operating device 20F located remotely from the work machine 1.

[0229] Similar to the embodiments described above, the first imaging device 31 and the second imaging device 32 are each provided on the work machine 1. In addition, a remotely controlled imaging device (not shown) that images the area in front of the work machine 1 is provided on the work machine 1. The remotely controlled imaging device captures image data of the view in front of the work machine 1, similar to the view seen by a driver seated in the driver's seat of the driver's cab through the windshield, as shown in Figure 4.

[0230] The output device 24F, input device 25F, monitoring controller 33F, and some or all of the operation device 20F are located in a remote control facility at a remote location from the work machine 1. Each of the output device 24F, input device 25F, monitoring controller 33F, and operation device 20F is provided separately from the work machine 1. In addition, the remote control facility is equipped with a remote control display device that displays image data captured by the remote control imaging device.

[0231] The work machine 1 and the monitoring controller 33F communicate wirelessly via a wireless communication system. The work machine 1 is equipped with a wireless communication device 301 of the wireless communication system, and the monitoring controller 33F is connected to a wireless communication device 302 of the wireless communication system. Note that the wireless communication device 301 of the work machine 1 and the wireless communication device 302 of the monitoring controller 33F may communicate via other devices.

[0232] Image data captured by the remotely controlled imaging device is transmitted to the remotely controlled display device via a wireless communication system. The operator of the remotely controlled facility can operate the operation control device 20F while viewing the image data displayed on the remotely controlled display device. The operation signals generated by the operation of the operation control device 20F are transmitted to the work machine 1 via the wireless communication system. This allows the work machine 1 to be remotely controlled.

[0233] The monitoring controller 33F has a defect monitoring unit 34 and a boulder monitoring unit 35, as described with reference to Figure 13.

[0234] Image data captured by the first imaging device 31 and the second imaging device 32 is transmitted from the work machine 1 to the monitoring controller 33F via a wireless communication system. The defect monitoring unit 34 of the monitoring controller 33F can determine whether or not there is a defect in the bucket tooth 18 based on the image data captured by the first imaging device 31. The defect monitoring unit 34 of the monitoring controller 33F can also output a warning to the output device 24F indicating that there is a defect in the bucket tooth 18. The boulder monitoring unit 35 of the monitoring controller 33F can determine whether or not there is a boulder 240 on the ground 200 based on the image data captured by the second imaging device 32. The boulder monitoring unit 35 of the monitoring controller 33F can also output a warning to the output device 24F indicating that there is a boulder 240 on the ground 200.

[0235] In the above-described embodiment, the first measuring device 31 and the second measuring device 32 are each imaging devices. At least one of the first measuring device 31 and the second measuring device 32 may be a laser device. At least one of the first measuring device 31 and the second measuring device 32 may be a radar device. The laser device acquires measurement data of the object to be measured by irradiating the object to be measured with laser light. The radar device acquires measurement data of the object to be measured by irradiating the object to be measured with sound waves. The laser device can acquire measurement data of the work machine 10 at a predetermined sampling rate by irradiating the work machine 10 with laser light. The radar device can acquire measurement data of the work machine 10 at a predetermined sampling rate by irradiating the work machine 10 with sound waves. The missing data monitoring unit 34 can change the form of the warning output from the output device 24 based on the number of missing data determination samples that indicate measurement data in which the work machine 10 has been determined to have a missing data.

[0236] In the above embodiment, the work machine 1 is a wheel loader. The work machine 1 may be other work machines such as a bulldozer and a hydraulic excavator. Each of the bulldozer and hydraulic excavator has a work mechanism and tracks that rotate in contact with the ground 200. The tracks are rotating members that rotate in contact with the ground 200. The work machine moves as the tracks rotate. [Explanation of symbols]

[0237] 1...Working machine, 2...Body, 2A...Joint mechanism, 2F...Front body, 2R...Rear body, 3...Driver's cab, 4...Running gear, 5...Wheels, 5F...Front wheels, 5R...Rear wheels, 6...Tires, 6F...Front tires, 6R...Rear tires, 7...Front fender, 8...Support member, 9...Housing, 10...Working machine, 11...Boom, 12...Bucket, 12E...End, 12M...Opening, 13...Boom cylinder, 14...Bucket cylinder, 15...Bell crank, 16...Bucket link, 17...Bucket body, 17A...Bottom plate, 17B...Top plate, 17C...Left plate, 17D...Right plate, 18...Bucket teeth, 1 9…Tooth protector, 20…Driving control device, 20A…Accelerator pedal, 20B…Brake pedal, 20F…Driving control device, 21…Control panel, 22…Monitor device, 23…Rear view monitor device, 24…Output device, 24A…Display device, 24B…Sound device, 24F…Output device, 25…Input device, 25F…Input device, 27…Pillar, 28…Headlight, 29…Turn signal lamp, 30…Monitoring system, 31…First imaging device (first measurement device), 31L…First imaging device, 31R…First imaging device, 32…Second imaging device (second measurement device), 32L…Second imaging device, 32R… Second imaging device, 33... Monitoring controller, 33F... Monitoring controller, 34... Missing data monitoring unit, 34A... First measurement data acquisition unit, 34B... Missing data determination unit, 34C... Counting unit, 34D... First warning control unit, 35... Rolling stone monitoring unit, 35A... Second measurement data acquisition unit, 35B... Judgment criterion setting unit, 35C... Warning criterion setting unit, 35D... Standard value storage unit, 35E... Input data acquisition unit, 35F... Operating status acquisition unit, 35G... Rolling stone determination unit, 35H... Second warning control unit, 36... Warning area, 37... Target image, 37L... Target image, 37R... Target image, 40... Drive system, 41... Engine, 42 ...Fuel injection system, 43...Power take-off, 44...Transmission, 44F...Forward gear, 44R...Reverse gear, 45F...Front axle, 45R...Rear axle, 46...Hydraulic pump, 47...Control valve, 48...Drive controller, 50...Recognition range, 51...Root, 52...Blade tip, 53...Search range, 60...Symbol, 61...Symbol, 62...Symbol, 63...Symbol, 64...Symbol, 65...Symbol, 66...Symbol, 67...Symbol, 68...Symbol, 70...Symbol, 70L...Symbol, 70R...Symbol, 80...Frame image, 80L...Frame image80R...frame image, 81...frame image, 82...background image, 181...bucket tooth, 182...bucket tooth, 183...bucket tooth, 184...bucket tooth, 185...bucket tooth, 186...bucket tooth, 187...bucket tooth, 188...bucket tooth, 191...tooth-tooth protector, 192...tooth-tooth protector, 193...tooth-tooth protector, 194...tooth-tooth protector, 195...tooth-tooth protector, 196...tooth-tooth protector, 197...tooth-tooth protector, 200...ground, 210...natural ground, 220...dump truck, 230...dump truck body, 240...boulder 300…Surveillance system, 301…Wireless communication device, 302…Wireless communication device, 360…Area image, 1000…Computer system, 1001…Processor, 1002…Main memory, 1003…Storage, 1004…Interface, CL…Center, FX…Rotation axis, G…Distance between reference teeth, G1…Distance, G2…Distance, G3…Distance, L…Reference tooth length, L1…Distance, L2…Distance, L3…Distance, L4…Distance, M1…Arrow, M2…Arrow, M3…Arrow, M4…Arrow, Px…Threshold, Py…Threshold, Qx…Dimension, Qy…Dimension, Ra…Measurement range, Rb…Measurement range, RX…Rotation axis, α…First field of view, β…Second field of view.

Claims

1. A first imaging device with a first field of view is provided on the work machine and captures images of the work machine itself. A second imaging device is provided on the aforementioned work machine and has a second field of view that is wider than the first field of view, which images the ground on which the work machine travels. A defect determination unit determines whether or not there is a defect in the work machine based on the image data captured by the first imaging device, The system includes a boulder detection unit that determines the presence or absence of boulders on the ground based on image data captured by the second imaging device, The work machine has a bucket and a boom connecting the body of the work machine and the bucket. The first imaging device, with the boom raised, images the bucket as it performs a dumping operation so that the excavated material held in the bucket is discharged into the dump body of a dump truck. A monitoring system for industrial machinery.

2. The second imaging device is positioned above the first imaging device. A monitoring system for a work machine according to claim 1.

3. The aforementioned work machine has a headlight and a housing that holds the headlight, Each of the first imaging device and the second imaging device is arranged in the housing, A monitoring system for a work machine according to claim 1.

4. In the vehicle width direction of the aforementioned work machine, the first imaging device and the second imaging device are each positioned outside the headlights. A monitoring system for a work machine according to claim 3.

5. In the housing, the second imaging device is positioned above the first imaging device. A monitoring system for a work machine according to claim 3 or claim 4.

6. The headlight and the housing are positioned to the left and right of the center of the work machine in the vehicle width direction of the work machine, respectively. The first imaging device is positioned in the left housing and the right housing, respectively. A monitoring system for a work machine according to any one of claims 3 to 5.

7. The first imaging device is positioned to the left and right of the center of the work machine in the vehicle width direction of the work machine, A monitoring system for a work machine according to claim 1 or claim 2.

8. The first imaging device on the left image the left portion of the work machine, The first imaging device on the right side images the right portion of the work machine. A monitoring system for a work machine according to claim 6 or claim 7.

9. The bucket includes a bucket body and a replacement member attached to the bucket body. The aforementioned defect includes the defect of the replacement member. A monitoring system for a work machine according to claim 1.

10. The aforementioned defect includes the detachment of the replacement member from the bucket body. A monitoring system for a work machine according to claim 9.

11. The replacement member includes a bucket tooth, A monitoring system for a work machine according to claim 9 or claim 10.

12. The aforementioned work machine has a rotating member that contacts the ground, and moves along the ground by the rotation of the rotating member. The second imaging device images the ground at least in the direction of travel of the rotating member. A monitoring system for a work machine according to any one of claims 1 to 11.

13. The rotating member includes a front rotating member and a rear rotating member positioned behind the front rotating member. The second imaging device images the ground in front of the front rotating member. A monitoring system for a work machine according to claim 12.

14. The headlight and the housing are positioned to the left and right of the center of the work machine in the vehicle width direction of the work machine, respectively. The second imaging device is positioned in the left housing and the right housing, respectively. A monitoring system for a work machine according to any one of claims 3 to 5.

15. The second imaging device is positioned to the left and right of the center of the work machine in the vehicle width direction of the work machine, A monitoring system for a work machine according to claim 1 or claim 2.

16. The aforementioned work machine has a rotating member that contacts the ground, and moves along the ground by the rotation of the rotating member. The rotating member is positioned to the left and right of the center of the work machine in the width direction of the work machine, The second imaging device on the left side images the ground in the direction of travel of the rotating member on the left side. The second imaging device on the right side images the ground in the direction of travel of the rotating member on the right side. A monitoring system for a work machine according to claim 14 or claim 15.

17. The aforementioned work machine is a wheel loader. A monitoring system for a work machine according to any one of claims 1 to 16.

18. The first imaging device with a first field of view, which is installed on the work machine, will capture an image of the work machine itself. The work machine is equipped with a second imaging device having a second field of view wider than the first field of view, which images the ground on which the work machine travels. Based on the image data captured by the first imaging device, it is determined whether or not there is a defect in the work machine. This includes determining the presence or absence of boulders on the ground based on the image data captured by the second imaging device, The work machine has a bucket and a boom connecting the body of the work machine and the bucket. The first imaging device, with the boom raised, images the bucket as it performs a dumping operation so that the excavated material held in the bucket is discharged into the dump body of a dump truck. Methods for monitoring industrial machinery.

Images