Abnormality determination system of work machine and abnormality determination method of work machine

Abstract

An abnormality determination system of a work machine includes a stereo camera (22) mounted on a wheel loader having a work machine (10), a position data calculation section (83) that estimates a position of the work machine (10) in an image captured by the stereo camera (22) based on a posture of the work machine (10) at the time of capturing by the stereo camera (22), and a determination section (91) that determines whether the stereo camera (22) is normal based on the estimated position of the work machine (10).

Description

Technical Field

[0001] This disclosure relates to an anomaly detection system and method for operating machinery. Background Technology

[0002] Patent Document 1 discloses an example of a work-operating machine that can accurately measure its relative position to the work object in order to automate its operation. In Patent Document 1, the relative position of a wheel loader to the work object is measured based on measurement data from a three-dimensional measuring device.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2019-132068 Summary of the Invention

[0004] In automated operations, high-precision measurement of the position of the machine is required. Therefore, at the start of operation, it is necessary to properly check whether the three-dimensional measuring device used to measure the position of the machine is functioning correctly.

[0005] The purpose of this disclosure is to properly determine whether a three-dimensional measuring device used to measure the position of a machine in a work operation is functioning correctly.

[0006] According to the present disclosure, an abnormality judgment system for operating machinery is provided, comprising: a shooting device mounted on the operating machinery having the operating machine; a position calculation unit that calculates the position of the operating machine in the image captured by the shooting device based on the posture of the operating machine when the shooting device takes a picture; and a judgment unit that determines whether the shooting device is normal based on the calculated position of the operating machine.

[0007] According to the present disclosure, a method for judging abnormalities in operating machinery is provided, comprising: calculating the position of the operating machine in the captured image based on the posture of the operating machine when the shooting device mounted on the operating machinery is taking a picture; and judging whether the shooting device is normal based on the calculated position of the operating machine.

[0008] According to the present disclosure, it is possible to appropriately determine whether a three-dimensional measuring device used to measure the position of a machine in a work operation is functioning properly. Attached Figure Description

[0009] Figure 1 This is a side view showing an example of the working machinery involved in this embodiment.

[0010] Figure 2 This is a schematic diagram illustrating the operation of the working machinery involved in this embodiment.

[0011] Figure 3 This is a schematic diagram illustrating the loading operation mode of the operating machinery involved in this embodiment.

[0012] Figure 4 This is a functional block diagram representing the control system of the operating machinery involved in this embodiment.

[0013] Figure 5 This is a diagram illustrating an example of the dimensional data of the working machine of the working machinery involved in this embodiment.

[0014] Figure 6 This is a diagram representing an example of image data acquired by a stereo camera.

[0015] Figure 7 It is a diagram used to illustrate the specified area and angle range of the operating machinery.

[0016] Figure 8 This is a flowchart illustrating the abnormality judgment method for the operating machinery involved in this embodiment.

[0017] Figure 9 It is a block diagram representing an example of a computer system. Detailed Implementation

[0018] The embodiments of this disclosure will now be described with reference to the accompanying drawings, but this disclosure is not limited thereto. The constituent elements of the embodiments described below can be appropriately combined. Furthermore, there are cases where some constituent elements are not used. The malfunction detection system for the working machinery is a system that appropriately determines whether the imaging device used to measure the position of the working machinery 10 is functioning properly when the working machinery begins operation. The malfunction detection system for the working machinery is installed by assembling the various parts of the working machinery.

[0019] Implementation

[0020] wheel loader

[0021] Figure 1This is a side view illustrating an example of the wheel loader 1 according to this embodiment. The work machine 1 performs a prescribed operation on a work object at a work site. In this embodiment, the work machine 1 is described as an articulated work machine, specifically a wheel loader 1. The prescribed operation includes digging and loading operations. The work object includes a digging object and a loading object for loading the excavated material. The wheel loader 1 performs a digging operation to excavate the digging object and a loading operation to load the excavated material into the loading object. The loading operation is a concept that includes a discharge operation to discharge the excavated material into a discharge object. As a digging object, at least one of a hill, a rocky hill, coal, and a wall can be shown. A hill is a mountain composed of sand and soil, and a rocky hill is a mountain composed of rocks or stones. Examples of loading objects can include at least one of a transport vehicle, a designated area of ​​the work site, a hopper, a belt conveyor, and a crusher.

[0022] like Figure 1 As shown, the wheel loader 1 includes: a vehicle body 2; a cab 3 with a driver's seat; a travel device 4 for moving the vehicle body 2; a transmission device 30; a workpiece 10 supported on the vehicle body 2; an angle sensor 50 for detecting the angle of the workpiece 10; a three-dimensional measuring device 20 for measuring the workpiece in front of the vehicle body 2; a buzzer 7 located around the cab 3; lights 8 located around the cab 3; and a control device 80. The angle sensor 50 is an example of an angle detection unit. The three-dimensional measuring device 20 is an example of a camera.

[0023] The vehicle body 2 includes a front part 2F and a rear part 2R. The front part 2F and the rear part 2R are connected by a joint mechanism 9.

[0024] The cab 3 is supported on the vehicle body 2. At least a portion of the wheel loader 1 is operated by the driver who sits on the cab 3.

[0025] The traveling device 4 supports the vehicle body 2. The traveling device 4 is capable of traveling on the ground RS. The traveling device 4 has wheels 5. The wheels 5 rotate based on the driving force generated by the engine mounted on the vehicle body 2. The wheels 5 include: two front wheels 5F mounted on the front part 2F of the vehicle body, and two rear wheels 5R mounted on the rear part 2R of the vehicle body. Tires 6 are mounted on the wheels 5. The tires 6 include front tires 6F mounted on the front wheels 5F and rear tires 6R mounted on the rear wheels 5R. The front wheels 5F and the front tires 6F are capable of rotating around the rotation axis FX. The rear wheels 5R and the rear tires 6R are capable of rotating around the rotation axis RX. When the vehicle body 2 is traveling in a straight line, the rotation axis FX is parallel to the rotation axis RX.

[0026] In the following explanation, the direction parallel to the rotation axis FX of the front wheel 5F is called the vehicle width direction. The direction orthogonal to the contact surface of the front tire 6F (RS) is called the vertical direction. The direction orthogonal to both the vehicle width direction and the vertical direction is called the front-rear direction.

[0027] The traveling mechanism 4 includes a drive unit 4A, a braking unit 4B, and a steering unit 4C. The drive unit 4A generates a driving force for accelerating the wheel loader 1. The drive unit 4A may include, for example, an internal combustion engine such as a diesel engine. The driving force generated by the drive unit 4A is transmitted to the wheels 5 via a transmission 30, causing the wheels 5 to rotate. The braking unit 4B generates a braking force for decelerating or stopping the wheel loader 1. The steering unit 4C is capable of adjusting the traveling direction of the wheel loader 1. The traveling direction of the wheel loader 1 includes the orientation of the front part 2F of the vehicle body. The steering unit 4C adjusts the traveling direction of the wheel loader 1 by bending the front part 2F of the vehicle body via a hydraulic cylinder.

[0028] In this embodiment, the driving device 4 is operated by a driver seated on a cab 3. A driving control device 40 for operating the driving device 4 is mounted on the cab 3. The driver operates the driving control device 40 to move the driving device 4. The driving control device 40 includes an accelerator pedal, a brake pedal, a steering lever, and a gearshift lever 41 for switching between forward and reverse. Operating the accelerator pedal increases the speed of the wheel loader 1. Operating the brake pedal decreases the speed of the wheel loader 1 or stops it. Operating the steering lever steers the wheel loader 1. Operating the gearshift lever 41 switches between forward and reverse movement of the wheel loader 1.

[0029] The transmission device 30 transmits the driving force generated by the drive device 4A to the wheels 5.

[0030] The work machine 10 is controlled by a control device 80. The work machine 10 has a boom 11 rotatably connected to the front part 2F of the vehicle body, and a bucket 12 rotatably connected to the boom 11.

[0031] The boom 11 is powered by the boom cylinder 13. The boom 11 is raised or lowered by the extension and retraction of the boom cylinder 13. The boom cylinder 13 has a boom control valve (not shown) for controlling the flow and direction of hydraulic oil supplied by a hydraulic pump (not shown).

[0032] Bucket 12 is a working component having a front end 12B containing a cutting edge. Bucket 12 is positioned in front of the front wheel 5F. Bucket 12 is connected to the front end of the boom 11. Bucket 12 is connected to bucket cylinder 14 via crank 15 and connecting rod 16. Bucket 12 is actuated by power generated by bucket cylinder 14. Bucket cylinder 14 has a bucket control valve (not shown) for controlling the flow and direction of hydraulic oil supplied from the hydraulic pump. Bucket 12 performs tipping or retracting actions by extending and retracting bucket cylinder 14. The tipping action discharges excavated material from bucket 12. The retracting action scoops up excavated material.

[0033] An angle sensor 50 is mounted on the work machine 10 and is used to detect the attitude of the work machine 10. The angle sensor 50 is used to detect the angle of the work machine 10. The angle sensor 50 includes: a boom angle sensor 51 for detecting the angle of the boom 11, and a bucket angle sensor 52 for detecting the angle of the bucket 12. The boom angle sensor 51, for example, detects the angle of the boom 11 relative to the reference axis of the vehicle body coordinate system defined by the front part of the vehicle body 2F. The bucket angle sensor 52 detects the angle of the bucket 12 relative to the boom 11. The angle sensor 50 can be a potentiometer, a stroke sensor for detecting the stroke of a hydraulic cylinder, an inertial measurement unit, or a tiltmeter. The angle data representing the angle of the work machine 10 is output to the position data calculation unit 83 and the judgment unit 91, which will be described later.

[0034] A three-dimensional measuring device 20 is mounted on a wheel loader 1. The three-dimensional measuring device 20 measures the work object in front of the front part 2F of the loader. The three-dimensional measuring device 20 measures the relative positions of each of multiple measuring points on the surface of the work object from the three-dimensional measuring device 20 to each of these measuring points, thereby measuring the three-dimensional shape of the work object. The control device 80 calculates parameters related to the work object based on the measured three-dimensional shape of the work object. As described later, when the work object is a loading object, the parameters related to the loading object include at least one of the following: the distance to the loading object, the position of the upper end of the loading object, and the height of the loading object.

[0035] The three-dimensional measuring device 20 includes a type of photogrammetric device, namely a stereo camera 22. The stereo camera 22 is respectively disposed on the right and left sides of the vehicle body 2 in the vehicle width direction. In the following description, the stereo camera 22 on one side will be described.

[0036] Stereo camera 22 captures images of the area in front. Stereo camera 22 also captures images of the work object to measure it. In this embodiment, stereo camera 22 measures the work object, which includes at least a loading object such as a transport vehicle LS. The measurement data from stereo camera 22 includes image data of the work object. The image data consists of multiple pixels. The image data is an example of measurement data.

[0037] The stereo camera 22 has a pair of first imaging devices 22A and second imaging devices 22B. The first imaging device 22A and the second imaging device 22B are arranged spaced apart. The first image data acquired by the first imaging device 22A and the second image data acquired by the second imaging device 22B are output to the control device 80. The first image data and the second image data are two-dimensional image data.

[0038] Buzzer 7 is located near the control panel 3. Buzzer 7 is a buzzing device used to output a warning sound. Buzzer 7 is used to output the judgment result of the judgment unit 91. Buzzer 7 outputs a warning sound when the judgment unit 91 determines that an abnormality exists.

[0039] The light 8 is located near the control panel 3. The light 8 is used to output the judgment result of the judgment unit 91. When the judgment unit 91 determines that everything is normal, the light 8 illuminates as a warning light. When the judgment unit 91 determines that an abnormality exists, the light 8 causes the warning light to flash.

[0040] Work

[0041] Figure 2 This is a schematic diagram illustrating the operation of the wheel loader 1 according to this embodiment. The wheel loader 1 operates in multiple working modes. These working modes include: a digging working mode in which the bucket 12 of the loader 10 digs an object to be dug, and a loading working mode in which the excavated material scooped up by the bucket 12 in the digging working mode is loaded onto a loading object. The object to be dug is, for example, a hill DS on the ground RS. The loading object is, for example, the bucket BE of a transport vehicle LS capable of traveling on the ground RS. The transport vehicle LS is, for example, a dump truck.

[0042] In excavation mode, the wheel loader 1 moves towards the hill DS with no excavated material in its bucket 12. The driver operates the driving control device 40 to make the wheel loader 1 move as... Figure 2 As indicated by arrow M1, proceed forward and approach hill DS. Control device 80 controls machine 10 to cause bucket 12 to excavate hill DS. Hill DS is excavated by bucket 12, and the excavated material is scooped up by bucket 12.

[0043] The wheel loader 1, with the excavated material in its bucket 12, reverses away from the hill DS. The driver operates the driving control device 40 to move the wheel loader 1 as follows: Figure 2As indicated by arrow M2, retreat and move away from hill DS.

[0044] Next, the loading operation mode is executed. In loading operation mode, the wheel loader 1, with the excavated material in the bucket 12, moves towards the transport vehicle LS. The driver operates the driving control device 40 to make the wheel loader 1 move as... Figure 2 As indicated by arrow M3, the machine turns and moves forward to approach the transport vehicle LS. At this time, the three-dimensional measuring device 20 mounted on the wheel loader 1 measures the transport vehicle LS. The control device 80 controls the machine 10 based on the measurement data from the three-dimensional measuring device 20 to load the excavated material held in the bucket 12 into the bucket BE of the transport vehicle LS. That is, while the wheel loader 1 is moving forward in a manner approaching the transport vehicle LS, the control device 80 controls the machine 10 to raise the boom 11. After the boom 11 has raised and the bucket 12 has been positioned above the bucket BE, the control device 80 controls the machine 10 to tip the bucket 12. The excavated material is discharged from the tipped bucket 12 and loaded into the bucket BE.

[0045] After the excavated material is loaded into the bucket BE, the wheel loader 1 reverses away from the transport vehicle LS with no excavated material remaining in the bucket 12. The driver operates the driving control device 40 to make the wheel loader 1... Figure 2 As indicated by arrow M4, turn and reverse to move away from the transport vehicle LS.

[0046] The driver and control device 80 repeatedly perform the above actions until the bucket BE is full of excavated material, or until the excavation of the hill DS is completed.

[0047] Figure 3 This is a schematic diagram illustrating the loading operation mode of the wheel loader 1 according to this embodiment. The driver operates the driving control device 40 to steer the wheel loader 1 and move it forward to approach the transport vehicle LS. Figure 3 As shown in (A), the three-dimensional measuring device 20 measures the three-dimensional shape of the transport vehicle LS and its relative position to the transport vehicle LS. Based on the measurement data from the three-dimensional measuring device 20, the control device 80 detects the distance Db between the wheel loader 1 and the transport vehicle LS, as well as the height Hb of the upper end BEt of the bucket BE.

[0048] like Figure 3As shown in (B), when the wheel loader 1 is moving forward in a manner close to the transport vehicle LS, the control device 80 controls the angle of the bucket 12 and raises the boom 11 based on the measurement data of the three-dimensional measuring device 20, so that the bucket 12 is positioned above the upper end BEt of the bucket BE, and the excavated material in the bucket 12 is kept from falling out of the bucket 12.

[0049] like Figure 3 As shown in (C), after the boom 11 is raised and the bucket 12 is positioned above the hopper BE, the control device 80 controls the work machine 10 to tip the bucket 12. As a result, the excavated material is discharged from the bucket 12 and loaded into the hopper BE.

[0050] exist Figure 3 After (C), the driver operates the driving control device 40 to turn the wheel loader 1 and reverse it away from the transport vehicle LS.

[0051] Control device

[0052] Figure 4 This is a functional block diagram illustrating the control system 200 of the wheel loader 1 according to this embodiment. The control device 80 includes a computer system. The control device 80 controls the wheel loader 1. The control device 80 is connected to the work machine 10, the three-dimensional measuring device 20, the angle sensor 50, the driving operation device 40, the buzzer 7, and the light 8. The control device 80 includes: a measurement data acquisition unit 81, a storage unit 82, a position data calculation unit 83, an object calculation unit 86, a work machine control unit 87, a judgment unit 91, and an output control unit 92. The buzzer 7 is an example of an output unit. The light 8 is an example of an output unit. The position data calculation unit 83 is an example of a position calculation unit.

[0053] Control system 200 is an example of an anomaly detection system. Control system 200 includes: a work machine 10, a three-dimensional measuring device 20, an angle sensor 50, a driving operation device 40, a buzzer 7, a light 8, and a control device 80.

[0054] The measurement data acquisition unit 81 is used to acquire measurement data from the three-dimensional measurement device 20. In this embodiment, the measurement data acquisition unit 81 acquires first image data from the first imaging device 22A of the stereo camera 22 and second image data from the second imaging device 22B. The image data of the work object acquired by the measurement data acquisition unit 81 is output to the object calculation unit 86 and the judgment unit 91.

[0055] Storage unit 82 is used to store machine data. Machine data includes, for example, design data or specification data including CAD (Computer-Aided Design) data of machine 10. Machine data also includes shape data including dimensional data of machine 10.

[0056] In this embodiment, the machine data includes: boom length, bucket length, and bucket shape. Boom length refers to the distance between the boom rotation axis and the bucket rotation axis. Bucket length refers to the distance between the bucket rotation axis and the front end 12B of the bucket 12. The boom rotation axis refers to the rotation axis of the boom 11 relative to the front part 2F of the vehicle body, including the connecting pin connecting the front part 2F of the vehicle body and the boom 11. The bucket rotation axis refers to the rotation axis of the bucket 12 relative to the boom 11, including the connecting pin connecting the boom 11 and the bucket 12. The bucket shape includes the shape and size of the bucket 12. The dimensions of the bucket 12 include: the bucket width (indicating the distance between the left and right ends of the bucket 12), the height of the opening of the bucket 12, and the length of the bucket bottom surface, etc.

[0057] The dimensional data of the bucket 12 defines the shape of the bucket 12. In this embodiment, the dimensional data refers to the positional data of multiple locations on the outer periphery of the bucket 12. For example, the dimensional data refers to the positional data of five locations on the outer periphery of the bucket 12.

[0058] Figure 5 This is an example diagram illustrating the dimensional data of the bucket 12 of the wheel loader 1 according to this embodiment. In this embodiment, the dimensional data of the bucket 12 is the position data of five points PA0, PB0, PC0, PD0, and PE0 on the outer periphery of the bucket 12. The shape connecting points PA0, PB0, PC0, PD0, and PE0 is a pentagon. Taking into account the measurement error of each point, the pentagon is enlarged to define points PA, PB, PC, PD, and PE. The bucket 12 is located inside the pentagon formed by points PA, PB, PC, PD, and PE.

[0059] The position data calculation unit 83 calculates position data representing the attitude of the work machine 10 based on the detection results of the angle sensor 50. More specifically, the position data calculation unit 83 calculates the position data of the work machine 10 based on the angle data of the work machine 10 detected by the angle sensor 50 and the work machine data of the work machine 10 stored in the storage unit 82. The position data of the work machine 10 includes, for example, the position data of each part of the bucket 12 in the vehicle coordinate system. The position data of the work machine 10 calculated by the position data calculation unit 83 is output to the judgment unit 91.

[0060] The position data calculation unit 83 calculates the position of the work machine 10 in the image captured by the stereo camera 22 based on the posture of the work machine 10 when the stereo camera 22 takes pictures.

[0061] An example of a method for calculating the position of the work machine 10 in a captured image will be explained. The position data calculation unit 83 calculates the three-dimensional position of the work machine 10 in the vehicle body coordinate system based on the boom angle sensor 51 used to detect the angle of the boom 11, the bucket angle sensor 52 used to detect the angle of the bucket 12, and the size data of the work machine 10. The position data calculation unit 83 performs coordinate transformation on the three-dimensional position of the work machine 10 in the vehicle body coordinate system to calculate the three-dimensional position of the work machine 10 in the imaging device coordinate system. Furthermore, imaging device coordinate systems are defined for the first imaging device 22A and the second imaging device 22B of the stereo camera 22. The imaging device coordinate system is a coordinate system with the origin fixed at the first imaging device 22A and the second imaging device 22B, respectively, as a reference. By transforming the three-dimensional position of the work machine 10 in the imaging device coordinate system to the projection planes of the first imaging device 22A and the second imaging device 22B obtained from the installation position of the stereo camera 22, the position data calculation unit 83 is able to calculate the position of the work machine 10 in the captured image. As a method of transformation to the projection plane, perspective projection can be used, for example. Alternatively, the position of the work machine 10 in the image captured by either the first imaging device 22A or the second imaging device 22B can be calculated.

[0062] The object calculation unit 86 calculates the three-dimensional data of the work object measured by the stereo camera 22 based on the measurement data acquired by the measurement data acquisition unit 81. The work object refers to the transport vehicle LS containing the container BE. The three-dimensional data of the work object represents the three-dimensional shape of the transport vehicle LS.

[0063] The object calculation unit 86 performs image processing on the image data acquired by the first imaging device 22A and the image data acquired by the second imaging device 22B based on the principle of triangulation to measure the three-dimensional shape of the work object. The object calculation unit 86 performs stereoscopic processing on the image data, namely the first image data and the second image data, to calculate the distance from the stereo camera 22 to multiple measurement points on the surface of the work object displayed in each pixel. Based on the distances to each measurement point, the object calculation unit 86 calculates, for example, three-dimensional data in a vehicle body coordinate system.

[0064] In this embodiment, the object calculation unit 86 calculates parameters related to the transport vehicle LS based on the three-dimensional data of the transport vehicle LS. The parameters related to the transport vehicle LS include: the position (height) of the upper end BEt of the transport vehicle LS (bucket BE) relative to the ground RS, and the distance Db from the wheel loader 1 to the transport vehicle LS. The distance Db from the wheel loader 1 to the transport vehicle LS is, for example, the distance between the front end 12B of the bucket 12 and the nearest contact point, where the nearest contact point represents the part of the transport vehicle LS closest to the front end 12B of the bucket 12 in the horizontal direction.

[0065] The machine control unit 87 controls the operation of the machine 10 loading excavators onto the work object based on the three-dimensional data of the work object calculated by the object calculation unit 86. In this embodiment, the machine control unit 87 controls the operation of the machine 10 loading excavators into the bucket BE based on the calculated three-dimensional data of the transport vehicle LS. The machine control unit 87 controls the operation of the machine 10 loading excavators into the bucket BE based on height data representing the height Hb of the upper end BEt of the bucket BE and distance data representing the distance Db from the wheel loader 1 to the transport vehicle LS.

[0066] The machine control unit 87 controls the movement of the machine 10, including controlling the movement of at least one of the boom cylinder 13 and the bucket cylinder 14. More specifically, the machine control unit 87 outputs a control signal to the boom control valve to control the flow rate and direction of the hydraulic oil supplied to the boom cylinder 13, thereby controlling the raising and lowering movement of the boom 11. The machine control unit 87 outputs a control signal to the bucket control valve to control the flow rate and direction of the hydraulic oil supplied to the bucket cylinder 14, thereby controlling the raising and lowering movement of the bucket 12.

[0067] In this embodiment, the wheel loader 1 has a transmission control unit 88 and a travel control unit 89.

[0068] The transmission control unit 88 outputs control signals for controlling the transmission device 30.

[0069] The driving control unit 89 controls the operation of the driving device 4 based on the driver's operation of the driving operation device 40. The driving control unit 89 outputs operating commands to activate the driving device 4. The driving control unit 89 outputs acceleration commands to activate the drive unit 4A. The driving control unit 89 outputs braking commands to activate the braking device 4B. The driving control unit 89 outputs steering commands to activate the steering device 4C.

[0070] The judgment unit 91 determines whether the stereo camera 22 is functioning correctly based on the position of the work machine 10 calculated by the position data calculation unit 83. More specifically, the judgment unit 91 determines whether the stereo camera 22 is functioning correctly based on the position of the work machine 10 in the captured image calculated by the position data calculation unit 83 and the actual position of the work machine 10 in the captured image. If the calculated position of the work machine 10 in the captured image matches the actual position of the work machine 10 in the captured image, or if the error is within a specified range, the judgment unit 91 determines that the stereo camera 22 is functioning correctly. If the calculated position of the work machine 10 in the captured image does not match the actual position of the work machine 10 in the captured image, or if the error is outside the specified range, the judgment unit 91 determines that the stereo camera 22 is malfunctioning.

[0071] Other judgment methods in the judgment unit 91 will be explained. The position of the work machine 10 in the captured images by the stereo camera 22 is mapped to the posture of the work machine 10 in advance and stored in the storage unit 82. The position data calculation unit 83 obtains the position of the work machine 10 in the captured images from the storage unit 82 based on the posture of the work machine 10 when the stereo camera 22 takes the picture. Obtaining the position of the work machine 10 in the captured images from the storage unit 82 is an example of estimating the position of the work machine. The judgment unit 91 can determine whether the stereo camera 22 is functioning properly by comparing the position of the work machine 10 in the captured images obtained from the storage unit 82 with the actual position of the work machine 10 in the captured images by the stereo camera 22 based on the posture of the work machine 10 when the stereo camera 22 takes the picture. Furthermore, the position of the work machine 10 in the captured images by the stereo camera 22 can be mapped to a combination of the size data of each part of the work machine 10 and the position of the work machine 10 in advance and stored in the storage unit 82.

[0072] use Figure 6 The judgment method of the judgment unit 91 will be explained in detail. Figure 6 This diagram illustrates an example of image data 100 captured by the stereo camera 22. Based on the position of the bucket 12, the range within which the bucket 12 is captured in the image data 100 can be defined. More specifically, firstly, the determination unit 91 performs image processing such as pattern matching on the image data 100 to identify the bucket 12. Then, the determination unit 91 counts the number of pixels of the bucket 12 within the range of the calculated position of the machine 10 by the position data calculation unit 83, and the number of pixels of the bucket 12 outside the calculated range of the machine 10's position. If the number of pixels of the bucket 12 within the range is above a threshold, the determination unit 91 determines it to be normal. If the number of pixels of the bucket 12 within the range is below the threshold, the determination unit 91 determines it to be abnormal. For example, in... Figure 6 In the image data 100 shown, the estimated position of the bucket 12 is within region 101. In other words, in the image data 100, the bucket 12 is represented by pixels within region 101.

[0073] In this embodiment, the determination unit 91 can also perform the determination when the work machine 10 is within the specified area A1 and the specified angle A2. The image data 100 may include, for example, images of the ground RS, surrounding objects, or the work machine 10. This is to avoid making incorrect determinations based on image data where objects other than the work machine 10 are detected.

[0074] Figure 7 This diagram illustrates the defined area A1 and defined angle range A2 of the working machine 10 of the wheel loader 1. In this embodiment, the determination unit 91 makes a determination when the bucket 12 is within the defined area A1 and the defined angle range A2. The determination unit 91 makes a determination when the bucket 12 is within the defined area A1, which is forward of the wheel loader 1, and the angle of the bucket 12 is within the defined angle range A2. There is a risk of false detection of the vehicle body 2 on the side closer to the wheel loader 1 than the defined area A1. There is a risk of false detection of surrounding buildings and obstacles such as those at the work site on the side farther away from the defined area A1. When the angle of the bucket 12 is outside the defined angle range A2, the bucket 12 is not captured in the image data.

[0075] In this embodiment, the determination unit 91 can also determine whether the positional relationship of the work machine 10 obtained based on the stereo camera 22 and the angle sensor 50 is normal. In other words, the determination unit 91 can also determine whether the positional relationship of the work machine 10 obtained based on the stereo camera 22 and the angle sensor 50 is normal.

[0076] For example, if the positional relationship of the machine 10 obtained based on the stereo camera 22 and the angle sensor 50 is determined to be normal, then all of the following points are satisfied: the installation posture of the unit of the stereo camera 22 is appropriate, the relative posture of the first shooting device 22A and the second shooting device 22B of the stereo camera 22 is appropriate, the installation posture of the boom angle sensor 51 and the bucket angle sensor 52 of the angle sensor 50 is appropriate, and the dimensional data is input appropriately.

[0077] For example, if the positional relationship of the machine 10 obtained based on the stereo camera 22 and the angle sensor 50 is determined to be abnormal, the cause is at least one of the following: the installation posture of the unit of the stereo camera 22 is deviated, the relative posture of the first shooting device 22A and the second shooting device 22B of the stereo camera 22 is deviated, the installation posture of the boom angle sensor 51 or the bucket angle sensor 52 of the angle sensor 50 is deviated, or the dimensional data is incorrect.

[0078] The determination unit 91 performs the aforementioned determination process on both the right-side stereo camera 22 and the left-side stereo camera 22. If it determines that both the right-side and left-side stereo camera 22 are abnormal, it can determine that an abnormality exists outside of the stereo camera 22. If it determines that only one of the right-side and left-side stereo camera 22 is abnormal, it can determine that an abnormality exists within the stereo camera 22.

[0079] The output control unit 92 controls the judgment result of the output judgment unit 91. If the judgment unit 91 determines that an abnormality exists, the output control unit 92 controls the buzzer 7 to output a warning sound. If the judgment unit 91 determines that an abnormality exists, the output control unit 92 controls the light 8 to flash. If the judgment unit 91 determines that the condition is normal, the output control unit 92 controls the light 8 to illuminate.

[0080] Methods for identifying anomalies during the initial inspection of a task

[0081] Figure 8 This is a flowchart illustrating the abnormality detection method of the wheel loader 1 according to this embodiment. At the start of operation using the wheel loader 1, the driver starts the wheel loader 1 in operation start check mode via an operation unit (not shown).

[0082] The control device 80 receives the operation start check mode indication (step S11) via an operation receiving unit (not shown). The control device 80 then proceeds to step S12.

[0083] When the driver checks the mode at the start of the operation, he causes the work machine 10 of the wheel loader 1 to rise and fall.

[0084] The work machine 10 is photographed (step S12). More specifically, the stereo camera 22 measures the front. The measurement data from the stereo camera 22 is output to the measurement data acquisition unit 81 of the control device 80. The control device 80 acquires image data of the front of the vehicle body 2, including the work machine 10 photographed by the stereo camera 22, through the measurement data acquisition unit 81. The image data including the work machine 10 acquired by the measurement data acquisition unit 81 is output to the judgment unit 91. The control device 80 proceeds to step S13.

[0085] The angle of the work machine 10 is detected (step S13). The angle sensor 50 detects the angle of the work machine 10 when the stereo camera 22 is taking pictures. The angle data representing the angle of the work machine 10 is output to the position data calculation unit 83 and the judgment unit 91 of the control device 80. The control device 80 calculates the position data representing the attitude of the work machine 10 based on the angle data detected by the angle sensor 50 using the position data calculation unit 83. The position data calculated by the position data calculation unit 83 is output to the judgment unit 91. The control device 80 proceeds to step S14.

[0086] These steps S12 and S13 are performed during the raising and lowering of the work machine 10. For example, they can be repeated at predetermined time intervals during the raising and lowering of the work machine 10. For example, they can be performed when the work machine 10 reaches a predetermined position during the raising and lowering of the work machine 10. For example, they can be performed during the raising and lowering of the work machine 10, such as... Figure 7 As shown, the process is repeated when the machine 10 is within the specified area A1 and the specified angle A2.

[0087] The control device 80 determines whether an abnormality exists via the judgment unit 91 (step S14). In this embodiment, the control device 80, via the judgment unit 91, compares the calculated position of the work machine 10 in the measurement data, defined by the posture of the work machine 10 during the stereo camera 22's capture, with the position of the work machine 10 in the measurement data to determine whether the installation position of the stereo camera 22 is normal. More specifically, the position data calculation unit 83 calculates the range in the image data 100 where the bucket 12 can be captured, based on the work machine's position data and the work machine 10's size data. Then, if the judgment unit 91 captures the bucket 12 at the calculated position in the image data, it determines that the stereo camera 22 is normal. If the judgment unit 91 does not capture the bucket 12 at the calculated position in the image data, it determines that the stereo camera 22 is abnormal. The control device 80 proceeds to step S15.

[0088] The following describes a scenario where steps S12 and S13 are performed multiple times, resulting in the stereo camera 22 capturing multiple image data. In this case, the determination unit 91 determines whether the mounting position of the stereo camera 22 is normal based on each image data. Furthermore, if a predetermined proportion or more of the image data is found to be abnormal, the determination unit 91 can determine that the stereo camera 22 is malfunctioning.

[0089] The control device 80 outputs the judgment result of the judgment unit 91 through the output control unit 92 (step S15). In this embodiment, if an abnormality is determined, the control device 80 controls the buzzer 7 to output a warning sound and causes the light 8 to flash through the output control unit 92. If a normal condition is determined, the control device 80 controls the vehicle light 8 to illuminate through the output control unit 92. The control device 80 then terminates the process.

[0090] Computer System

[0091] Figure 9 This is a block diagram illustrating an example of a computer system 1000. The aforementioned control device 80 is constituted by the computer system 1000. The computer system 1000 includes: a processor 1001, such as a CPU (Central Processing Unit); main memory 1002, which includes non-volatile memory such as ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory); memory 1003; and an interface 1004, which includes input / output circuitry. The functions of the aforementioned control device 80 are stored as a program in memory 1003. The processor 1001 reads the program from memory 1003 and loads it into main memory 1002, thereby executing the aforementioned processing according to the program. Furthermore, the program can also be transmitted to the computer system 1000 via a network.

[0092] Effect

[0093] As described above, this embodiment can determine whether the three-dimensional measuring device 20 is functioning correctly based on the estimated position of the work machine 10 in the measurement data measured by the three-dimensional measuring device 20, which is defined by the posture of the work machine 10 during measurement. According to this embodiment, it is possible to appropriately determine whether the three-dimensional measuring device 20 for measuring the position of the work machine 10 of the wheel loader 1 is functioning correctly.

[0094] This embodiment can determine whether the three-dimensional measuring device 20 is normal, whether the angle sensor 50 is normal, and whether the input of the size data is appropriate by judging whether the positional relationship of the machine 10 obtained based on the three-dimensional measuring device 20 and the angle sensor 50 is normal.

[0095] For example, if the positional relationship of the machine 10 obtained based on the three-dimensional measuring device 20 and the angle sensor 50 is determined to be normal, then it can be confirmed that all of the following points are satisfied: the installation posture of the unit of the stereo camera 22 is appropriate, the relative posture of the first shooting device 22A and the second shooting device 22B of the stereo camera 22 is appropriate, the installation posture of the boom angle sensor 51 and the bucket angle sensor 52 of the angle sensor 50 is appropriate, and the dimensional data is input appropriately.

[0096] For example, if the positional relationship of the machine 10 obtained based on the three-dimensional measuring device 20 and the angle sensor 50 is determined to be abnormal, the cause can be determined to be at least one of the following: deviation of the installation posture of the unit of the stereo camera 22, deviation of the relative posture of the first shooting device 22A and the second shooting device 22B of the stereo camera 22, deviation of the installation posture of the boom angle sensor 51 or the bucket angle sensor 52 of the angle sensor 50, or incorrect dimensional data.

[0097] For example, if it is determined that both the right-side stereo camera 22 and the left-side stereo camera 22 are abnormal, it can be determined that there is an abnormality outside of stereo camera 22. For example, if it is determined that either the right-side stereo camera 22 or the left-side stereo camera 22 is abnormal, it can be determined that there is an abnormality within stereo camera 22.

[0098] This embodiment can further and appropriately determine whether the three-dimensional measuring device 20 is functioning properly based on the posture of the machine 10 during measurement by the three-dimensional measuring device 20 and the estimated position of the machine 10 in the measurement data as specified by the size data of the machine 10.

[0099] In this embodiment, position data from multiple locations on the outer periphery of the work machine 10 are used as dimensional data. This embodiment can appropriately define the shape of the work machine 10. Furthermore, in this embodiment, position data from five locations on the outer periphery of the work machine 10 are used as dimensional data. This embodiment can more appropriately define the shape of the work machine 10.

[0100] This embodiment can suppress false detections of the vehicle body 2, surrounding buildings, and obstacles by the three-dimensional measuring device 20 by determining when the work machine 10 is within a specified area and angle. According to this embodiment, the state of the bucket 12 not captured in the image data captured by the three-dimensional measuring device 20 can be removed for judgment.

[0101] This embodiment can compare the calculated position of the machine 10 in the measurement data, which is defined by the posture of the machine 10 when the three-dimensional measuring device 20 is performing the measurement, with the position of the machine 10 in the measurement data to determine whether the installation position of the three-dimensional measuring device 20 is normal.

[0102] This embodiment calculates position data representing the attitude of the machine 10 based on the detection results of the angle sensor 50. This embodiment can appropriately calculate the attitude of the machine 10.

[0103] This embodiment can determine whether the positional relationship of the work machine 10 obtained based on the three-dimensional measuring device 20 and the angle sensor 50 is normal. According to this embodiment, it is possible to determine whether the three-dimensional measuring device 20 and the angle sensor 50 are functioning properly.

[0104] This embodiment can make a judgment based on the posture of the machine 10 during measurement by the three-dimensional measuring device 20 and the calculated position of the machine 10 in the measurement data specified by the machine data stored in the storage unit 82. According to this embodiment, the judgment can be made more appropriately.

[0105] This embodiment can output the judgment result through, for example, a buzzer 7 or a lamp 8.

[0106] Other implementation methods

[0107] In the above embodiments, the three-dimensional measuring device 20 is not limited to the stereo camera 22; for example, it can also be a camera or a laser scanner. The stereo camera 22 can be disposed on either the right or left side of the vehicle body 2. Figure 1 The configuration of the stereo camera 22 shown is an example, but it can also be configured elsewhere.

[0108] The work site where the wheel loader 1 performs its operations can be a mining site, a construction site, or an engineering site.

[0109] The wheel loader 1 can be used for snow removal, agricultural and livestock operations, and forestry operations.

[0110] In the above embodiments, the bucket 12 may have multiple bucket teeth or a straight blade.

[0111] The working component connected to the front end of the boom 11 may not be the bucket 12. It may be a snowplow or snowplow used in snow removal operations, a hay bale grab or fork used in agricultural and livestock operations, or a fork or bucket used in forestry operations.

[0112] The angle sensor 50 may also include either the boom angle sensor 51 or the bucket angle sensor 52.

[0113] Alternatively, the judgment result can be displayed on a monitor (not shown) located on the wheel loader 1, instead of the light 8. The wheel loader 1 does not necessarily need to have a buzzer 7, a light 8, or a monitor; it may have any one or more of them. Furthermore, the buzzer 7, light 8, and monitor may also be located on the exterior of the wheel loader 1.

[0114] The control system 200 described in the above embodiments may also have some components mounted inside the work machine 1, while other components are located outside the work machine 1. Furthermore, the control system 200 described in the above embodiments includes the work machine 10, the three-dimensional measuring device 20, the angle sensor 50, the driving operation device 40, the buzzer 7, the light 8, and the control device 80, but it is not limited to this and may not include some of these components. As an example, a control system 200 may also be provided without the buzzer 7 and the light 8.

[0115] The control device 80 described in the above embodiments can be a single computer, or the configuration of the control device 80 can be separately configured in multiple computers, and the multiple computers can work together to perform the function of the control device 80.

[0116] The working machine 1 is not limited to wheel loaders. For example, the control device 80 and the abnormality judgment method described in the above embodiments can also be applied to working machines with working machines, such as hydraulic excavators or bulldozers.

[0117] Symbol Explanation

[0118] 1…Wheel loader (operating machinery); 2…Body; 2F…Front of the body; 2R…Rear of the body; 3…Crossboard; 4…Travel mechanism; 4A…Drive mechanism; 4B…Brake mechanism; 4C…Steering mechanism; 5…Wheel; 5F…Front wheel; 5R…Rear wheel; 6…Tire; 6F…Front tire; 6R…Rear tire; 7…Buzzer (output unit); 8…Light (output unit); 9…Joint mechanism; 10…Working machine; 11…Boom; 12…Bucket; 12B…Front end; 13…Boom cylinder; 14…Bucket cylinder; 15…Crank; 16…Connecting rod; 20…Three-dimensional measuring device; 22…Stereo camera; 22A…First imaging device; 22B…Second imaging device; 30…Speed ​​changeover; 40…Travel operation device; 50…Angle sensor (angle detection unit); 51…Boom angle sensor; 52…Bucket angle sensor; 80…Control device; 81…Measurement data acquisition unit; 82…Storage unit; 83…Position data calculation unit (position calculation unit); 86…Object calculation unit; 87…Worker control unit; 88…Speed ​​changeover control unit; 89…Travel control unit; 91…Judgment unit; 92…Output control unit; 100…Image data; 200…Control system (anomaly judgment system); BE…Bucket (loading object); DS…Shore (excavation object); FX…Spinning shaft; LS…Transport vehicle; RX…Spinning shaft; RS…Ground.

Claims

1. An anomaly detection system for operating machinery, characterized in that, have: The filming device is mounted on a machine equipped with a working machine; The position calculation unit calculates the position of the machine in the image captured by the shooting device based on the posture of the machine when the shooting device takes the picture; as well as The judgment unit, based on the calculated position of the work machine, determines whether the shooting device is functioning properly. The position calculation unit calculates the position of the work machine in the captured image based on the posture of the work machine and the size data of the work machine when the shooting device takes the picture. The judgment unit judges the shooting device to be normal if the calculated position of the work machine is consistent with the actual position of the work machine in the captured image, or if the error is within the specified range.

2. The abnormality judgment system for operating machinery according to claim 1, characterized in that, The dimensional data refers to the positional data of multiple locations on the outer periphery of the machine.

3. The abnormality judgment system for operating machinery according to claim 2, characterized in that, The dimensional data refers to the positional data of five locations on the outer periphery of the machine.

4. The abnormality judgment system for operating machinery according to any one of claims 1 to 3, characterized in that, The judgment unit makes a judgment when the machine is within a specified area and at a specified angle.

5. The abnormality judgment system for operating machinery according to claim 1, characterized in that, The judgment unit compares the calculated position of the work machine in the captured image, determined by the posture of the work machine when the shooting device takes the picture, with the actual position of the work machine in the captured image to determine whether the shooting device is working properly.

6. The abnormality judgment system for operating machinery according to any one of claims 1 to 3, characterized in that, The judgment unit determines whether the installation position of the shooting device is normal.

7. The abnormality judgment system for operating machinery according to any one of claims 1 to 3, characterized in that, have: An angle detection unit, mounted on the work machine, is used to detect the attitude of the work machine; and The position data calculation unit calculates position data representing the attitude of the machine based on the detection results of the angle detection unit.

8. The abnormality judgment system for operating machinery according to claim 7, characterized in that, The judgment unit determines whether the shooting device and the angle detection unit are functioning properly.

9. The abnormality judgment system for operating machinery according to claim 8, characterized in that, The judgment unit determines whether the positional relationship of the work machine obtained based on the shooting device and the angle detection unit is normal.

10. The abnormality judgment system for operating machinery according to any one of claims 1 to 3, characterized in that, have: The storage unit stores work machine data, including the size and shape data of the work machine. The judgment unit makes a judgment based on the posture of the machine when the shooting device takes the picture and the estimated position of the machine in the captured image as defined by the machine data stored in the storage unit.

11. The abnormality judgment system for operating machinery according to any one of claims 1 to 3, characterized in that, have: The output unit outputs the judgment result of the judgment unit.

12. The abnormality judgment system for operating machinery according to claim 11, characterized in that, The output unit is a buzzer.

13. The abnormality judgment system for operating machinery according to claim 11, characterized in that, The output unit is a lamp.

14. A method for judging abnormalities in operating machinery, characterized in that, include: Based on the posture of the machine when the shooting device mounted on the machine is taking pictures, the position of the machine in the image captured by the shooting device is estimated; Based on the calculated position of the work machine, determine whether the shooting device is functioning properly. Based on the posture and size data of the machine when the shooting device takes the picture, the position of the machine in the captured image is calculated. If the calculated position of the machine matches the actual position of the machine in the captured image, or if the error is within a specified range, the shooting device is judged to be normal.

15. The method for judging abnormalities in operating machinery according to claim 14, characterized in that, The position of the machine in the captured image, which is determined by the posture of the machine when the shooting device takes the picture, is compared with the actual position of the machine in the captured image to determine whether the shooting device is working properly.

Images