Surrounding monitoring system for industrial machinery and method for monitoring the surroundings of industrial machinery

Abstract

To improve the accuracy of detecting people and other objects around working machinery. [Solution] The surrounding monitoring system 10 for the work machine includes a camera 11 that acquires the surrounding conditions of the hydraulic excavator 100, a radar 13 that acquires the surrounding conditions of the hydraulic excavator 100, and a determination unit 25 that determines that a person has been detected based on the detection results from the image taken by the camera 11 when the value indicating human-likeness is greater than a first threshold, or when the value indicating human-likeness is less than the first threshold and greater than a second threshold, and when an obstacle is detected from the detection data detected by the radar 13.

Description

Technical Field

[0001] The present disclosure relates to a peripheral monitoring system for a work machine and a method for monitoring the periphery of a work machine.

Background Art

[0002] In the technical field related to work machines, there is known a technique of obtaining a plurality of cameras for acquiring the surrounding situation of a work machine and displaying an overhead image and a single-camera image captured by the plurality of cameras, as disclosed in Patent Document 1. In the technique described in Patent Document 1, obstacles are detected based on the information of a radar, and when an obstacle is detected, a buzzer is sounded for notification.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the detection of people and the like around a work machine, improvement in accuracy is desired.

[0005] Therefore, an object is to improve the accuracy of detecting people and the like around a work machine.

Means for Solving the Problems

[0006] According to the present disclosure, there is provided a peripheral monitoring system for a work machine, including a first detection unit that acquires the surrounding situation of the work machine, a second detection unit that acquires the surrounding situation of the work machine, and a determination unit that determines that a person has been detected when a value indicating humanity is greater than a first threshold based on the detection result of the first detection unit, or when the value indicating humanity is less than the first threshold and greater than a second threshold smaller than the first threshold and the second detection unit has detected an obstacle.

[0007] The present disclosure provides a method for monitoring the surroundings of a work machine, comprising: a first detection unit for acquiring the surrounding conditions of the work machine; a second detection unit for acquiring the surrounding conditions of the work machine; and a controller, wherein the work machine detects a person when, based on the detection result of the first detection unit, the value indicating human-likeness is greater than a first threshold, or when the value indicating human-likeness is less than the first threshold and greater than a second threshold that is less than the first threshold, and the second detection unit has detected an obstacle. [Effects of the Invention]

[0008] According to this disclosure, the accuracy of detecting people and other objects around working machinery can be improved. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a schematic diagram showing an example of a work machine. [Figure 2] Figure 2 is a block diagram showing the surrounding monitoring system for a work machine according to an embodiment. [Figure 3] Figure 3 is a schematic diagram showing an example of an overhead view. [Figure 4] Figure 4 is a block diagram showing a computer system according to an embodiment. [Figure 5] Figure 5 is a diagram showing an example of a flowchart illustrating a method for monitoring the surroundings of a work machine according to the embodiment. [Figure 6] Figure 6 shows another example of a flowchart illustrating a method for monitoring the surroundings of a work machine according to the embodiment. [Modes for carrying out the invention]

[0010] The embodiments described below will be explained with reference to the drawings, but the disclosure is not limited thereto. The components of the embodiments described below can be combined as appropriate. In addition, some components may not be used.

[0011] [Embodiment] Figure 1 is a schematic diagram showing an example of a work machine. In this embodiment, the work machine 100 is a hydraulic excavator. In the following description, the work machine 100 will be referred to as hydraulic excavator 100 as appropriate.

[0012] <Working machinery> The hydraulic excavator 100 comprises a hydraulically operated work implement 101, a slewing body 102 that supports the work implement 101, and a traveling body 103 that supports the slewing body 102. The slewing body 102 is capable of rotatable around the slewing axis RX while being supported by the traveling body 103.

[0013] The vehicle 103 has a pair of tracks 103C. The hydraulic excavator 100 moves as the tracks 103C rotate.

[0014] The work machine 101 has a boom 106 connected to a slewing body 102, an arm 107 connected to the tip of the boom 106, and a bucket 108 connected to the tip of the arm 107. The bucket 108 has a cutting edge 109.

[0015] The boom 106 is rotatable relative to the slewing body 102 about the boom axis AX1. The arm 107 is rotatable relative to the boom 106 about the arm axis AX2. The bucket 108 is rotatable relative to the arm 107 about the bucket axis AX3, the tilt axis AX4, and the rotate axis AX5, respectively. The boom axis AX1, the arm axis AX2, and the bucket axis AX3 are parallel to the Y axis. The tilt axis AX4 is perpendicular to the bucket axis AX3. The rotate axis AX5 is perpendicular to both the bucket axis AX3 and the tilt axis AX4. The slewing axis RX is parallel to the Z axis.

[0016] The X-axis direction is the front-to-back direction of the slewing body 102. The Y-axis direction is the vehicle width direction of the slewing body 102. The Z-axis direction is the up-and-down direction of the slewing body 102. The direction in which the work implement 101 is located relative to the slewing body 102 is forward.

[0017] The working machine 101 is operated by the power generated by the hydraulic cylinder 110. The hydraulic cylinder 110 is driven based on the hydraulic oil supplied from a hydraulic pump (not shown). The hydraulic cylinder 110 includes a boom cylinder 111, an arm cylinder 112, and a bucket cylinder 113. The boom cylinder 111 operates the boom 106. The boom cylinder 111 generates power to rotate the boom 106 around the boom axis AX1. The arm cylinder 112 operates the arm 107. The arm cylinder 112 generates power to rotate the arm 107 around the arm axis AX2. The bucket cylinder 113 operates the bucket 108. The bucket cylinder 113 generates power to rotate the bucket 108 around the bucket axis AX3.

[0018] <Peripheral Monitoring System for Construction Machinery> FIG. 2 is a block diagram showing a peripheral monitoring system for a construction machine according to an embodiment. The display system 10 includes a camera 11 which is a first detection unit, a radar 13 which is a second detection unit, a peripheral monitoring monitor 15 which is a display unit, a buzzer 17, a peripheral monitoring controller 20, and a controller 40.

[0019] The camera 11 is a plurality of camera groups that photograph the surrounding situation of the hydraulic excavator 100. The camera 11 detects people located around the hydraulic excavator 100. The number of cameras 11 is not particularly limited. In the embodiment, the camera 11 includes a camera 11A, a camera 11B, a camera 11C, and a camera 11D.

[0020] The camera 11A photographs the front of the hydraulic excavator 100. The camera 11A is arranged facing forward on the upper part of the revolving body 102 of the hydraulic excavator 100. The camera 11A outputs the photographed image to the image processing unit 21 of the peripheral monitoring controller 20.

[0021] Camera 11B captures the right side of the hydraulic excavator 100. Camera 11B is positioned on the upper part of the slewing body 102 of the hydraulic excavator 100, facing to the right. Camera 11B outputs the captured image to the image processing unit 21 of the surrounding monitoring controller 20.

[0022] Camera 11C captures the left side of the hydraulic excavator 100. Camera 11C is positioned on the upper part of the slewing body 102 of the hydraulic excavator 100, facing left. Camera 11C outputs the captured image to the image processing unit 21 of the surrounding monitoring controller 20.

[0023] Camera 11D photographs the rear of the hydraulic excavator 100. Camera 11D is positioned on top of the rotating body 102 of the hydraulic excavator 100, facing rear. Camera 11D outputs the captured image to the image processing unit 21 of the surrounding monitoring controller 20.

[0024] Radar 13 is a group of radars that detect the surrounding conditions of the hydraulic excavator 100. Radar 13 detects people or objects that are obstacles located around the hydraulic excavator 100. Radar 13 has a shorter detection time compared to camera 11. The number of radars 13 is not particularly limited. In this embodiment, radar 13 comprises radar 13A, radar 13B, radar 13C, and radar 13D.

[0025] Radar 13A detects an obstacle to the right front of the hydraulic excavator 100. Radar 13A is positioned on the upper part of the slewing body 102 of the hydraulic excavator 100, facing forward to the right. Radar 13A outputs detection data of the detected obstacle to the obstacle processing unit 27 of the surrounding monitoring controller 20.

[0026] Radar 13B detects an obstacle to the right rear of the hydraulic excavator 100. Radar 13B is positioned on the upper part of the slewing body 102 of the hydraulic excavator 100, facing to the right rear. Radar 13B outputs detection data of the detected obstacle to the obstacle processing unit 27 of the surrounding monitoring controller 20.

[0027] Radar 13C detects obstacles behind the hydraulic excavator 100. Radar 13C is positioned facing rearward on the upper part of the slewing body 102 of the hydraulic excavator 100. Radar 13C outputs detection data of detected obstacles to the obstacle processing unit 27 of the surrounding monitoring controller 20.

[0028] Radar 13D detects an obstacle to the left rear of the hydraulic excavator 100. Radar 13D is positioned on top of the rotating body 102 of the hydraulic excavator 100, facing the left rear. Radar 13D outputs detection data of the detected obstacle to the obstacle processing unit 27 of the surrounding monitoring controller 20.

[0029] The surrounding area monitor 15 is a monitor for monitoring the area around the hydraulic excavator 100. The surrounding area monitor 15 displays an overhead view image 200 (see Figure 3). The surrounding area monitor 15 is located, for example, inside the operator's cab of the hydraulic excavator 100.

[0030] <Peripheral monitoring controller> The peripheral monitoring controller 20 includes a numerical processing unit (processor) such as a CPU. The peripheral monitoring controller 20 is located on the hydraulic excavator 100. The peripheral monitoring controller 20 comprises an image processing unit 21, an obstacle processing unit 27, and a display control unit 29.

[0031] Figure 4 is a block diagram illustrating a computer system according to an embodiment. The peripheral monitoring controller 20 and the controller 40 (described later) include a computer system 1000. The computer system 1000 includes a processor 1001 such as a CPU, a main memory 1002 including non-volatile memory such as ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory), a storage 1003, and an interface 1004 including input / output circuits. The functions of the peripheral monitoring controller 20 and the controller 40 are stored as programs in the storage 1003. The processor 1001 reads the programs from the storage 1003, loads them into the main memory 1002, and executes the above-described processes according to the programs. The programs may be distributed to the computer system 1000 via a network.

[0032] The image processing unit 21 acquires images from the camera 11 and performs image processing. In this embodiment, the overhead image generation unit 22 acquires four images from cameras 11A, 11B, 11C, and 11D and generates an overhead image 200. The image processing unit 21 comprises the overhead image generation unit 22, the image synthesis unit 23, the first recognition unit 24, and the determination unit 25.

[0033] Figure 3 is a schematic diagram showing an example of an overhead view image and a single-camera image. The overhead view image generation unit 22 generates an overhead view image 200 based on multiple images acquired from camera 11. In this embodiment, the overhead view image generation unit 22 converts four images acquired from cameras 11A, 11B, 11C, and 11D into an image viewed from above. The overhead view image generation unit 22 converts the image into an image viewed from a predetermined virtual viewpoint located above the hydraulic excavator 100. More specifically, the overhead view image generation unit 22 performs an image conversion that projects from the virtual viewpoint above the hydraulic excavator 100 onto a predetermined virtual projection plane corresponding to the ground surface level. Subsequently, the overhead view image generation unit 22 extracts the converted images corresponding to each area of ​​the frame displaying the overhead view image 200 and synthesizes each converted image within the frame. The overhead view image 200 generated by the overhead view image generation unit 22 includes an icon image 210 corresponding to the plan view of the hydraulic excavator 100. The method for generating the overhead image 200 is not limited to this, and known methods can be used.

[0034] The display control unit 29 acquires multiple images from the camera 11 and generates a single-camera image. The multiple images may also be acquired from the image processing unit 21. For example, the display control unit 29 generates a display signal to display the single-camera image 220 of the rear of the hydraulic excavator 100, captured by the camera 11D, on the surrounding monitoring monitor 15. The rear single-camera image 220 displays guide lines 221 indicating a predetermined distance from the rear of the hydraulic excavator. Although the rear single-camera image 220 displays three guide lines 221A, 221B, and 221C, the number of guide lines 221 displayed is not limited to three; there may be two or fewer, or four or more.

[0035] The guideline 221 of the rear single-camera image 220 may be displayed at positions corresponding to the first guideline 213 and the second guideline 215 in the overhead view image 200. In this case, the operator can visually confirm, not only in the overhead view image 200 but also in the rear single-camera image 220, where the range for controlling at least one of the travel and rotation of the hydraulic excavator 100 is located.

[0036] The image synthesis unit 23 synthesizes an image onto the overhead view image 200. When obstacle information is input from the obstacle processing unit 27 (described later), the image synthesis unit 23 generates an image in which the obstacle information is synthesized onto the overhead view image 200.

[0037] The image synthesis unit 23 generates an image by compositing guide lines 211 onto the overhead view image 200. The image synthesis unit 23 composites guide lines 211, which indicate the range for controlling at least one of driving and turning, onto the overhead view image 200.

[0038] The guideline 211 indicates the range within which at least one of the travel and rotation of the hydraulic excavator 100 is controlled when an obstacle is detected around the hydraulic excavator 100. The guideline 211 is an image surrounding the hydraulic excavator 100. In this embodiment, the guideline 211 consists of a first guideline 213 and a second guideline 215.

[0039] The first guideline 213 is the range (stopping area) within which the movement and rotation of the hydraulic excavator 100 are controlled to stop when an obstacle is detected around the hydraulic excavator 100. On the side of the hydraulic excavator 100's body, the first guideline 213 indicates the stopping distance from the center of rotation of the body. In the example shown in Figure 3, the first guideline 213 is a straight line on the side of the hydraulic excavator 100's body. On the rear of the hydraulic excavator 100's body, the first guideline 213 indicates the stopping distance from the rear end of the body. In the example shown in Figure 3, the first guideline 213 is an arc on the rear of the hydraulic excavator 100's body.

[0040] The second guideline 215 is a range (deceleration area) that controls the deceleration of the hydraulic excavator 100 when an obstacle is detected around the hydraulic excavator 100. On the side of the hydraulic excavator 100's body, the second guideline 215 is a straight line indicating a predetermined distance outward from the first guideline 213 on the side of the body. On the rear of the hydraulic excavator 100's body, it is a straight line passing through a predetermined distance outward from the position of the first guideline 213 at the rear of the body that is furthest from the center of rotation.

[0041] The image synthesis unit 23 may overlay the location where a person or object has been detected onto the overhead image 200 based on obstacle information acquired from the determination unit 25, which will be described later. More specifically, if the determination unit 25, which will be described later, determines that a person or object has been detected, the image synthesis unit 23 may synthesize a marker image indicating the detection of a person or object onto the overhead image 200 based on the obstacle information.

[0042] The first recognition unit 24 recognizes a person from an image captured by the camera 11. In this embodiment, the first recognition unit 24 recognizes a person from images captured by cameras 11A, 11B, 11C, and 11D using a person recognition dictionary. The person recognition dictionary is, for example, a dictionary of feature quantities extracted from each of several known images containing a person. Examples of feature quantities include HOG (Histograms of Oriented Gradients) and CoHOG (Co-occurrence HOG). Known methods can be used to recognize a person from an image.

[0043] In this embodiment, the first recognition unit 24 uses a human recognition dictionary to calculate a value (score) indicating human-likeness from the image captured by the camera 11 and recognizes a person. A higher value indicating human-likeness indicates a higher probability that the person is human. The first recognition unit 24 recognizes a person as "human" if the value indicating human-likeness is greater than a first threshold. The first recognition unit 24 recognizes a person as "human-like" if the value indicating human-likeness is less than the first threshold and greater than a second threshold that is smaller than the first threshold. The first recognition unit 24 outputs the recognition result to the determination unit 25.

[0044] The recognition result of the first recognition unit 24 includes coordinates indicating the location where the person was recognized.

[0045] The determination unit 25 will be explained after the obstacle processing unit 27 has been described.

[0046] The obstacle processing unit 27 detects people and objects as obstacles from the detection data of the radar 13. The obstacle processing unit 27 includes a second recognition unit 28.

[0047] The second recognition unit 28 recognizes people and objects as obstacles from the detection data of the radar 13. In this embodiment, the second recognition unit 28 recognizes people and objects from the detection data of radars 13A, 13B, 13C, and 13D. A known method can be used to recognize people and objects from the detection data. The second recognition unit 28 outputs the recognition result to the determination unit 25 of the image processing unit 21.

[0048] The recognition result of the second recognition unit 28 includes coordinates indicating the location where a person or obstacle was recognized.

[0049] The determination unit 25 determines whether or not to detect a person based on the recognition result of the first recognition unit 24 and the recognition result of the second recognition unit 28. In this embodiment, the determination unit 25 determines whether or not to detect a person or an object based on the recognition result of the first recognition unit 24 and the recognition result of the second recognition unit 28.

[0050] The determination unit 25 determines that a person has been detected based on the recognition result from the image captured by the camera 11 if the value indicating human-likeness is greater than the first threshold, or if the value indicating human-likeness is less than the first threshold and greater than the second threshold which is less than the first threshold, and if the radar 13 has recognized an obstacle based on the recognition result from the detection data it has detected. More specifically, the determination unit 25 determines that a person has been detected if the recognition result from the first recognition unit 24 indicates that the value indicating human-likeness is greater than the first threshold, or the value indicating human-likeness is less than the first threshold and greater than the second threshold which is less than the first threshold, and if the second recognition unit 28 has recognized an obstacle.

[0051] If the determination unit 25 determines that a person or object has been detected, it sounds the buzzer 17 to notify the system. If the determination unit 25 determines that a person or object has been detected, it may also output obstacle information, such as the size and position (coordinates) of the detected person or object, to the image synthesis unit 23.

[0052] The determination unit 25 may determine that a person has been detected if, based on the recognition result from the image captured by the camera 11, the value indicating human-likeness is less than the first threshold and greater than the second threshold, and the radar 13 has recognized an obstacle from the recognition result from the detection data, and the coordinates where the person was recognized from the image from the camera 11 and the coordinates where the obstacle was recognized from the detection data from the radar 13 are within a predetermined range. More specifically, the determination unit 25 may determine that a person has been detected if, based on the recognition result from the first recognition unit 24, the value indicating human-likeness is less than the first threshold and greater than the second threshold, and the second recognition unit 28 has recognized an obstacle, and the coordinates where the person was recognized from the image captured by the camera 11 and the coordinates where the obstacle was recognized from the detection data from the radar 13 are within a predetermined range. The predetermined range is, for example, a radius of about 2m.

[0053] If the determination unit 25 determines that a person has been detected, and then detects an obstacle from the recognition results of the radar 13 within a predetermined distance range, it may determine that a person has been detected. More specifically, if the determination unit 25 determines that a person has been detected, and then the second recognition unit 28 detects an obstacle from the recognition results of the radar 13 within a predetermined distance range, it may determine that a person has been detected.

[0054] Within a predetermined distance, for example, is a radius of about 2 meters. Within a predetermined distance, for example, is the distance a person can travel in a very short time, such as about 1 second.

[0055] After the determination unit 25 determines that a person has been detected, it may maintain the determination result that a person has been detected for a predetermined period of time.

[0056] The display control unit 29 controls the display of various images on the peripheral monitoring monitor 15. The display control unit 29 generates a display signal to display the overhead image 200 input from the image synthesis unit 23 on the peripheral monitoring monitor 15.

[0057] If the determination unit 25 determines that a person or object has been detected, the display control unit 29 may display a marker image on the overhead image 200 indicating that a person or object has been detected.

[0058] If the determination unit 25 determines that a person or object has been detected, the display control unit 29 may, for example, display an indicator or pop-up indicating that a person or object has been detected.

[0059] <Controller> The controller 40 includes a numerical processing unit (processor) such as a CPU. The controller 40 is located in the hydraulic excavator 100. The controller 40 includes a work machine control unit 41.

[0060] The work machine control unit 41 outputs various control signals to control the hydraulic excavator 100. When the determination unit 25 of the surrounding monitoring controller 20 determines that a person has been detected, the work machine control unit 41 controls at least one of the travel and rotation of the hydraulic excavator 100. When the determination unit 25 determines that a person or object has been detected within the stopping area, the work machine control unit 41 controls the travel and rotation of the hydraulic excavator 100 to stop. When the determination unit 25 determines that a person or object has been detected within the deceleration area, the work machine control unit 41 controls the travel and rotation of the hydraulic excavator 100 to slow down.

[0061] The work machine control unit 41 controls the hydraulic excavator 100 by changing the travel speed from "high" to "low," changing the travel speed from "medium" to "low," stopping travel, or stopping slewing.

[0062] <An example of a surrounding area monitoring method> Figure 5 is an example of a flowchart showing a method for monitoring the surroundings of a work machine according to an embodiment. When the hydraulic excavator 100 is turned on, the display system 10 of the hydraulic excavator 100 is activated. When the display system 10 of the hydraulic excavator 100 is activated, the processing shown in the flowchart of Figure 5 is started. During the execution of the processing shown in the flowchart of Figure 5, recognition processing by the first recognition unit 24 and the second recognition unit 28 is continuously performed.

[0063] The peripheral monitoring controller 20 determines, using the determination unit 25, whether the human-likeness is greater than the first threshold (step ST11). More specifically, the peripheral monitoring controller 20 determines, using the determination unit 25, whether the recognition result of the first recognition unit 24 indicates that the human-likeness is greater than the first threshold. If the peripheral monitoring controller 20 determines, using the determination unit 25, that the human-likeness is greater than the first threshold (Yes in step ST11), it proceeds to step ST14. If the peripheral monitoring controller 20 does not determine, using the determination unit 25, that the human-likeness is greater than the first threshold (No in step ST11), it proceeds to step ST12.

[0064] If the Peripheral Monitoring Controller 20 does not determine that the human-likeness is greater than the first threshold (No in step ST11), the determination unit 25 determines whether the human-likeness is greater than the second threshold. More specifically, the determination unit 25 determines whether the recognition result of the first recognition unit 24 indicates that the human-likeness is greater than the second threshold. If the determination unit 25 determines that the human-likeness is greater than the second threshold (Yes in step ST12), the Peripheral Monitoring Controller 20 proceeds to step ST13. If the determination unit 25 does not determine that the human-likeness is greater than the second threshold (No in step ST12), the Peripheral Monitoring Controller 20 terminates the processing of this flowchart.

[0065] If the surrounding monitoring controller 20 determines that the human-likeness is greater than the second threshold (Yes in step ST12), the determination unit 25 determines whether or not the radar has detected an obstacle (step ST13). More specifically, the surrounding monitoring controller 20 determines, using the determination unit 25, whether or not the second recognition unit 28 has recognized an obstacle. If the surrounding monitoring controller 20 determines, using the determination unit 25, that the radar has detected an obstacle (Yes in step ST13), it proceeds to step ST14. If the surrounding monitoring controller 20 does not determine, using the determination unit 25, that the radar has detected an obstacle (No in step ST13), it terminates the processing of this flowchart.

[0066] The peripheral monitoring controller 20 determines, based on the determination unit 25, that a person has been detected (step ST14).

[0067] <Other examples of perimeter monitoring methods> Figure 6 is another example of a flowchart showing a method for monitoring the surroundings of a work machine according to the embodiment. The processing in steps ST21 to ST23 and step ST25 is the same as the processing in steps ST11 to ST13 and step ST14 of the flowchart in Figure 5.

[0068] If the radar determines that an obstacle has been detected (Yes in step ST23), the surrounding monitoring controller 20 uses the determination unit 25 to determine whether the coordinates are within a predetermined range (step ST24). More specifically, the surrounding monitoring controller 20 uses the determination unit 25 to determine whether the coordinates where a person was recognized from the image of the camera 11 and the coordinates where an obstacle was recognized from the detection data of the radar 13 are within a predetermined range. If the surrounding monitoring controller 20 determines that the coordinates are within a predetermined range (Yes in step ST24), it proceeds to step ST25. If the surrounding monitoring controller 20 does not determine that the coordinates are within a predetermined range (No in step ST24), it proceeds to step ST26.

[0069] If the determination unit 25 does not determine that the human-likeness is greater than the second threshold (No in step ST22), or if it does not determine that an obstacle has been detected by the radar (No in step ST23), the surrounding monitoring controller 20 determines that no person or object has been detected (step ST27).

[0070] The peripheral monitoring controller 20 determines that an object has been detected by the determination unit 25 (step ST26).

[0071] <Further examples of peripheral monitoring methods> In the flowchart in Figure 6, it is explained that if the human-likeness is not determined to be greater than the second threshold (No in step ST22), the process proceeds to step ST27. However, the following is also possible: If the result in step ST22 is No, the process proceeds to step ST31, which is not shown.

[0072] If the surrounding monitoring controller 20 does not determine that the human-likeness is greater than the second threshold (No in step ST22), the determination unit 25 determines whether or not the radar has detected an obstacle (step ST31). More specifically, the surrounding monitoring controller 20 determines whether or not the second recognition unit 28 has recognized an obstacle using the determination unit 25. If the surrounding monitoring controller 20 determines that the radar has detected an obstacle using the determination unit 25 (Yes in step ST31), it proceeds to step ST32. If the surrounding monitoring controller 20 does not determine that the radar has detected an obstacle using the determination unit 25 (No in step ST31), it proceeds to step ST27.

[0073] The peripheral monitoring controller 20 determines, based on the determination unit 25, that an object has been detected (step ST32). The peripheral monitoring controller 20 then terminates the processing shown in this flowchart.

[0074] <Effects> As described above, in this embodiment, based on the detection results from the image captured by the camera 11, if the value indicating human characteristics is greater than a first threshold, or if the value indicating human characteristics is less than the first threshold and greater than a second threshold that is less than the first threshold, and if the radar 13 recognizes an obstacle from the detection data it has detected, it can be determined that a person has been detected. According to this embodiment, people can be detected more appropriately.

[0075] In this embodiment, if it is determined that a person or object has been detected, a marker image indicating that a person or object has been detected can be displayed.

[0076] In this embodiment, if it is determined that a person or object has been detected, an indicator or pop-up can be displayed.

[0077] In this embodiment, if it is determined that a person or object has been detected, the work machine can be controlled. In this embodiment, the control of the work machine can include changing the travel speed from high to low, changing the travel speed from medium to low, stopping travel, or stopping turning.

[0078] In this embodiment, based on the detection results from the image captured by the camera 11, if the value indicating human characteristics is less than a first threshold and greater than a second threshold, and if an obstacle is detected from the detection data detected by the radar 13, and the coordinates recognized from the image captured by the camera 11 and the coordinates recognized from the detection data detected by the radar 13 are within a predetermined range, then it can be determined that a person has been detected. According to this embodiment, people can be detected more appropriately.

[0079] In this embodiment, after determining that a person has been detected, the second recognition unit 28 can determine that a person has been detected if it recognizes an obstacle from the recognition results of the radar 13 within a predetermined distance. In this embodiment, after determining that a person has been detected, the second recognition unit 28 can determine that a person has been detected if it recognizes an obstacle from the recognition results of the radar 13 within a predetermined distance, regardless of the detection results from the image captured by the camera 11. In this embodiment, the period for recognizing a person from the image captured by the camera 11 is longer than the period for recognizing a person from the detection data of the radar 13. Also, in this embodiment, there is a possibility of false detection in the recognition process for recognizing a person from the image captured by the camera 11. Therefore, by determining the detection of a person in this way, false detections can be reduced. According to this embodiment, a person can be detected more appropriately.

[0080] In this embodiment, human detection can be continued for a predetermined period. According to this embodiment, even if there is a short period during which human detection is not possible, for example, processing can continue as if human detection had been continuously performed. According to this embodiment, it is possible to prevent the display on the surrounding monitoring monitor 15 or the control by the work machine control unit 41 from unintentionally returning to a state where human detection is not performed.

[0081] In the embodiments described above, the work machine is not limited to a hydraulic excavator. The work machine can be a dump truck, a wheel loader, or a display system for other work machines.

[0082] In the embodiments described above, the surrounding monitoring monitor 15 was described as being located in the operator's cab of the hydraulic excavator 100, but it is not limited to this. If the hydraulic excavator 100 is remotely operated, the surrounding monitoring monitor 15 may be located in the remote control room.

[0083] In the embodiments described above, the first detection unit was described as a camera, but it is not limited to this. The first detection unit may be, for example, LiDAR (Laser Imaging Detection and Ranging).

[0084] In the embodiments described above, the second detection unit was described as a radar, but it is not limited to this. The second detection unit may be, for example, a camera. The second detection unit may be a detection unit using a camera that has a short processing time required to detect a person or object. [Explanation of Symbols]

[0085] 10...Display system, 11...Camera (first detection unit), 13...Radar (second detection unit), 15...Surroundings monitoring (display unit), 17...Buzzer, 20...Surroundings monitoring controller, 21...Image processing unit, 22...Overview image generation unit, 23...Image synthesis unit, 24...First recognition unit, 25...Determination unit, 27...Obstacle processing unit, 28...Second recognition unit, 29...Display control unit, 40...Controller, 41...Work machine control unit, 100...Hydraulic excavator (work machine) Industrial machinery), 101...working equipment, 102...slewing body, 103...traveling body, 103C...track, 106...boom, 107...arm, 108...bucket, 109...cutting edge, 110...hydraulic cylinder, 111...boom cylinder, 112...arm cylinder, 113...bucket cylinder, 200...overhead view, AX1...boom axis, AX2...arm axis, AX3...bucket axis, AX4...tilt axis, AX5...rotate axis, RX...slewing axis.

Claims

1. A first detection unit acquires the surrounding conditions of the work machine, A second detection unit that acquires the surrounding conditions of the aforementioned work machine, Based on the detection results of the first detection unit, when the value indicating human-likeness is greater than the first threshold, Alternatively, if the value indicating human-likeness is smaller than the first threshold and larger than the second threshold which is smaller than the first threshold, and the second detection unit has detected an obstacle, the determination unit determines that a person has been detected. A peripheral monitoring system for work machinery equipped with the following features.

2. If the determination unit determines that a person or object has been detected, the display control unit displays a marker image indicating that a person or object has been detected. A peripheral monitoring system for a work machine according to claim 1, comprising:

3. If the determination unit determines that a person or object has been detected, the display control unit will display an indicator or pop-up. A peripheral monitoring system for a work machine according to claim 1, comprising:

4. If the determination unit determines that it has detected a person or object, the work machine control unit controls the work machine. A peripheral monitoring system for a work machine according to claim 1, comprising:

5. The aforementioned work machine control unit controls the work machine by changing the travel speed from high to low, changing the travel speed from medium to low, stopping travel, or stopping turning. A peripheral monitoring system for a work machine according to claim 4.

6. The determination unit determines that a person has been detected if the value indicating human-likeness is less than the first threshold and greater than the second threshold, and the second detection unit has detected an obstacle, and the coordinates detected by the first detection unit and the coordinates detected by the second detection unit are within a predetermined range. A peripheral monitoring system for a work machine as described in claim 1.

7. After determining that a person has been detected, the determination unit determines that a person has been detected if it has detected a person from the detection results of the second detection unit within a predetermined distance range. A peripheral monitoring system for a work machine as described in claim 1.

8. The determination unit continues to detect a person for a predetermined period of time. A peripheral monitoring system for a work machine according to claim 6.

9. The second detection unit has a shorter processing time required to detect a person or object than the first detection unit. A peripheral monitoring system for a work machine as described in claim 1.

10. The first detection unit is a camera, The aforementioned second detection unit is a radar, A peripheral monitoring system for a work machine as described in claim 1.

11. A first detection unit acquires the surrounding conditions of the work machine, A second detection unit that acquires the surrounding conditions of the aforementioned work machine, A method for monitoring the surroundings of a work machine, comprising a controller, Based on the detection results of the first detection unit, when the value indicating human-likeness is greater than the first threshold, Alternatively, if the value indicating human-likeness is smaller than the first threshold and larger than the second threshold which is smaller than the first threshold, and the second detection unit has detected an obstacle, then a person is detected. A method for monitoring the surroundings of a work machine while it is performing an action.

12. When a person or object is detected, a marker image indicating that a person or object has been detected will be displayed. A method for monitoring the surroundings of a work machine according to claim 11.

13. If a person or object is detected, an indicator or pop-up will be displayed. A method for monitoring the surroundings of a work machine according to claim 11.

14. When a person or object is detected, the work machine is controlled. A method for monitoring the surroundings of a work machine according to claim 11.

15. Control of the work machine includes changing the travel speed from high to low, changing the travel speed from medium to low, stopping travel, or stopping turning. A method for monitoring the surroundings of a work machine according to claim 14.

16. If the value indicating human-likeness is less than the first threshold and greater than the second threshold, and the second detection unit has detected an obstacle, and the coordinates detected by the first detection unit and the coordinates detected by the second detection unit are within a predetermined range, then a person is detected. A method for monitoring the surroundings of a work machine according to claim 11.

17. After detecting a person, if the second detection unit detects a person within a predetermined range, it will detect the person. A method for monitoring the surroundings of a work machine according to claim 11.

18. Continue detecting people for a predetermined period of time. A method for monitoring the surroundings of a work machine according to claim 16.

19. The second detection unit has a shorter processing time required to detect a person or object than the first detection unit. A method for monitoring the surroundings of a work machine according to claim 11.

20. The first detection unit is a camera, The aforementioned second detection unit is a radar, A method for monitoring the surroundings of a work machine according to claim 11.

Images