Surrounding detection device for work machines

Abstract

To reduce a processing load required to detect an object and shorten processing time.SOLUTION: A surroundings detection device 20 of a work machine 1 includes an imaging device 23 and a response processing unit 90. The imaging device 23 is provided in the work machine 1 and photographs an object A around the work machine 1 to obtain a captured image Im. The response processing unit 90 executes a predetermined response process when the object A enters inside a detection area D in the captured image Im. The captured image Im has the detection area D and a non-detection area C. In the captured image Im, when the side closer to the work machine 1 is defined as the machine side (Y2), the detection area D is located closer to the machine side (Y2) than the non-detection area C.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a surroundings detection device for a work machine that detects objects around the work machine. [Background technology]

[0002] For example, Patent Document 1 discloses a conventional surroundings detection device. In the technology disclosed in the document, an imaging device captures an object. Then, the coordinates of the object captured in the captured image are converted into virtual plane coordinates (see FIG. 5 in the document, etc.). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-102604 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the process of converting the coordinates of an object captured in a captured image into virtual plane coordinates is complicated, and there are problems in that the processing load is high and the processing speed is slow. In addition, when an object enters a detection area around a work machine, a response process such as a warning or operation restriction may be performed. In this case, converting the coordinates of an object captured in a captured image into virtual plane coordinates may delay the start of the response process.

[0005] It is therefore an object of the present invention to provide a surrounding detection device for a work machine that can reduce the processing load required for detecting an object and shorten the processing time. [Means for solving the problem]

[0006] The surrounding detection device for a work machine includes an imaging device, an object search unit, a detection area setting unit, and a response processing unit. The imaging device is provided on the work machine, and captures an image of an object around the work machine to obtain a captured image. The object search unit searches for the position of the object in the captured image. The detection area setting unit sets a detection area in the captured image that is an area in which the object is detected. The response processing unit executes a predetermined response process when the object enters inside the detection area in the captured image. The detection area setting unit sets a boundary line that is a boundary of the detection area in the captured image. In the captured image, the side close to the work machine is the machine side, the side far from the work machine is the anti-machine side, the direction connecting the machine side and the anti-machine side is the front direction, and the direction perpendicular to the front direction is the front orthogonal direction. The boundary line extends in the front orthogonal direction and has a fixed shape. Effect of the Invention

[0007] With the above configuration, it is possible to reduce the processing load required for detecting an object, and shorten the processing time. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing a working machine 1 as viewed from above. [Diagram 2] 2 is a diagram showing a surroundings detection device 20 of the work machine 1 shown in FIG. [Diagram 3] 3 is a flowchart showing the operation of a control unit 30 shown in FIG. 2. [Figure 4] 3 is a diagram showing a captured image Im acquired by the imaging device 23 shown in FIG. 2. FIG. [Diagram 5] 4 in the case where a plurality of boundary lines B shown in FIG. 4 are set. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The surroundings detection device 20 (see FIG. 2) of the work machine 1 (see FIG. 1) will be described with reference to FIGS. 1 to 5.

[0010] 1, the work machine 1 is a machine that performs work, for example a construction machine that performs construction work, and may be, for example, a shovel (e.g., a hydraulic shovel) or a crane. In the following, a case where the work machine 1 is a shovel will be described. The work machine 1 includes a lower traveling body 11, an upper rotating body 13, and an attachment 15.

[0011] The lower traveling body 11 allows the working machine 1 to travel. The lower traveling body 11 is equipped with, for example, a crawler, and may be equipped with a blade.

[0012] The upper rotating body 13 is mounted so as to be rotatable relative to the lower traveling body 11. The upper rotating body 13 includes a cab 13a and a counterweight 13b. The cab 13a is a section where an operator operates the work machine 1 when the operator gets on the work machine 1 and operates the work machine 1. The counterweight 13b is a weight for balancing the work machine 1 in the fore-and-aft direction.

[0013] The attachment 15 is a part that performs work, and includes, for example, a boom 15a, an arm 15b, and a tip attachment 15c. The boom 15a is attached to the upper rotating body 13 so that it can be raised and lowered. The arm 15b is attached rotatably to the boom 15a. The tip attachment 15c is provided at the tip of the attachment 15, and is attached rotatably to the arm 15b. The tip attachment 15c may be, for example, a bucket for scooping up soil and sand, a device for clamping objects (such as a grapple), or a device for crushing, excavating, etc. (such as a breaker).

[0014] The surroundings detection device 20 is a device that detects objects (target A) around the work machine 1. The surroundings detection device 20 detects the target A based on a captured image Im (see FIG. 4). The target A may be a person (worker) around the work machine 1, or an obstacle other than a person around the work machine 1. As shown in FIG. 2, the surroundings detection device 20 includes an operation unit 21, an imaging device 23, a control unit 30, a captured image display unit 80, and a response processing unit 90.

[0015] The operation unit 21 is a part that is operated by an operator and is used to operate the work machine 1 (see FIG. 1). The operation unit 21 is provided in the driver's cab 13a (see FIG. 1). When the work machine 1 is remotely operated, the operation unit 21 may be provided in a position away from the work machine 1 (see FIG. 1). The operation unit 21 includes a travel operation unit 21a and a boundary line adjustment input device 21c. The travel operation unit 21a is a part that is used to operate the travel (more specifically, forward travel and reverse travel) of the lower travel structure 11 (see FIG. 1). The travel operation unit 21a is a lever (operation lever). The boundary line adjustment input device 21c is a device that is used to adjust the position of a boundary line B (see FIG. 4) that will be described later. The boundary line adjustment input device 21c includes, for example, a jog dial 21c1 and a switch 21c2 (for example, a push switch). The jog dial 21c1 and the switch 21c2 may be configured integrally or separately.

[0016] As shown in FIG. 1, the imaging device 23 photographs (takes an image) of the object A and acquires a captured image Im (see FIG. 4). The imaging device 23 is a surroundings monitoring imaging device that monitors the surroundings of the work machine 1. The imaging device 23 is attached and fixed to the upper rotating body 13. It is preferable that the imaging range of the imaging device 23 covers the blind spot of the operator in the cab 13a. The number of imaging devices 23 may be one or more. The imaging device 23 is provided, for example, in the rear part of the upper rotating body 13 (for example, the counterweight 13b or its vicinity), the right part, and the left part. Note that, when the work machine 1 is remotely operated, the imaging device 23 may be provided in the front part of the upper rotating body 13 (for example, the cab 13a). The imaging device 23 is capable of photographing a wide range, and has, for example, a fisheye lens.

[0017] The captured image Im (see FIG. 4) acquired by the imaging device 23 may be a two-dimensional image (more specifically, a video) or a three-dimensional image (for example, an image captured by a stereo camera or the like). In the captured image Im shown in FIG. 4, the side closer to the work machine 1 is the image lower side Y2 (machine side), and the side farther from the work machine 1 is the image upper side Y1 (opposite machine side). In the captured image Im, the direction connecting the image upper side Y1 and the image lower side Y2 is the image up-down direction Y (front direction). In the captured image Im, the direction perpendicular to the image up-down direction Y is the image left-right direction X (direction perpendicular to the front). Note that "up", "down", "left", and "right" are merely terms used for convenience of explanation. For example, the imaging device 23 may be rotated or flipped.

[0018] The control unit 30 (see FIG. 2) performs input and output of information, calculations (determination, setting, etc.), and storage of information. The control unit 30 shown in FIG. 2 may be provided in the upper rotating body 13 (see FIG. 1) or outside the upper rotating body 13 (e.g., a server, etc.). For example, some of the components of the control unit 30 may be provided in the upper rotating body 13, and the other parts may be provided outside the upper rotating body 13. The control unit 30 includes an image correction unit 31, an object search unit 41, a search range limiting unit 43, and an elevation distortion area setting unit 45 (first elevation distortion area setting unit, second elevation distortion area setting unit). The control unit 30 includes a detection area setting unit 51, a boundary adjustment unit 52, a boundary limit position setting unit 53, a determination unit 61, a response processing control unit 63, and a display control unit 65.

[0019] The image correction unit 31 corrects the captured image Im acquired by the imaging device 23. The image correction unit 31 corrects distortion (image distortion) of the captured image Im. Any method may be used for distortion correction by the image correction unit 31. For example, the image correction unit 31 may reduce distortion by expanding or contracting the captured image Im. For example, the image correction unit 31 may remove parts of the captured image Im that are highly distorted while leaving parts with less distortion. The captured image Im corrected by the image correction unit 31 is also referred to as the "captured image Im".

[0020] The object search unit 41 searches for the position of the object A in the captured image Im shown in FIG. 4. More specifically, the object search unit 41 (see FIG. 2) automatically determines (recognizes, identifies) whether an object captured in the captured image Im is the object A by image processing. The object search unit 41 performs this determination based on the characteristics (e.g., shape, color, etc.) of the object captured in the captured image Im. The object search unit 41 also calculates the position (coordinates Ax, Ay) of the object A at the closest position to the work machine 1 in the captured image Im. For example, the object search unit 41 calculates the coordinates of the lower Y2 portion of the image of the object A (e.g., the feet of the worker, etc.). It is not necessary for the object search unit 41 to set the point of the object A that is strictly closest to the work machine 1 (closest point) as the coordinates (Ax, Ay) of the object A. The object coordinate calculation unit 41a may determine a position of the object A near the closest point to the work machine 1 as the coordinates (Ax, Ay) of the object A. It is preferable that the object coordinate calculation unit 41a determine the closest point of the object A to the work machine 1 as the coordinates (Ax, Ay) of the object A.

[0021] The search range limiting unit 43 (see FIG. 2) may set a search upper limit position Ea (search counter-machine side limit position) or a search lower limit position Eb (search machine side limit position) for the captured image Im. The search upper limit position Ea may be set in advance in the search range limiting unit 43, or may be set manually or automatically (similar to the search lower limit position Eb). When both the search upper limit position Ea and the search lower limit position Eb are set, the search lower limit position Eb is set on the lower side Y2 of the image than the search upper limit position Ea.

[0022] [Example 1] When the search range limiting unit 43 sets the search upper limit position Ea, the search range of the object A by the object searching unit 41 (see FIG. 2) is within the area Y2 below the search upper limit position Ea in the image. The above "within the area" does not have to be the entire area, but at least a part of the area (same below). The object searching unit 41 does not search for the object A in the area Y1 above the search upper limit position Ea in the image. [Example 2] When the search range limiting unit 43 sets the search lower limit position Eb, the search range of the object A by the object searching unit 41 (see FIG. 2) is within the area Y1 above the search lower limit position Eb in the image. The object searching unit 41 does not search for the object A in the area Y2 below the search lower limit position Eb in the image. [Example 3] When the search range limiting unit 43 sets the search upper limit position Ea and the search lower limit position Eb, the search range of the object A by the object searching unit 41 (see FIG. 2) is within the area between the search upper limit position Ea and the search lower limit position Eb. The object searching unit 41 does not search for the object A in the area Y1 above the search upper limit position Ea. In each of the above [Example 1], [Example 2], and [Example 3], the object searching unit 41 does not search for the object A in a part of the area of ​​the captured image Im. Therefore, compared to the case where the object searching unit 41 searches the entire captured image Im, the search range of the object searching unit 41 is limited, the processing load is reduced, and the processing time is shortened. The search upper limit position Ea and the search lower limit position Eb are set based on the necessity of searching the object A, for example.

[0023] The search upper limit position Ea is set, for example, as follows. The sky may be reflected (contained) in the upper Y1 portion of the captured image Im. If the object A is detected up to the portion where the sky is reflected, this may cause overdetection and may also cause erroneous detection. Therefore, the search upper limit position Ea is set based on the portion where the sky is expected to be reflected. For example, the search upper limit position Ea is set so that the portion where the sky is expected to be reflected is on the upper Y1 side of the image than the search upper limit position Ea.

[0024] The search lower limit position Eb is set, for example, as follows. A part of the upper rotating body 13 (see FIG. 1) may appear in the lower Y2 portion of the captured image Im. In this case, the target A will not appear in the portion where the upper rotating body 13 appears. Therefore, the search range limiting unit 43 sets the search lower limit position Eb based on the portion where the upper rotating body 13 is expected to appear. For example, the search range limiting unit 43 sets the search lower limit position Eb so that the portion where the upper rotating body 13 is expected to appear is located on the lower Y2 side of the image than the search lower limit position Eb.

[0025] The high-low distortion area setting unit 45 (see FIG. 2) (first high-low distortion area setting unit, second high-low distortion area setting unit) sets a high distortion area Fb (first high distortion area, second high distortion area) and a low distortion area Fa (first low distortion area, second low distortion area) in the captured image Im. The high distortion area Fb may be set in advance in the high-low distortion area setting unit 45, or may be set manually or automatically (similar to the low distortion area Fa). The low distortion area Fa is an area where image distortion is smaller than that of the high distortion area Fb.

[0026] The elevation distortion area setting unit 45 sets the high distortion area Fb and the low distortion area Fa, for example, as follows. The upper Y1 part and the lower Y2 part of the captured image Im may have larger distortion than the central part in the vertical direction Y of the image. Therefore, the upper Y1 part and the lower Y2 part of the captured image Im are set as the high distortion area Fb, and the part of the captured image Im other than the high distortion area Fb is set as the low distortion area Fa. As described above, since it is expected that the sky will be reflected in the upper Y1 part of the captured image Im, no problem will occur even if the object A is not searched for. Therefore, the search range of the object A by the object search unit 41 (see FIG. 2) is within the low distortion area Fa and within the high distortion area Fb in the lower Y2 part of the image. The object search unit 41 does not search for the object A in the high distortion area Fb in the upper Y1 part of the image. Therefore, compared to when the object searching unit 41 searches the entire captured image Im, the search range of the object searching unit 41 is limited, the processing load is reduced, and the processing time is shortened. On the other hand, the object searching unit 41 searches for the object A in the high distortion region Fb in the lower Y2 portion of the image. Therefore, the object A approaching the work machine 1 can be appropriately detected.

[0027] For example, a case may be considered in which a search upper limit position Ea, a search lower limit position Eb, a high distortion region Fb, and a low distortion region Fa are set. In this case, the object search unit 41 (see FIG. 2) searches for the object A in the region between the search upper limit position Ea and the search lower limit position Eb and in the low distortion region Fa, and does not search in other regions. Also, the image upper side Y1 part and the image lower side Y2 part of the captured image Im are the high distortion region Fb, and the center part in the image vertical direction Y is the low distortion region Fa, but this is merely an example. Depending on the angle of view and arrangement of the imaging device 23, the type of correction performed by the image correction unit 31, and the like, it is possible to consider various parts of the captured image Im that become the high distortion region Fb and the low distortion region Fa.

[0028] The detection area setting unit 51 (see FIG. 2) sets a detection area D. The detection area D is an area in the captured image Im where an object A is detected, and is an area for determining whether or not a predetermined response process is to be performed by the response processing unit 90 (see FIG. 2). The area of ​​the captured image Im other than the detection area D is set as a non-detection area C.

[0029] This detection area setting unit 51 sets a boundary line B that is a boundary of the detection area D in the captured image Im. Only one boundary line B may be set, or multiple boundary lines B may be set. The boundary line B includes at least a line that is a boundary between the detection area D and the non-detection area C. When multiple boundary lines B are set, the boundary line B on the side farthest from the work machine 1 (upper side Y1 of the image) becomes the boundary between the detection area D and the non-detection area C (the limit area of ​​the detection area D). The boundary line B may include a line that separates multiple detection areas D. Specifically, the boundary line B may include a line that separates the detection area D1 and the detection area D2 shown in FIG. 5. As shown in FIG. 4, it is preferable that the boundary line B is set so that the processing load on the control unit 30 (see FIG. 2) is as light as possible. Specifically, the boundary line B is a line that extends in the left-right direction X of the image and has a constant shape (a shape that does not change). The boundary line B is preferably substantially linear in the captured image Im, and more preferably linear. The above-mentioned "substantially straight" may be, for example, a curved line that follows a part of the upper rotating body 13 (see FIG. 1) (such as the rear end of the counterweight 13b (see FIG. 1)) that appears in the captured image Im.

[0030] The boundary line B is preferably set in the captured image Im corrected by the image correction unit 31 (see FIG. 2). The reason is as follows. In a portion of the captured image Im where distortion is large, even if the position of the object A in the image vertical direction Y is the same, the actual distance from the work machine 1 to the object A varies greatly depending on the position of the object A in the image horizontal direction X. Therefore, if a linear or approximately linear boundary line B is set in a portion of the captured image Im where distortion is large, even if the actual distance from the work machine 1 to the object A is the same, the object A may or may not enter the detection area D depending on the position of the object A in the image horizontal direction X. Then, the actual distance between the object A and the work machine 1 when the response processing by the response processing unit 90 (see FIG. 2) is started (the response processing start distance) changes depending on the position of the object A in the image horizontal direction X of the captured image Im. Therefore, it becomes difficult for the operator of the work machine 1 to grasp how close the object A needs to be to the work machine 1 before the response processing is started (the operator has difficulty in grasping the sense of distance). On the other hand, in this embodiment, a substantially straight or straight boundary line B is set on the captured image Im whose distortion has been corrected by the image correction unit 31 (see FIG. 2). Therefore, the correspondence processing start distance is constant or substantially constant regardless of the position of the object A in the left-right direction X of the image. This makes it easy for the operator to get a sense of distance. In addition, since the correspondence processing start distance is constant or substantially constant, the correspondence processing is performed appropriately.

[0031] The detection area setting unit 51 (see FIG. 2) sets the boundary line B within the low distortion area Fa for the same reason as above. This makes the response processing start distance approximately constant regardless of the position of the object A in the left-right direction X of the image, making it easier for the operator to get a sense of distance. As described above, it is assumed that the sky is reflected in the upper Y1 part of the captured image Im, and the lower traveling structure 11 is reflected in the lower Y2 part of the captured image Im. Therefore, if the boundary line B is set within the low distortion area Fa at the center of the image in the up-down direction Y of the captured image Im, the object A can be properly detected. Furthermore, since the boundary line B is set in the low distortion area Fa, the object A can be detected earlier than when the boundary line B is set in the high distortion area Fb on the work machine 1 side (lower Y2 of the image).

[0032] (First high and low distortion area setting section, second high and low distortion area setting section, etc.) Here, the height-low distortion area setting unit 45 is both the "first height-low distortion area setting unit" and the "second height-low distortion area setting unit." The "first height-low distortion area setting unit" sets a low distortion area Fa (i.e., the first low distortion area) where the boundary line B may be set, and a high distortion area Fb (i.e., the first high distortion area) where the boundary line B is not set. The "second height-low distortion area setting unit" sets a low distortion area Fa (i.e., the second low distortion area) where the object searching unit 41 may perform a search, and a high distortion area Fb on the lower side Y2 of the image (i.e., the second high distortion area on the lower side Y2 of the image). In addition, the "second height-low distortion area setting unit" sets a high distortion area Fb on the upper side Y1 of the image (i.e., the second high distortion area on the upper side Y1 of the image) where the object searching unit 41 will not perform a search. The low distortion area Fa (first low distortion area) as the range in which the boundary line B is set and the low distortion area Fa (second low distortion area) as the range in which the object searching unit 41 searches for the object A may be the same or different. Since the range appropriate for the first low distortion area and the range appropriate for the second low distortion area do not necessarily coincide, it is preferable that the first low distortion area and the second low distortion area are arranged differently. Also, only one of the first low distortion area and the second low distortion area may be set.

[0033] This detection area setting unit 51 (see Fig. 2) may set a plurality of boundary lines B as shown in Fig. 5. The detection area setting unit 51 may set a plurality of detection areas D1-D2. In this case, the detection area setting unit 51 sets a plurality of boundary lines B at intervals in the vertical direction Y of the image. Then, as will be described later, the response process of the response processing unit 90 is changed depending on which boundary line B the object A has crossed.

[0034] When a plurality of imaging devices 23 (see FIG. 1) are provided, the detection area setting unit 51 (see FIG. 2) sets a boundary line B individually (for each imaging device 23) for a plurality of captured images Im acquired by the plurality of imaging devices 23. The detection area setting unit 51 sets the boundary line B, for example, so that the corresponding process start distance is constant or approximately constant for each imaging device 23. The detection area setting unit 51 includes a boundary line adjustment unit 52 (see FIG. 2).

[0035] The boundary adjustment unit 52 (see FIG. 2) is a part that adjusts the position of the boundary line B shown in FIG. 4 in the up-down direction Y of the image. Even if the position of the boundary line B is adjusted by the boundary adjustment unit 52, the shape of the boundary line B in the captured image Im remains constant. If the position of the boundary line B is changed in a portion of the captured image Im where distortion is large, it is necessary to change the curvature of the boundary line B in accordance with the distortion of the captured image Im, and multiple parameters must be prepared, but in this embodiment, the shape of the boundary line B remains constant. The adjustment of the position of the boundary line B by the boundary adjustment unit 52 may be manual or automatic.

[0036] A specific example of a case where the position of the boundary line B is adjusted manually is as follows. For example, the control unit 30 (see FIG. 2) is set to an "adjustment mode." Then, the captured image display unit 80 (see FIG. 2), which will be described later, displays the boundary line B superimposed on the captured image Im. While looking at this display, the worker moves the boundary line B in the vertical direction Y of the image and determines the position of the boundary line B. This operation is performed by the boundary line adjustment input device 21c (see FIG. 2). Specifically, for example, the worker moves the boundary line B in the vertical direction Y of the image by rotating the jog dial 21c1 (see FIG. 2) and determines the position of the boundary line B by operating (for example, pressing) the switch 21c2 (see FIG. 2).

[0037] A specific example of a case in which the position of the boundary line B is automatically adjusted is as follows. The position of the boundary line B is adjusted, for example, based on the travel speed of the work machine 1. Specifically, the boundary line adjustment unit 52 (see FIG. 2) sets the position of the boundary line B in the captured image Im when the travel speed of the work machine 1 in the direction approaching the object A is a first speed as a first position B1. A travel speed of the work machine 1 in the direction approaching the object A that is faster than the first speed is set as a second speed. The position of the boundary line B in the captured image Im at this second speed is set as a second position B2. At this time, the boundary line adjustment unit 52 sets the second position B2 to a position Y1 above the first position B1 in the image. The boundary line adjustment unit 52 may set the position of the boundary line B to the upper side Y1 of the image in a stepwise manner as the travel speed of the work machine 1 increases, or may set the position of the boundary line B to the upper side Y1 of the image continuously. Whether the traveling direction of the work machine 1 is a direction approaching the target object A or not can be determined based on the traveling direction (forward, backward) of the lower traveling body 11, the rotation angle of the upper rotating body 13 relative to the lower traveling body 11, and the shooting direction of the imaging device 23. The position of the boundary line B does not need to be adjusted (i.e., the boundary line adjustment unit 52 does not need to be provided). The boundary line B is set in advance in the detection area setting unit 51, and the position of this boundary line B does not need to be changed.

[0038] When the boundary line adjustment unit 52 adjusts the position of the boundary line B, the boundary line adjustment unit 52 directly or indirectly acquires the traveling speed of the work machine 1. For example, the boundary line adjustment unit 52 shown in FIG. 2 may acquire the traveling speed from the operation amount of the travel operation unit 21a. For example, the boundary line adjustment unit 52 may acquire the traveling speed from information (rotation speed, flow rate of hydraulic oil, etc.) relating to the motor that drives the lower traveling body 11 (see FIG. 1). For example, the boundary line adjustment unit 52 may acquire the traveling speed from a sensor that detects the position of the work machine 1.

[0039] As shown in FIG. 4, the boundary line limit position setting unit 53 sets a boundary line upper limit position Ga (boundary line counter-machine side limit position) and a boundary line lower limit position Gb (boundary line machine side limit position) for the captured image Im. The boundary line lower limit position Gb is set on the image lower side Y2 of the boundary line upper limit position Ga. The boundary line B is set within the area between the boundary line upper limit position Ga and the boundary line lower limit position Gb. The boundary line upper limit position Ga and the boundary line lower limit position Gb are set based on the necessity of searching for the object A, for example, like the search upper limit position Ea and the search lower limit position Eb. For example, when the boundary line upper limit position Ga, the boundary line lower limit position Gb, the high distortion area Fb, and the low distortion area Fa are each set, the detection area setting unit 51 sets the boundary line B within the area between the boundary line upper limit position Ga and the boundary line lower limit position Gb and within the low distortion area Fa. Also, in the example shown in FIG. 4, the boundary line upper limit position Ga and the search upper limit position Ea coincide with each other, but they do not have to coincide with each other (the same applies to the boundary line lower limit position Gb and the search lower limit position Eb).

[0040] The determination unit 61 (see FIG. 2) determines whether or not a part or all of the object A is present inside the detection area D. Specifically, the determination unit 61 determines, for example, whether or not the coordinates (Ax, Ay) of the end of the frame K on the lower side Y2 of the image are present within the detection area D. If the boundary line B is linear, it is possible to determine whether or not the object A is present within the detection area D by simply judging whether the coordinate (Ay) of the object A in the vertical direction Y of the image is on the work machine 1 side (the lower side Y2 of the image) with respect to the coordinate of the boundary line B in the vertical direction Y of the image.

[0041] The response processing control unit 63 (see FIG. 2) causes the response processing unit 90 (see FIG. 2) to execute a predetermined response processing (details will be described later) when it is detected that the object A has entered inside the detection area D. When it is not detected that the object A has entered inside the detection area D, the response processing control unit 63 does not cause the response processing unit 90 to execute the response processing.

[0042] The display control unit 65 (see FIG. 2) controls the display of the captured image display unit 80 (see FIG. 2).

[0043] The captured image display unit 80 (see FIG. 2) displays the captured image Im. The captured image display unit 80 is provided, for example, in the cab 13a shown in FIG. 1. In cases such as when the work machine 1 is remotely operated, the captured image display unit 80 (see FIG. 2) may be disposed outside the work machine 1. The captured image display unit 80 is, for example, a part or all of a display. The display of the captured image Im (see FIG. 4) on the captured image display unit 80 is a display in which the captured image Im captured by the imaging device 23 is inverted horizontally, like an image reflected in a rearview mirror when viewed from the cab 13a. The captured image display unit 80 may display a boundary line B superimposed on the captured image Im shown in FIG. 4. The captured image display unit 80 may display a detection area D superimposed on the captured image Im. Note that the boundary line B does not have to be displayed on the captured image display unit 80, and may be used only for inside processing (such as determination by the determination unit 61 (see FIG. 2)) (the same applies to the detection area D). When multiple imaging devices 23 are provided, the captured image display unit 80 may switch between displaying captured images Im for each imaging device 23, or may display a captured image Im that is a combination of captured images Im taken by the multiple imaging devices 23.

[0044] The response processing unit 90 (see FIG. 2) executes a predetermined response process when the object A enters inside the detection area D. The response process executed by the response processing unit 90 shown in FIG. 2 may be a warning, a restriction on the operation of the work machine 1, or other processing. When the response process executed by the response processing unit 90 is a warning, this warning may be, for example, a warning to the operator of the work machine 1, or may be, for example, a warning to workers around the work machine 1. This warning may be a warning by sound, a warning by light, a warning by a screen display (text, graphics, etc.) on a display, or a warning by a combination of these. When the response process executed by the response processing unit 90 is a restriction on the operation of the work machine 1, this restriction may be, for example, a deceleration of the operation of the work machine 1, or may be a stop of the operation of the work machine 1. The operation of the work machine 1 restricted by the response processing unit 90 may be the traveling of the lower traveling body 11 shown in FIG. 1, the rotation of the upper rotating body 13 relative to the lower traveling body 11, or the operation of the attachment 15. 5, when multiple boundary lines B are set, the response processing unit 90 may change the response processing to be performed depending on the boundary line B crossed by the object A. In this case, the response processing unit 90 may increase the level of warning or restriction the closer the boundary line B crossed by the object A is to the work machine 1 (the closer to the lower side Y2 of the image). For example, the response processing unit 90 may change the warning pattern (warning content, number of times, duration, etc.) or change the level of restriction on the operation of the work machine 1 depending on the boundary line B crossed by the object A.

[0045] (Activation) A specific example of the operation of the control unit 30 shown in Fig. 2 will be described with reference to Fig. 3. The object search unit 41 shown in Fig. 2 searches for the object A from the captured image Im (step S11) and calculates the coordinates of the object A (for example, the coordinates of the feet) (step S12). When the coordinate of the object A in the vertical direction Y of the image falls below the coordinate (predetermined value) of the boundary line B in the vertical direction Y of the image (step S13), the response processing control unit 63 outputs a command to the response processing unit 90 (step S14).

[0046] (Consideration of processing load and processing speed) Let us consider the case where the coordinates of the captured image Im shown in Fig. 4 are converted into virtual plane coordinates. In this case, the processing is carried out as follows. [Step 1] The position of the object A in the captured image Im is calculated. [Step 2] The coordinates of the captured image Im are converted into virtual plane coordinates based on the work machine 1 shown in Fig. 1 (a virtual plane area is created). [Step 3] The coordinates of the object A in the virtual plane are calculated. [Step 4] The distance from the work machine 1 to the object A in the virtual plane is calculated. [Step 5] If the distance from the work machine 1 to the object A falls below a predetermined value, a corresponding processing (warning, operation restriction, etc.) is carried out.

[0047] In this way, when the coordinates of the captured image Im shown in FIG. 4 are converted into virtual plane coordinates, complex processing is required in the above steps 2, 3, and 4, which results in a high processing load and a long processing time. For example, to calculate the coordinates of the target A on the virtual plane, information on the mounting position and mounting angle of the imaging device 23 (see FIG. 1) is required, and the calculation logic is complex. If the processing time is long, it takes a long time from when the target A enters the inside of the detection area D (after crossing the boundary line B) until the response processing starts (the response processing is delayed). In addition, the mounting position and mounting angle of the imaging device 23 usually differ depending on the model and specifications of the work machine 1. Therefore, it is necessary to set parameters for each model and specifications of the work machine 1, and this setting is time-consuming (complex).

[0048] On the other hand, in this embodiment, the coordinates of the captured image Im are not converted into virtual plane coordinates (see FIG. 3), so the processes of steps 2, 3, and 4 can be omitted. Therefore, the processing load on the control unit 30 (see FIG. 2) can be reduced, the processing speed can be improved, and the processing time can be shortened. As a result, the time from when the object A enters the inside of the detection area D to when the response process (warning, operation restriction) is started can be shortened (the response process can be performed quickly). In addition, since the coordinates of the captured image Im are not converted into virtual plane coordinates, information on the mounting position and mounting angle of the imaging device 23 (see FIG. 1) is not required. Therefore, even if complex parameters are not set, the surroundings detection device 20 (see FIG. 2) can be applied to various types of work machines 1 (high versatility).

[0049] (Compared to distance sensors) In general, searching for the object A by image processing of the captured image Im requires a higher processing load and a longer processing time than searching for the object A by a sensor that acquires distance information (e.g., a TOF (Time Of Flight) sensor, an ultrasonic sensor, an infrared sensor, a radio wave (e.g., millimeter wave) sensor, etc.). On the other hand, in this embodiment, as described above, the object A is searched only in a part of the captured image Im, so that the processing load can be reduced and the processing time can be shortened. In addition, in this embodiment, by omitting processes other than the search for the object A (e.g., the processes of steps 2, 3, and 4 above), the processing load of the control unit 30 can be reduced and the processing time can be shortened. In addition, for example, if the object A is a person, the surroundings detection device 20 can detect only this person as a detection target, but a sensor that acquires distance information cannot detect only this person as a detection target.

[0050] (effect) The effects of the surroundings detection device 20 of the work machine 1 shown in FIG. 2 are as follows.

[0051] (Effects of the first invention) The surroundings detection device 20 includes an imaging device 23, an object searching unit 41, a detection area setting unit 51, and a response processing unit 90. The imaging device 23 is provided on the work machine 1, and captures an object A (see FIG. 4) around the work machine 1 to obtain a captured image Im. The object searching unit 41 searches for the position of the object A (see FIG. 4) in the captured image Im.

[0052] [Configuration 1-1] The detection area setting unit 51 sets a detection area D in the captured image Im, which is an area for detecting an object A, as shown in Fig. 4. The response processing unit 90 (see Fig. 2) executes a predetermined response process when the object A enters inside the detection area D in the captured image Im.

[0053] [Configuration 1-2] The detection area setting unit 51 (see Fig. 2) sets a boundary line B that is the boundary of the detection area D in the captured image Im. In the captured image Im, the side closer to the work machine 1 is the image lower side Y2 (machine side), the side farther from the work machine 1 is the image upper side Y1 (opposite machine side), and the direction connecting the image lower side Y2 and the image upper side Y1 is the image up-down direction Y (front direction). In the captured image Im, the direction perpendicular to the image up-down direction Y is the image left-right direction X (direction perpendicular to the front). The boundary line B extends in the image left-right direction X and has a fixed shape.

[0054] In the above [Configuration 1-1], when the object A enters the inside of the detection area D set in the captured image Im, a response process is executed. In the surroundings detection device 20 (see FIG. 2), for example, it is not necessary to perform a complicated process such as converting the coordinates of the object A in the captured image Im into coordinates in a virtual plane and calculating the distance from the work machine 1 to the object A in the virtual plane. Also, in the above [Configuration 1-2], the boundary line B extends in the left-right direction X of the image and has a fixed shape. Therefore, compared to the case where the shape of the boundary line B is variable, it is possible to reduce the load of the process of determining whether the object A has entered the inside of the detection area D (see the above [Configuration 1-1]). Therefore, the above [Configuration 1-1] and [Configuration 1-2] can reduce the load of the process required for detecting the object A and shorten the processing time. As a result, it is possible to suppress a delay in the response process by the response processing unit 90 (see FIG. 2) when the object A enters the inside of the detection area D.

[0055] (Effects of the second invention) [Configuration 2] The boundary line B is a straight line in the captured image Im.

[0056] The above [Configuration 2] can further reduce the load of the process of determining whether or not the object A has entered inside the detection area D (see the above [Configuration 1-1]).

[0057] (Effects of the third invention) [Configuration 3] As shown in Fig. 2, the surrounding detection device 20 includes a high / low distortion area setting unit 45 (first high / low distortion area setting unit). As shown in Fig. 4, the high / low distortion area setting unit 45 sets a high distortion area Fb (first high distortion area) and a low distortion area Fa (first low distortion area) in which image distortion is smaller than that of the high distortion area Fb in the captured image Im. The boundary line B is set within the low distortion area Fa.

[0058] The above [Configuration 3] provides the following effects. When a boundary line B extending in the image left-right direction X (for example, linear or substantially linear) is set in an area of ​​the captured image Im with large distortion, the response process start distance changes significantly depending on the position of the object A in the image left-right direction X of the captured image Im. This makes it difficult for the operator of the work machine 1 to grasp the response process start distance (it is difficult to get a sense of distance). Therefore, in the above [Configuration 3], the boundary line B is set within the low distortion area Fa. Therefore, the response process start distance can be made constant or substantially constant regardless of the position of the object A in the image left-right direction X of the captured image Im. Therefore, the operator can easily grasp the response process start distance (the operator can easily get a sense of distance). As a result, the operator can be prompted to perform an appropriate avoidance operation (an operation to avoid contact between the object A and the work machine 1).

[0059] (Effect of the fourth invention) [Configuration 4] As shown in Fig. 2, the surrounding detection device 20 includes a boundary line limit position setting unit 53. As shown in Fig. 4, the boundary line limit position setting unit 53 sets, for the captured image Im, a boundary line upper limit position Ga (boundary line anti-machine side limit position) and a boundary line lower limit position Gb (boundary line machine side limit position) on the image lower side Y2 of the boundary line upper limit position Ga. The boundary line B is set within the area between the boundary line upper limit position Ga and the boundary line lower limit position Gb.

[0060] The above [Configuration 4] provides the following effects. It is assumed that the upper Y1 portion and the lower Y2 portion of the captured image Im are inappropriate positions for the boundary line B (for example, positions where the sky or the lower running structure 11 are reflected). Therefore, in the above [Configuration 4], the boundary line B is set within the area between the boundary line upper limit position Ga and the boundary line lower limit position Gb. This makes it possible to appropriately detect the object A. For example, it is possible to prevent an object other than the object A from being erroneously detected as the object A.

[0061] (Effect of the fifth aspect of the invention) [Configuration 5] The surroundings detection device 20 (see FIG. 2) includes a boundary adjustment unit 52 (see FIG. 2) that adjusts the position of the boundary line B in the up-down direction Y of the image.

[0062] For example, the angle of view and mounting angle of the imaging device 23 (see FIG. 1) vary within the tolerance range. Therefore, if the position of boundary line B in the vertical direction Y of the image remains unchanged, the correspondence processing start distance will vary. In the above [Configuration 5], the correspondence processing start distance can be adjusted by adjusting the position of boundary line B using boundary line adjustment unit 52 (see FIG. 2). This makes it possible to adjust the correspondence processing start distance to a predetermined distance even if there is variation in the angle of view and mounting angle of the imaging device 23 (the variation in the angle of view and mounting angle of the imaging device 23 can be absorbed).

[0063] Also, for example, it is conceivable that there may be cases where it is desired to change the response processing start distance depending on the situation at the work site of the work machine 1. In the above [Configuration 5], the boundary line adjustment unit 52 adjusts the position of the boundary line B, thereby making it possible to change the response processing start distance, thereby improving convenience.

[0064] When the surrounding detection device 20 (see FIG. 2) is equipped with the above-mentioned [Configuration 4] and [Configuration 5], the position of the boundary line B is adjusted within the region between the boundary line upper limit position Ga and the boundary line lower limit position Gb. Thus, the position of the boundary line B is adjusted within a range in which the object A can be appropriately detected. This makes it possible to eliminate, for example, erroneous adjustment work in which the position of the boundary line B is adjusted to an inappropriate position (such as a position in which the sky or the lower running structure 11 is reflected). This makes it possible to improve the workability of the work of adjusting the position of the boundary line B.

[0065] (Effect of the sixth aspect of the invention) [Configuration 6] The position of the boundary line B in the captured image Im when the working machine 1 is traveling at a first speed in a direction approaching the object A is defined as a first position B1. The position of the boundary line B in the captured image Im when the working machine 1 is traveling at a second speed that is faster than the first speed in a direction approaching the object A is defined as a second position B2. The boundary line adjustment unit 52 sets the second position B2 to a position Y1 above the first position B1 in the image.

[0066] The above [Configuration 6] provides the following effect. It is expected that the faster the travel speed of the work machine 1 in the direction approaching the object A, the shorter the time it takes for the work machine 1 to reach the object A, and the response process may be delayed. In the above [Configuration 6], when the travel speed of the work machine 1 is the second speed (faster) than when it is the first speed (slower), the object A can be caused to enter the detection area D at a position farther from the work machine 1 (the response process start distance becomes longer). Therefore, when the work machine 1 is traveling at the second speed (faster) than when it is traveling at the first speed (slower), the response process can be started earlier.

[0067] (Effects of the seventh aspect of the invention) [Configuration 7] As shown in Fig. 2, the surroundings detection device 20 includes a search range limiting unit 43. As shown in Fig. 4, the search range limiting unit 43 sets a search upper limit position Ea (search limit position on the opposite side to the machine) in the captured image Im. The search range of the object A by the object searching unit 41 is within the area Y2 below the search upper limit position Ea in the image.

[0068] The above [Configuration 7] provides the following effects. It is assumed that the upper Y1 portion of the captured image Im is an inappropriate position for searching for the object A (for example, a position where the sky is reflected). Therefore, in the above [Configuration 7], the search range for the object A by the object searching unit 41 is within an area Y2 below the search upper limit position Ea. Therefore, the object A can be appropriately searched for. Also, the search range can be limited compared to when the object searching unit 41 (see FIG. 2) searches for the object A in the entire captured image Im. Therefore, it is possible to reduce the processing load on the object searching unit 41 (see FIG. 2) and shorten the processing time.

[0069] (Effect of the eighth aspect of the invention) [Configuration 8] As shown in Fig. 2, the surroundings detection device 20 includes a search range limiting unit 43. As shown in Fig. 4, the search range limiting unit 43 sets a search lower limit position Eb (search machine side limit position) in the captured image Im. The search range of the object A by the object searching unit 41 is within an area Y1 above the image from the search lower limit position Eb.

[0070] The above [Configuration 8] provides the following effects. It is assumed that the lower Y2 portion of the captured image Im is an inappropriate position for searching for the object A (for example, a position where only the upper rotating body 13 (see FIG. 1) is captured). Therefore, in the above [Configuration 8], the search range for the object A by the object search unit 41 is within an area Y1 above the search lower limit position Eb. Therefore, the object A can be appropriately searched for. Also, the search range can be limited compared to when the object search unit 41 (see FIG. 2) searches for the object A in the entire captured image Im. Therefore, the processing load on the object search unit 41 (see FIG. 2) can be reduced, and the processing time can be shortened.

[0071] (Effect of the ninth aspect of the invention) [Configuration 9] As shown in Fig. 2, the surroundings detection device 20 includes a height-low distortion area setting unit 45 (second height-low distortion area setting unit). As shown in Fig. 4, the height-low distortion area setting unit 45 sets high distortion areas Fb (second high distortion areas) on the upper side Y1 and lower side Y2 of the image in the captured image Im. The height-low distortion area setting unit 45 sets a low distortion area Fa (second low distortion area) in which image distortion is smaller than that of the high distortion area Fb. The search range of the object A by the object search unit 41 is within the low distortion area Fa and the high distortion area Fb on the lower side Y2 of the image.

[0072] The above [Configuration 9] provides the following effects. It is assumed that the region with large distortion in the upper Y1 portion of the captured image Im is an inappropriate position for searching for the object A (for example, a position where the sky is reflected). Therefore, in the above [Configuration 9], the search range for the object A by the object search unit 41 is within the low distortion region Fa and the high distortion region Fb in the lower Y2 portion of the image. Therefore, the object A can be searched for within an appropriate region. Also, the search range can be limited compared to when the object search unit 41 (see FIG. 2) searches for the object A in the entire captured image Im. Therefore, it is possible to reduce the processing load of the object search unit 41 (see FIG. 2) and shorten the processing time.

[0073] (Effect of the 10th Invention) [Configuration 10] The detection area setting unit 51 (see Fig. 2) sets a plurality of boundary lines B at intervals in the vertical direction Y of the image as shown in Fig. 5. The correspondence processing unit 90 (see Fig. 2) changes the correspondence processing to be performed depending on the boundary line B that the object A has crossed.

[0074] The above [Configuration 10] makes it possible to change the content of the response process (for example, the content of the warning or operation restriction) according to the distance from the work machine 1 to the object A. As a result, the location of the object A, etc. can be appropriately notified to the operator.

[0075] (Effects of the eleventh aspect of the invention) [Configuration 11] A plurality of imaging devices 23 (see FIG. 1) are provided. A detection area setting unit 51 (see FIG. 2) sets a boundary line B for each of the captured images Im acquired by the plurality of imaging devices 23 (see FIG. 1).

[0076] According to the above [Configuration 11], the correspondence processing start distance can be set for each imaging device 23, thereby improving convenience. More specifically, for example, the angle of view, mounting angle, mounting position, etc. are different between the multiple imaging devices 23 shown in FIG. 1. Therefore, if a boundary line B (see FIG. 4) is set at a common position on the captured image Im (see FIG. 4) of each imaging device 23, the correspondence processing start distance will change for each imaging device 23. Therefore, by setting the boundary line B for each imaging device 23 as in the above [Configuration 11], the correspondence processing start distance can be made to match or approximately match between the multiple imaging devices 23.

[0077] (Modification) The above embodiment may be modified in various ways. For example, the connection of each component of the above embodiment shown in FIG. 2 may be changed. For example, each region (such as low distortion region Fa) shown in FIG. 4 and the like, and the position that defines the region (such as search upper limit position Ea), may be fixed, may be changed by manual operation, or may be automatically changed according to some conditions. For example, the number of components may be changed, and some of the components may not be provided. For example, what has been described as multiple different members or parts may be treated as one member or part. For example, what has been described as one member or part may be provided as multiple different members or parts. [Explanation of symbols]

[0078] 1. Work Machinery 20 Surroundings detection device 23 Imaging device 41 Object Search Unit 43 Search range restriction section 45 High and low distortion area setting section (first high and low distortion area setting section, second high and low distortion area setting section) 51 Detection area setting unit 52 Boundary Adjustment Section 53 Boundary line limit position setting section 90 Response Department A. Object B Border B1 1st position B2 2nd position C Non-detection area D, D1, D2 detection area Ea Search upper limit position (search anti-machine side limit position) Eb Search lower limit position (search machine side limit position) Fa low strain area (first low strain area, second low strain area) Fb High strain area (1st high strain area, 2nd high strain area) Ga Boundary line upper limit position (Boundary line opposite machine side limit position) Gb Boundary line lower limit position (Boundary line machine side limit position) Im Captured image X Image left / right direction (orthogonal to the front) Y Image up / down direction (front direction) Y1 Top of image (opposite machine side) Y2 Bottom of image (machine side)

Claims

1. an imaging device provided in a work machine for photographing objects around the work machine and acquiring captured images; A corresponding processing unit; Equipped with In the captured image, when the side close to the work machine is defined as the machine side, The captured image includes a detection area that is an area where the object is detected and a non-detection area that does not detect the object, the detection area is closer to the machine than the non-detection area; the response processing unit executes a predetermined response process when the object searched for within the search range in the captured image enters inside the detection area, The search range exists in at least a part of the non-detection area. Surrounding detection device for work machines.

2. A surrounding detection device for a work machine according to claim 1, The search range exists only in a part of the non-detection area. Surrounding detection device for work machines.

3. The surrounding detection device for a work machine according to claim 1, In the captured image, when the side farther from the work machine is defined as the opposite side to the machine, The non-detection area is located on the machine side and the opposite machine side of the detection area, Surrounding detection device for work machines.

4. The surrounding detection device for a work machine according to claim 1, In the captured image, when the side farther from the work machine is defined as the opposite side to the machine, the detection area is set from an end of the captured image on the machine side to a position on the opposite side to the machine side from the end, Surrounding detection device for work machines.

5. The surrounding detection device for a work machine according to claim 1, In the captured image, when the side farther from the work machine is defined as the opposite side to the machine, The captured image includes high distortion regions on the anti-machine side and the machine side, and low distortion regions in which image distortion is smaller than that of the high distortion regions, The detection area is within the low strain area and within the high strain area on the machine side. Surrounding detection device for work machines.

6. The surrounding detection device for a work machine according to claim 1, a boundary line limit position setting unit that sets a limit position of a boundary line that is a boundary of the detection area in the captured image, In the captured image, when the side farther from the work machine is defined as the opposite side to the machine, The boundary line limit position setting unit sets, for the captured image, a boundary line counter-machine side limit position and a boundary line machine side limit position that is closer to the machine than the boundary line counter-machine side limit position, The boundary line is set within an area between the boundary line counter-machine side limit position and the boundary line machine side limit position. Surrounding detection device for work machines.

7. The surrounding detection device for a work machine according to claim 1, a boundary adjustment unit that adjusts a position of a boundary line that is a boundary of the detection area in the captured image, In the captured image, when the side farther from the work machine is defined as the opposite side to the machine, a position of the boundary line in the captured image when the travel speed of the work machine in a direction approaching the object is a first speed is defined as a first position; When the position of the boundary line in the captured image when the traveling speed of the work machine in the direction approaching the object is a second speed that is faster than the first speed, the boundary line adjustment unit sets the second position on the opposite side to the machine than the first position. Surrounding detection device for work machines.

8. The surrounding detection device for a work machine according to claim 1, When the side of the captured image farther from the work machine is defined as the opposite side to the machine, the detection area is within an area on the opposite side to the machine than a limit position set based on a portion of the captured image in which the object is not captured. Surrounding detection device for work machines.

9. The surrounding detection device for a work machine according to claim 1, A boundary limit position setting unit is provided, In the captured image, when the side farther from the work machine is defined as the opposite side to the machine, the boundary line limit position setting unit sets, for the captured image, a boundary line machine side limit position which is a limit position of a boundary line that is a boundary of the detection area in the captured image; The boundary line is set on the opposite side to the machine side of the boundary line machine side limit position. Surrounding detection device for work machines.

10. The surrounding detection device for a work machine according to claim 1, A boundary limit position setting unit is provided, In the captured image, when the side farther from the work machine is defined as the opposite side to the machine, the boundary line limit position setting unit sets, for the captured image, a boundary line limit position on the opposite side to the machine side, which is a limit position of a boundary line that is a boundary of the detection area in the captured image; The boundary line is set on the machine side of the boundary line counter-machine side limit position. Surrounding detection device for work machines.

Images