Obstacle warning system for work machine and obstacle warning method for work machine

Abstract

To alter an obstacle detection range set in a work machine.SOLUTION: An obstacle determination unit determines whether or not an obstacle exists within a detection range of an object of obstacle detection. When it is determined that the obstacle exists, a notification unit makes a notification indicating the obstacle. An operation input unit accepts an operation performed on a display unit in order to alter the detection range. An alteration unit alters the size of the detection range on the basis of the operation.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present disclosure relates to an obstacle warning system for a work machine and an obstacle warning method for a work machine. [Background technology]

[0002] Patent Document 1 discloses a technique related to a perimeter monitoring system that detects people around a work machine. According to the technique described in Patent Document 1, the perimeter monitoring system detects surrounding obstacles. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-035791 A Summary of the Invention [Problem to be solved by the invention]

[0004] When the perimeter monitoring system detects an obstacle, it notifies the operator of the presence of the obstacle via a display, speaker, etc. Upon receiving the notification from the perimeter monitoring system, the operator of the work machine confirms that an obstacle exists and that safety is ensured.

[0005] However, depending on the type of work being performed by a work machine, there are cases where it is not necessary to monitor for obstacles all around the work machine. For example, when a dump truck is being loaded and turned, it may not be necessary to detect obstacles in the direction in which the dump truck is located. An object of the present disclosure is to provide an obstacle warning system for a work machine and an obstacle warning method for a work machine that are capable of changing the detection range of an obstacle in the work machine. [Means for solving the problem]

[0006] According to one aspect of the present invention, an obstacle warning system for a work machine includes an obstacle determination unit that determines whether an obstacle is present within a detection range of an obstacle detection target, a warning unit that issues a warning indicating the obstacle when it is determined that the obstacle is present, an operation input unit that accepts operation on a display unit to change the detection range, and a change unit that changes the size of the detection range based on the operation. Effect of the Invention

[0007] According to the above aspect, the operator of the work machine can change the obstacle detection range of the work machine. [Brief description of the drawings]

[0008] [Figure 1] 1 is a schematic diagram showing the configuration of a work machine according to a first embodiment. [Diagram 2] 2 is a diagram showing the imaging ranges of a plurality of cameras 121 equipped on the work machine according to the first embodiment. FIG. [Diagram 3] FIG. 2 is a diagram showing the internal configuration of a driver's cab according to the first embodiment. [Figure 4] 1 is a schematic block diagram showing a configuration of a control device according to a first embodiment. [Diagram 5] FIG. 2 is a diagram showing an example of a display screen according to the first embodiment. [Figure 6] 5 is a flowchart showing the operation of the control device according to the first embodiment. [Figure 7] FIG. 4 is a diagram illustrating an example of the operation of the control device according to the first embodiment. [Figure 8] FIG. 11 is a diagram illustrating an example of the operation of a control device according to a modified example of the first embodiment. [Figure 9] FIG. 11 is a diagram showing an example of a display screen according to the second embodiment. [Figure 10] 10 is a flowchart showing the operation of the control device according to the second embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of the operation of the control device according to the second embodiment. [Figure 12]FIG. 11 is a diagram illustrating an example of the operation of the control device according to the first modified example. [Figure 13] FIG. 11 is a diagram illustrating an example of the operation of the control device according to the second modified example. [Figure 14] FIG. 13 is a diagram illustrating an example of the operation of a control device according to a third modified example. [Figure 15] FIG. 13 is a diagram illustrating an example of the operation of a control device according to a fourth modified example. [Figure 16] FIG. 13 is a diagram illustrating an example of the operation of a control device according to a fifth modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] First embodiment Hereinafter, the embodiments will be described in detail with reference to the drawings.

[0010] Configuration of the work machine 100 FIG. 1 is a schematic diagram showing the configuration of a work machine 100 according to a first embodiment. The work machine 100 operates at a construction site and works on a construction target such as earth and sand. The work machine 100 according to the first embodiment is, for example, a hydraulic excavator. The work machine 100 includes a traveling body 110, a rotating body 120, a work implement 130, and a cab 140. The running body 110 supports the working machine 100 so that the working machine 100 can run. The running body 110 is, for example, a pair of left and right caterpillar tracks. The rotating body 120 is supported by the running body 110 so as to be capable of rotating about a rotation center. The work machine 130 is hydraulically driven. The work machine 130 is supported on the front part of the rotating body 120 so as to be drivable in the vertical direction. The operator's cab 140 is a space where an operator sits and operates the work machine 100. The operator's cab 140 is provided on the left front part of the rotating body 120. Here, the part of the rotating body 120 to which the work machine 130 is attached is referred to as the front part. Furthermore, with respect to the rotating body 120, the part opposite the front part is referred to as the rear part, the left part as the left part, and the right part as the right part.

[0011] Configuration of the rotating body 120 The revolving unit 120 is provided with a plurality of cameras 121 that capture images of the surroundings of the work machine 100. Fig. 2 is a diagram showing the imaging ranges of the plurality of cameras 121 provided on the work machine 100 according to the first embodiment. Specifically, the rotating body 120 is provided with a left rear camera 121A that images a left rear area Ra of the periphery of the rotating body 120, a rear camera 121B that images a rear area Rb of the periphery of the rotating body 120, a right rear camera 121C that images a right rear area Rc of the periphery of the rotating body 120, and a right front camera 121D that images a right front area Rd of the periphery of the rotating body 120. Note that the imaging ranges of the multiple cameras 121 may partially overlap each other. The imaging ranges of the multiple cameras 121 cover the entire circumference of the work machine 100 excluding the left front area Re visible from the cab 140. Note that, although the cameras 121 according to the first embodiment capture images of the left rear, rear, right rear, and right front of the revolving body 120, other embodiments are not limited to this. For example, the number and imaging ranges of the cameras 121 according to other embodiments may be different from the examples shown in Figs. 1 and 2.

[0012] The left rear camera 121A is for imaging the left side area and the left rear area of ​​the revolving body 120 as shown in the left rear area Ra in FIG. 2, but may image either one of the areas. Similarly, the right rear camera 121C is for imaging the right side area and the right rear area of ​​the revolving body 120 as shown in the right rear area Rc in FIG. 2, but may image either one of the areas. Similarly, the right front camera 121D is for imaging the right front area and the right side area of ​​the revolving body 120 as shown in the right front area Rd in FIG. 2, but may image either one of the areas. In another embodiment, the entire periphery of the work machine 100 may be the imaging range using multiple cameras 121. For example, a left front camera that images the left front range Re may be provided, and the entire periphery of the work machine 100 may be the imaging range.

[0013] Configuration of the work machine 130 The work machine 130 includes a boom 131, an arm 132, a bucket 133, a boom cylinder 131C, an arm cylinder 132C, and a bucket cylinder 133C.

[0014] The base end of the boom 131 is attached to the revolving body 120 via a boom pin 131P. The arm 132 connects the boom 131 and the bucket 133. The base end of the arm 132 is attached to the tip of the boom 131 via an arm pin 132P. The bucket 133 includes a blade for excavating earth and sand, and a storage portion for storing the excavated earth and sand. A base end portion of the bucket 133 is attached to a tip portion of the arm 132 via a bucket pin 133P.

[0015] The boom cylinder 131C is a hydraulic cylinder for operating the boom 131. A base end of the boom cylinder 131C is attached to the rotating body 120. A tip end of the boom cylinder 131C is attached to the boom 131. The arm cylinder 132C is a hydraulic cylinder for driving the arm 132. A base end of the arm cylinder 132C is attached to the boom 131. A tip end of the arm cylinder 132C is attached to the arm 132. The bucket cylinder 133C is a hydraulic cylinder for driving the bucket 133. A base end of the bucket cylinder 133C is attached to the arm 132. A tip end of the bucket cylinder 133C is attached to a link member connected to the bucket 133.

[0016] Configuration of the cab 140 FIG. 3 is a diagram showing the internal configuration of the operator's cab 140 according to the first embodiment. A driver's seat 141, an operating device 142, and a control device 145 are provided in the driver's cab 140.

[0017] The operation device 142 is a device for driving the traveling body 110, the rotating body 120, and the work machine 130 by manual operation by an operator. The operation device 142 includes a left operation lever 142LO, a right operation lever 142RO, a left foot pedal 142LF, a right foot pedal 142RF, a left traveling lever 142LT, and a right traveling lever 142RT.

[0018] The left operating lever 142LO is provided on the left side of the driver's seat 141. The right operating lever 142RO is provided on the right side of the driver's seat 141.

[0019] The left operation lever 142LO is an operation mechanism for performing the rotation operation of the revolving body 120 and the excavation / dumping operation of the arm 132. Specifically, when the operator of the work machine 100 tilts the left operation lever 142LO forward, the arm 132 performs the dumping operation. When the operator of the work machine 100 tilts the left operation lever 142LO rearward, the arm 132 performs the excavation operation. When the operator of the work machine 100 tilts the left operation lever 142LO to the right, the revolving body 120 rotates to the right. When the operator of the work machine 100 tilts the left operation lever 142LO to the left, the revolving body 120 rotates to the left. Note that in other embodiments, when the left operation lever 142LO is tilted in the front-rear direction, the revolving body 120 rotates to the right or left, and when the left operation lever 142LO is tilted in the left-right direction, the arm 132 performs the excavation operation or the dumping operation.

[0020] The right operation lever 142RO is an operation mechanism for performing an excavation / dump operation of the bucket 133 and a raising / lowering operation of the boom 131. Specifically, when the operator of the work machine 100 tilts the right operation lever 142RO forward, a lowering operation of the boom 131 is performed. When the operator of the work machine 100 tilts the right operation lever 142RO rearward, a raising operation of the boom 131 is performed. When the operator of the work machine 100 tilts the right operation lever 142RO to the right, a dump operation of the bucket 133 is performed. When the operator of the work machine 100 tilts the right operation lever 142RO to the left, an excavation operation of the bucket 133 is performed. Note that in other embodiments, when the right operation lever 142RO is tilted in the front-rear direction, the bucket 133 may perform a dump operation or an excavation operation, and when the right operation lever 142RO is tilted in the left-right direction, the boom 131 may perform a raising operation or a lowering operation.

[0021] The left foot pedal 142LF is located on the left side of the floor surface in front of the driver's seat 141. The right foot pedal 142RF is located on the right side of the floor surface in front of the driver's seat 141. The left travel lever 142LT is pivoted to the left foot pedal 142LF and configured so that the tilt of the left travel lever 142LT and the depression of the left foot pedal 142LF are linked. The right travel lever 142RT is pivoted to the right foot pedal 142RF and configured so that the tilt of the right travel lever 142RT and the depression of the right foot pedal 142RF are linked.

[0022] The left foot pedal 142LF and the left traveling lever 142LT correspond to the rotational drive of the left track of the traveling body 110. Specifically, when the operator of the work machine 100 pushes the left foot pedal 142LF or the left traveling lever 142LT forward, the left track rotates in the forward direction. Conversely, when the operator of the work machine 100 pushes the left foot pedal 142LF or the left traveling lever 142LT backward, the left track rotates in the reverse direction.

[0023] The right foot pedal 142RF and the right traveling lever 142RT correspond to the rotational drive of the right crawler belt of the traveling body 110. Specifically, when the operator of the work machine 100 pushes the right foot pedal 142RF or the right traveling lever 142RT forward, the right crawler belt rotates in the forward direction. Conversely, when the operator of the work machine 100 pushes the right foot pedal 142RF or the right traveling lever 142RT backward, the right crawler belt rotates in the reverse direction.

[0024] The control device 145 is equipped with a display 145D that displays information related to a plurality of functions possessed by the work machine 100. The control device 145 is an example of a display system. The display 145D is an example of a display unit. The input means of the control device 145 according to the first embodiment is a touch panel.

[0025] Configuration of the control device 145 FIG. 4 is a schematic block diagram showing the configuration of the control device 145 according to the first embodiment. The control device 145 is a computer including a processor 210, a main memory 230, a storage 250, and an interface 270. The control device 145 also includes a display 145D and a speaker 145S. The control device 145 according to the first embodiment is provided integrally with the display 145D and the speaker 145S, but in other embodiments, at least one of the display 145D and the speaker 145S may be provided separately from the control device 145. When the display 145D and the control device 145 are provided separately, the display 145D may be provided outside the cab 140. In this case, the display 145D may be a mobile display. When the work machine 100 is driven by remote control, the display 145D may be provided in a remote control room provided remotely from the work machine 100. Similarly, when the speaker 145S and the control device 145 are provided separately, the speaker 145S may be provided outside the cab 140. Furthermore, when the work machine 100 is driven by remote control, the speaker 145S may be provided in a remote control room provided remotely from the work machine 100.

[0026] The control device 145 may be configured by a single computer, or the configuration of the control device 145 may be divided and arranged among multiple computers that work together to function as an obstacle warning system for the work machine. The work machine 100 may be equipped with multiple computers that function as the control device 145. Some of the computers that make up the control device 145 may be mounted inside the work machine 100, and other computers may be provided outside the work machine 100. The above-mentioned single control device 145 is also one example of an obstacle warning system for a work machine. In other embodiments, some of the components constituting the obstacle warning system for a work machine may be mounted inside the work machine 100, and other components may be provided outside the work machine 100. For example, the obstacle warning system for a work machine may be configured so that the display 145D is provided in a remote control room provided remotely from the work machine 100. In still other embodiments, one or more computers constituting the obstacle warning system for a work machine may all be provided outside the work machine 100.

[0027] The camera 121 , the display 145 D and the speaker 145 S are connected to the processor 210 via an interface 270 . Examples of the storage 250 include an optical disk, a magnetic disk, a magneto-optical disk, a semiconductor memory, etc. The storage 250 may be an internal medium directly connected to the bus of the control device 145, or may be an external medium connected to the control device 145 via the interface 270 or a communication line. The storage 250 stores a program for implementing monitoring of the surroundings of the work machine 100. In addition, a plurality of images including icons to be displayed on the display 145D are stored in advance in the storage 250.

[0028] The program may be for realizing a part of the function to be performed by the control device 145. For example, the program may be for realizing a function by combining with another program already stored in the storage 250 or by combining with another program implemented in another device. In another embodiment, the control device 145 may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, a part or all of the function to be realized by the processor 210 may be realized by the integrated circuit.

[0029] The storage 250 also stores obstacle dictionary data D1 for detecting obstacles. The obstacle dictionary data D1 may be, for example, dictionary data of feature amounts extracted from each of a plurality of known images in which obstacles are captured. Examples of feature amounts include Histograms of Oriented Gradients (HOG) and Co-occurrence HOG (CoHOG).

[0030] By executing the program, the processor 210 is provided with an acquisition unit 211, an overhead image generation unit 212, an obstacle detection unit 213, an operation input unit 214, a change unit 215, a display screen generation unit 216, a display control unit 217, and an alarm control unit 218. In addition, the processor 210 secures a storage area for a detection range storage unit 231 in the main memory 230 by executing the program.

[0031] The detection range storage unit 231 stores a detection range within which an obstacle is to be detected by the obstacle detection unit 213. The detection range storage unit 231 according to the first embodiment stores any one of the left rear area Ra, the rear area Rb, the right rear area Rc, and the right front area Rd as the detection range.

[0032] The acquisition unit 211 acquires captured images from the multiple cameras 121 . The overhead image generating unit 212 generates an overhead image of a work site viewed from above in plan, with the work machine 100 at the center, by transforming and combining the multiple captured images acquired by the acquiring unit 211. Hereinafter, the captured image transformed by the overhead image generating unit 212 is also referred to as a transformed image. The overhead image generating unit 212 may generate an overhead image by cutting out a portion of each of the transformed captured images and combining the cut-out captured images. An image of the work machine 100 viewed from above in plan is attached in advance to the center of the overhead image generated by the overhead image generating unit 212. That is, the overhead image is a surrounding image that shows the surroundings of the work machine 100.

[0033] The obstacle detection unit 213 detects obstacles for those captured images acquired by the acquisition unit 211 that are within the detection range stored in the detection range storage unit 231. In other words, the obstacle detection unit 213 is an example of an obstacle determination unit that determines whether or not an obstacle exists around the work machine 100. Examples of obstacles include people, vehicles, and rocks. Note that the obstacle detection unit 213 according to other embodiments may detect obstacles in each captured image and mask obstacles that are not included in the detection range stored in the detection range storage unit 231.

[0034] The obstacle detection unit 213 detects an obstacle, for example, in the following procedure. The obstacle detection unit 213 extracts feature amounts from each captured image acquired by the acquisition unit 211. The obstacle detection unit 213 detects an obstacle from the captured image based on the extracted feature amounts and obstacle dictionary data. Examples of the method of detecting an obstacle include pattern matching and object detection processing based on machine learning. In the first embodiment, the obstacle detection unit 213 detects a person using image features, but is not limited to this. For example, in other embodiments, the obstacle detection unit 213 may detect an obstacle based on a measurement value of LiDAR (Light Detection and Ranging) or the like.

[0035] The operation input unit 214 accepts input of a touch operation by an operator on a touch panel of the control device 145. In particular, the operation input unit 214 accepts a swipe operation on the touch panel as an operation of changing the detection range on the display 145D. The swipe operation refers to an operation of sliding a finger that has touched the touch panel. The operation input unit 214 specifies start coordinates and end coordinates related to the swipe operation on the touch panel. Note that in this embodiment, the swipe operation will be described as including a flick operation. The change unit 215 rewrites the detection range stored in the detection range storage unit 231 based on an input to the operation input unit 214 .

[0036] The display screen generating unit 216 generates display screen data G1 in which a marker G12 indicating the position of an obstacle is placed at a position corresponding to the detected position of the obstacle by superimposing it on the overhead image G11 generated by the overhead image generating unit 212. The placement of the marker G12 on the display screen data G1 is an example of notifying the presence of an obstacle. In addition, the display screen generating unit 216 places a frame line G13 representing the detection range stored in the detection range storage unit 231 on the display screen data G1. An example of the display screen will be described later. In other embodiments, the frame line G13 does not need to be displayed.

[0037] The display control unit 217 outputs the display screen data G1 generated by the display screen generating unit 216 to the display 145D. As a result, the display 145D displays the display screen data G1. The display control unit 217 is an example of a notification unit. The alarm control unit 218 outputs an alarm sound signal to the speaker 145S when the obstacle detection unit 213 detects an obstacle. The alarm control unit 218 is an example of a notification unit.

[0038] About the display screen FIG. 5 is a diagram showing an example of a display screen according to the first embodiment. As shown in FIG. 5, the display screen data G1 includes an overhead image G11, a marker G12, a frame line G13, and a single camera image G14. The overhead image G11 is an image of the site viewed from above. The overhead image G11 has a left rear area Ra in which a deformed image related to the left rear camera 121A is captured, a rear area Rb in which a deformed image related to the rear camera 121B is captured, a right rear area Rc in which a deformed image related to the right rear camera 121C is captured, a right front area Rd in which a deformed image related to the right front camera 121D is captured, and a left front area Re in which no image is captured. Note that the boundaries of the left rear area Ra, the rear area Rb, the right rear area Rc, the right front area Rd, and the left front area Re are not displayed in the overhead image G11.

[0039] The marker G12 indicates the position of an obstacle. The shape of the marker G12 may be, for example, a circle, an ellipse, a regular polygon, or a polygon. The frame line G13 surrounds the detection range among the left rear region Ra, the rear region Rb, the right rear region Rc, and the right front region Rd. The single camera image G14 is a single camera image captured by one camera 121.

[0040] <<Method of reporting an obstacle>> FIG. 6 is a flowchart showing the operation of the control device 145 according to the first embodiment. When the control device 145 starts the surroundings monitoring process, the acquisition unit 211 acquires captured images from the multiple cameras 121 (step S1). Next, the overhead image generating unit 212 transforms and synthesizes the multiple captured images acquired in step S1 to generate an overhead image G11 in which the work site is viewed in plan from above, with the work machine 100 at the center (step S2). Next, the obstacle detecting unit 213 executes an obstacle detection process for those of the captured images acquired in step S1 that are related to the detection range stored in the detection range storage unit 231, and determines whether or not an obstacle has been detected (step S3). For example, when the detection range is the left rear area Ra, the obstacle detecting unit 213 executes an obstacle detection process for the captured image captured by the left rear camera 121A.

[0041] If an obstacle is detected in the captured image (step S3: YES), the alarm control unit 218 outputs an alarm audio signal to the speaker 145S (step S4). In addition, the display screen generating unit 216 places a marker G12 at a position corresponding to the detected obstacle in the overhead image G11 generated in step S2 (step S5).

[0042] The operation input unit 214 determines whether or not a swipe operation has been performed on the touch panel (step S6). If a swipe operation has been performed (step S6: YES), the change unit 215 determines whether or not the start coordinates of the swipe operation are included in the detection range stored in the detection range storage unit 231 (step S7). If the start coordinates are not included in the detection range (step S7: NO), the change unit 215 treats the operation as not being a change operation.

[0043] If the start coordinate is included in the detection range (step S7: YES), the change unit 215 changes the detection range based on the start coordinate and end coordinate of the swipe operation, and rewrites the detection range stored in the detection range storage unit 231 (step S8). For example, if the end coordinate of the swipe operation is located in an area other than the detection range among the left rear area Ra, the rear area Rb, the right rear area Rc, and the right front area Rd, the operation input unit 214 sets the area as the detection range and cancels the detection range before the change. Also, for example, if the end coordinate of the swipe operation remains within the detection range, the operation input unit 214 may set the area located in the direction of the swipe operation among the left rear area Ra, the rear area Rb, the right rear area Rc, and the right front area Rd as the detection range, and cancel the detection range before the change.

[0044] If the target range is changed in step S8, if the swipe operation is not performed in step S6 (step S7: NO), or if the start coordinates of the swipe operation are outside the target range (step S7: NO), the display screen generation unit 216 arranges a frame line G13 surrounding the detection range stored in the detection range storage unit 231 on the overhead image G11 (step S9). Then, the display screen generation unit 216 generates display screen data G1 in which the overhead image G11 generated in step S2, the marker G12 arranged in step S5, the frame line G13 arranged in step S9, and the single camera image G14 acquired in step S1 are arranged (step S10). The display control unit 217 outputs the generated display screen data G1 to the display 145D (step S11).

[0045] If no obstacle is detected in the captured image in step S3 (step S3: NO), the alarm control unit 218 stops outputting the audio signal (step S12). Then, the display screen generating unit 216 places a frame G13 surrounding the detection range stored in the detection range storage unit 231 on the overhead image G11 (step S9), and generates display screen data G1 (step S10). The display control unit 217 outputs the generated display screen data G1 to the display 145D (step S11).

[0046] By repeatedly executing the above process, the control device 145 can change the detection range for obstacle detection in response to a change operation by the operator, and perform detection within the detection range after the change.

[0047] 6 is an example, and in other embodiments, not all steps may be executed. For example, in other embodiments, if no notification is given by an alarm, the processes of steps S4 and S12 may not be executed. Also, for example, in other embodiments, if no notification is given by the marker G12, the process of step S5 may not be executed.

[0048] Example of operation An example of the operation of the control device 145 according to the first embodiment will be described below with reference to the drawings. FIG. 7 is a diagram showing an example of the operation of the control device 145 according to the first embodiment. The detection range storage unit 231 of the control device 145 stores the left rear area Ra as the detection range. In this case, the obstacle detection unit 213 determines the presence or absence of an obstacle in the left rear area Ra in step S3. When an obstacle is detected in the left rear area Ra, the display screen generation unit 216 places a marker G12 at a position corresponding to the detected position of the obstacle. The operator recognizes the presence of an obstacle in the left rear area Ra by an alarm issued from the speaker 145S or the marker G12 displayed on the display 145D.

[0049] Here, in order to change the detection range of the obstacle detection, the operator touches the left rear area Ra of the display 145D and performs a swipe operation toward the rear area Rb. When the change unit 215 determines that the start coordinate of the swipe operation is within the current detection range and the end coordinate is within the rear area Rb, it rewrites the detection range stored in the detection range storage unit 231 to the rear area Rb. As a result, the arrangement position of the frame line G13 is switched from the left rear area Ra to the rear area Rb.

[0050] Thereafter, the obstacle detection unit 213 determines the presence or absence of an obstacle in the rear area Rb in step S3. When an obstacle is detected in the rear area Rb, the display screen generation unit 216 places a marker G12 at a position corresponding to the detected position of the obstacle. The operator recognizes the presence of an obstacle in the rear area Rb by hearing the alarm issued from the speaker 145S and viewing the display 145D.

[0051] Actions and Effects The control device 145 according to the first embodiment can change the direction of the detection range of the obstacle detection by a swipe operation by the operator. Changing the direction of the detection range is an example of changing the size of the detection range. This allows the operator to suppress notifications regarding obstacles in areas where detection of obstacles is not required depending on the work content, etc.

[0052] Moreover, the control device 145 according to the first embodiment specifies the changed detection range based on the swipe operation. This allows the operator to change the detection range by an intuitive operation. Moreover, the control device 145 determines whether the start coordinates of the swipe operation are within the detection range, thereby preventing the detection range from being changed due to an erroneous operation.

[0053] <<Variation>> Note that the control device 145 according to the first embodiment changes the detection range by a swipe operation on the overhead image G11, but is not limited to this. Fig. 8 is a diagram showing an example of the operation of the control device 145 according to the modified example of the first embodiment. For example, the display screen data G1 according to the first modified example may include a plurality of single camera images G14 as shown in Fig. 8. The single camera image G14 related to the detection range may be displayed larger than the other single camera images G14. In this case, the control device 145 may change the detection range by performing a swipe operation from the single camera image G14 related to the detection range to the other single camera images G14.

[0054] Second embodiment The control device 145 according to the first embodiment switches the detection range to any one of the left rear area Ra, the rear area Rb, the right rear area Rc, and the right front area Rd. In contrast, the control device 145 according to the second embodiment sets the detection range to any area set by the operator.

[0055] The configuration of the control device 145 according to the second embodiment is similar to that of the first embodiment. The control device 145 according to the second embodiment is different from the first embodiment in the information stored in the detection range storage unit 231, the operation of the obstacle detection unit 213, the change unit 215, and the display screen generation unit 216.

[0056] The detection range storage unit 231 according to the second embodiment stores overhead range data indicating the shape of the detection range in the overhead image G11. The shape of the detection range may be represented, for example, by a polygon made up of multiple vertices, or may be a closed area drawn by a curve. The closed area may be represented, for example, by a Bezier curve. The Bezier curve is represented by the coordinates of the vertices and the control points. The overhead range data is represented by the coordinates of the multiple vertices. The detection range storage unit 231 also stores image range data indicating the detection range in each captured image. Each image range data indicates a portion of the captured image corresponding to the detection range indicated by the overhead-view range data. The image range data is data obtained by converting the overhead-view range data based on a predetermined correspondence between pixels of the captured image and the overhead-view image. In other words, the image range data is data obtained by performing an inverse conversion of the image conversion by the overhead-view image generation unit 212 on the overhead-view range data.

[0057] The obstacle detection unit 213 according to the second embodiment detects an obstacle in each captured image acquired by the acquisition unit 211. The change unit 215 changes the shape of the detection range stored in the detection range storage unit 231 based on the swipe operation received by the operation input unit 214.

[0058] About the display screen FIG. 9 is a diagram showing an example of a display screen according to the second embodiment. Since the shape of the detection range according to the second embodiment is arbitrarily set by the operator, as shown in FIG. 9, the frame line G13 showing the shape of the detection range does not necessarily follow each region Ra-Re. Also, a handle G15 is drawn at each vertex of the frame line G13 of the display screen data G1. The operator changes the shape of the frame line G13, i.e., the shape of the detection range, by moving the handle G15 by a swipe operation. Note that in other embodiments, the frame line G13 or the handle G15 may not be displayed. When the handle G15 is not displayed, the operator changes the shape of the detection range by moving the vertex of the frame line G13 by a swipe operation. Also, in other embodiments, the frame line G13 or the handle G15 may be hidden until a touch operation is performed, and may be displayed by a touch operation.

[0059] <<Method of reporting an obstacle>> FIG. 10 is a flowchart showing the operation of the control device 145 according to the second embodiment. When the control device 145 starts the surroundings monitoring process, the acquisition unit 211 acquires captured images from the multiple cameras 121 (step S1). Next, the overhead image generation unit 212 transforms and combines the multiple captured images acquired in step S1 to generate an overhead image G11 in which the work site is viewed in plan from above, with the work machine 100 at the center (step S2). The obstacle detection unit 213 executes an obstacle detection process for each captured image acquired in step S1, and determines whether or not an obstacle has been detected (step S3).

[0060] If an obstacle is detected in the captured image (step S3: YES), the obstacle detection unit 213 determines whether the detected obstacle is present within the detection range indicated by the image range data stored in the detection range storage unit 231 (step S21). If the detection position of at least one obstacle is within the detection range (step S21: YES), the alarm control unit 218 outputs an alarm audio signal to the speaker 145S (step S4). In addition, the display screen generation unit 216 places a marker G12 at a position corresponding to the detected obstacle in the overhead image G11 generated in step S2 (step S5).

[0061] The operation input unit 214 determines whether or not a swipe operation has been performed on the touch panel (step S6). If a swipe operation has been performed (step S6: YES), the change unit 215 determines whether or not the start coordinates of the swipe operation are near the apex of the overhead range data stored in the detection range storage unit 231 (step S22). If the start coordinates are not near the apex of the overhead range data (step S22: NO), the change unit 215 treats the operation as not being a change operation.

[0062] If the start coordinate is near a vertex of the overhead range data (step S22: YES), modification unit 215 changes the position of the vertex near the start coordinate of the swipe operation to the end coordinate of the swipe operation, and rewrites the overhead range data stored in detection range storage unit 231 (step S23).Modification unit 215 also divides the rewritten overhead range data into regions Ra-Rd, and transforms each of them to generate new image range data, and overwrites the image range data stored in detection range storage unit 231 (step S24).

[0063] If the target range is changed in step S24, if a swipe operation is not performed in step S6 (step S6: NO), or if the start coordinates of the swipe operation are not in the vicinity of the handle (step S22: NO), the display screen generation unit 216 arranges the frame line G13 and the handle G15 indicating the outline of the overhead view range data stored in the detection range storage unit 231 on the overhead view image G11 (step S9). Then, the display screen generation unit 216 generates display screen data G1 in which the overhead view image G11 generated in step S2, the marker G12 arranged in step S5, the frame line G13 and the handle G15 arranged in step S9, and the single camera image G14 acquired in step S1 are arranged (step S10). The display control unit 217 outputs the generated display screen data G1 to the display 145D (step S11).

[0064] If no obstacle is detected in the captured image in step S3 (step S3: NO), the alarm control unit 218 stops outputting the audio signal (step S12). Then, the display screen generating unit 216 arranges the frame line G13 and the handle G15, which indicate the outline of the overhead range data stored in the detection range storage unit 231, on the overhead image G11 (step S9), and generates the display screen data G1 (step S10). The display control unit 217 outputs the generated display screen data G1 to the display 145D (step S11).

[0065] By repeatedly executing the above process, the control device 145 can change the detection range of obstacle detection to an arbitrary shape in response to a change operation by the operator, and can detect the detection range after the change.

[0066] 10 is an example, and in other embodiments, not all steps may be executed. For example, in other embodiments, when no notification is given by an alarm, the control device 145 may not execute the processes of steps S4 and S12. Also, for example, in other embodiments, when no notification is given by the marker G12, the control device 145 may not execute the process of step S5. 10, the control device 145 performs the obstacle detection process for the entire range of each captured image in step S3, and then masks obstacles that exist outside the detection range in step S21, but the control device 145 may perform the obstacle detection process only for the detection range in step S3. In this case, the control device 145 does not need to perform the determination in step S21.

[0067] Example of operation An example of the operation of the control device 145 according to the second embodiment will be described below with reference to the drawings. FIG. 11 is a diagram showing an example of the operation of the control device 145 according to the second embodiment. When the obstacle detection unit 213 of the control device 145 detects an obstacle in the rear area Rb and the right front area Rd in step S3, the obstacle detection unit 213 determines in step S21 whether the detected obstacle is within the detection range. Here, it is determined that both the obstacles in the rear area Rb and the right front area Rd are within the detection range. As a result, the control device 145 attaches a marker G12 to each of the obstacles in the rear area Rb and the right front area Rd.

[0068] After that, the operator narrows the detection range of the right rear area Rc and the right front area Rd by a swipe operation. The control device 145 moves the position of the handle G15 in response to the swipe operation, and changes the shape of the frame line G13.

[0069] After that, when the obstacle detection unit 213 of the control device 145 detects an obstacle in the rear area Rb and the right front area Rd again in step S3, it determines in step S21 whether the detected obstacle is within the detection range. At this time, the right front area Rd is outside the detection range due to the change in the detection range. Therefore, the control device 145 attaches the marker G12 only to the obstacles present in the rear area Rb.

[0070] Actions and Effects The control device 145 according to the second embodiment changes the contour shape of the detection range by a swipe operation. That is, the size of the detection range can be changed. Changing the size of the detection range is an example of changing the size of the detection range. The control device 145 determines whether or not to leave the detected obstacle based on the contour shape of the detection range. This allows the operator to intuitively set the detection range for obstacle detection with a high degree of freedom.

[0071] Other Embodiments Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes are possible. That is, in other embodiments, the order of the above-mentioned processes may be changed as appropriate. Also, some of the processes may be executed in parallel.

[0072] The control device 145 according to the embodiment described above notifies the user of an obstacle by displaying the marker G12 on the display 145D, displaying the alarm icon G13, and sounding an alarm from the speaker 145S, but this is not limited to the above in other embodiments. For example, the control device 145 according to other embodiments may notify the user of an obstacle by performing intervention control of the work machine 100.

[0073] Furthermore, the work machine 100 according to the embodiment described above is a hydraulic excavator, but is not limited to this. For example, the work machine 100 according to other embodiments may be another work machine, such as a dump truck, a bulldozer, or a wheel loader.

[0074] Further, the control device 145 according to the embodiment described above accepts a change operation by a swipe operation, but is not limited to this. For example, the control device 145 according to another embodiment may accept a change operation by a tap operation.

[0075] In addition, in the example of the display screen shown in Figure 5, etc., it has been described that the boundary lines of the left rear area Ra, the rear area Rb, the right rear area Rc, the right front area Rd, and the left front area Re are not displayed on the display screen, but this is not limited to this, and in other embodiments, the boundary lines of the areas may be displayed on the display screen.

[0076] The obstacle detection unit 213 of the control device 145 according to the embodiment described above specifies the area where the obstacle exists, but is not limited to this. For example, the control device 145 according to another embodiment may not specify the area where the obstacle exists. In this case, the control device 145 may specify the obstacle closest to the coordinates where the contact occurred based on an enlargement instruction or a type display instruction, and may enlarge the image around the obstacle, or may display the type of the obstacle near the obstacle.

[0077] Furthermore, the control device 145 according to the above-described embodiment accepts a change operation by a touch operation on a touch panel such as a swipe operation or a tap operation, but is not limited to this. For example, the control device 145 according to another embodiment may be provided with a hard key on the display 145D or outside the display 145D and accept a change operation by operating the hard key.

[0078] First Variation FIG. 12 is a diagram showing an example of the operation of the control device 145 according to the first modified example. For example, the control device 145 according to another embodiment may switch whether each region Ra-Rd is included in the detection range by a touch operation. In this case, the detection range storage unit 231 stores a target flag indicating whether the region is included in the detection range in association with each region Ra-Rd. When the operator taps on the overhead image G11, the value of the target flag associated with the region related to the tapped coordinate is switched ON / OFF. For example, as shown in FIG. 12, when only the left rear region Ra is set as the detection range, if the rear region Rb is tapped, the rear region Rb is switched from the non-detection range to the detection range, and the left rear region Ra and the rear region Rb become the detection range. After that, if the left rear region Ra is tapped, the left rear region Ra is switched from the detection range to the non-detection range, and only the rear region Rb becomes the detection range. In this way, the size of the detection range can be changed by a touch operation on each region.

[0079] Second Variation FIG. 13 is a diagram showing an example of the operation of the control device 145 according to the second modified example. For example, the control device 145 according to another embodiment may switch the detection range in the region Ra-Rd in order for each touch operation. In this case, the detection range storage unit 231 stores the detection range among the regions Ra-Rd, as in the first embodiment. When the operator taps on the overhead image G11, the control device 145 sets another region adjacent to the region currently set as the detection range as the detection range. For example, as shown in FIG. 13, the control device 145 may switch the detection range counterclockwise for each tap operation. In this way, the size of the detection range can be changed for each tap operation.

[0080] Third Variation FIG. 14 is a diagram showing an example of the operation of the control device 145 according to the third modified example. For example, the control device 145 according to another embodiment may set a combination of a plurality of regions as detection range candidates in advance, and switch the detection range among the candidates in order for each touch operation. In the example shown in Fig. 14, four candidates for the size of the detection range are set: only the left rear region Ra, a combination of the left rear region Ra, the rear region Rb, the right rear region Rc, and the right front region Rd, a combination of the right rear region Rc and the right front region Rd, and only the right front region Rd.

[0081] Fourth Variation FIG. 15 is a diagram showing an example of the operation of the control device 145 according to the fourth modified example. In the first modified example, whether or not each of the regions Ra-Rd is set as a detection range is switched by a touch operation on the overhead image G11, but this is not limited to the above. For example, the control device 145 may change the size of the detection range by switching whether or not each of the regions Ra-Rd is set as a detection range by a tap operation on the single camera image G14.

[0082] Fifth Variation FIG. 16 is a diagram showing an example of the operation of the control device 145 according to the fifth modified example. In the second embodiment, the position of the vertex of the frame line G13 shown in the overhead image G11 is moved by a touch operation, but this is not limited to this. For example, the control device 145 may change the size of the detection range of each single camera image G14 by a touch operation on the single camera image G14. For example, in the example shown in FIG. 16, a frame line G13 extending in the horizontal direction is displayed on each single camera image G14. The operator can set the detection range of each single camera image G14 by moving the frame line G14 in the vertical direction by a swipe operation. In the example shown in FIG. 14, when an obstacle exists below the frame line in the single camera image G14, a mark G12 is displayed. In this case, the detection range storage unit 231 may store the height of the detection range related to each single camera image as image range data.

[0083] In the above-described embodiment, the touch operations such as the tap operation described above may be performed with a finger or with a touch pen or the like. [Explanation of symbols]

[0084] REFERENCE SIGNS LIST 100 work machine 110 travelling body 120 revolving body 121 camera 145 control device 211 acquisition unit 212 bird's-eye image generating unit 213 obstacle detection unit 214 operation input unit 215 change unit 216 display screen generating unit 217 display control unit 218 alarm control unit 231 detection range storage unit

Claims

1. An obstacle warning system for a work machine, comprising: A plurality of cameras that capture images of the surroundings of the work machine; A control device; Equipped with The control device includes: generating an overhead image showing the surroundings of the work machine from captured images acquired from a plurality of cameras; displaying, on a display unit, an image in which the overhead image, a figure representing a shape of a detection range in which an obstacle is to be detected, and a marker representing the obstacle are arranged; accepting an operation on the display unit with respect to the overhead image; When the operation on the display unit is an operation for changing the detection range by an operation on the figure included in the image, changing the size of the detection range based on the operation; determining whether or not an obstacle is present in the captured image related to the changed detection range; When it is determined that the obstacle is present within the detection range, a notification indicating the obstacle is given. Obstacle warning system for work machinery.

2. The change in the size of the detection range is the change in the direction of the detection range.

2. An obstacle warning system for a work machine according to claim 1.

3. The change in the size of the detection range is to change the size of the detection range.

2. An obstacle warning system for a work machine according to claim 1.

4. The notification of the obstacle is performed by displaying information indicating the obstacle on a touch panel display, The operation on the display unit is a touch operation on the touch panel display.

4. An obstacle warning system for a work machine according to claim 1.

5. The figure representing the shape of the detection range is a frame line surrounding the detection range arranged on the overhead image. The obstacle warning system according to claim 1 .

6. An obstacle warning method for a work machine equipped with a plurality of cameras that capture images of the surroundings and a control device, comprising: The control device, generating an overhead image showing the surroundings of the work machine from captured images acquired from a plurality of cameras; displaying, on a display unit, an image in which the overhead image, a figure representing a shape of a detection range in which an obstacle is to be detected, and a marker representing the obstacle are arranged; accepting an operation on the display unit with respect to the overhead image; When the operation on the display unit is an operation for changing the detection range by an operation on the graphic included in the image, a size of the detection range is changed based on the operation; determining whether or not an obstacle is present in the captured image related to the changed detection range; When it is determined that the obstacle is present within the detection range, a notification indicating the obstacle is given. A method for notifying an obstacle in a work machine.

Images