Obstacle-to-work-machine notification system and obstacle-to-work-machine notification method

Abstract

To alter an obstacle detection range set in a work machine.SOLUTION: An obstacle determination unit determines whether or not an obstacle exists within a detection range of an object of obstacle detection. When it is determined that the obstacle exists, a notification unit makes a notification indicating the obstacle. An operation input unit accepts an operation performed on a display unit in order to alter the detection range. An alteration unit alters the size of the detection range on the basis of the operation.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present disclosure relates to an obstacle notification system for a work machine and a method for notifying an obstacle of a work machine.

Background Art

[0002] Patent Document 1 discloses a technique related to a peripheral monitoring system that detects a person around a work machine. According to the technique described in Patent Document 1, the peripheral monitoring system detects surrounding obstacles.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When the peripheral monitoring system detects an obstacle, it notifies the presence of the obstacle from a display, a speaker, or the like. The operator of the work machine receives the notification from the peripheral monitoring system, confirms the presence of the obstacle, and also confirms that safety is ensured.

[0005] By the way, depending on the work content of the work machine, it may not be necessary to monitor obstacles over the entire circumference of the work machine. For example, in the loading turning operation onto a dump truck, it may be unnecessary to detect obstacles in the direction where the dump truck is present. An object of the present disclosure is to provide an obstacle notification system for a work machine and a method for notifying an obstacle of a work machine that can change the detection range of an obstacle in the work machine.

Means for Solving the Problems

[0006] According to one aspect of the present invention, a work machine obstacle notification system includes an obstacle determination unit that determines whether an obstacle exists within a detection range of a detection target of the obstacle, a notification unit that performs a notification indicating the obstacle when it is determined that the obstacle exists, an operation input unit that receives an operation on a display unit with respect to a change in the detection range, and a change unit that changes the size of the detection range based on the operation.

Advantages of the Invention

[0007] According to the above aspect, an operator of a work machine can change the detection range of an obstacle in the work machine.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0009] 〈First Embodiment〉 Hereinafter, embodiments will be described in detail with reference to the drawings.

[0010] 《Configuration of Working Machine 100》 FIG. 1 is a schematic diagram showing the configuration of a working machine 100 according to the first embodiment. The working machine 100 operates at a construction site and constructs a construction target such as earth and sand. The working machine 100 according to the first embodiment is, for example, a hydraulic excavator. The working machine 100 includes a traveling body 110, a revolving body 120, a working device 130, and a cab 140. The traveling body 110 supports the working machine 100 so as to be able to travel. The traveling body 110 is, for example, a pair of left and right endless tracks. The revolving body 120 is supported by the traveling body 110 so as to be able to revolve around a revolving center. The working device 130 is driven by hydraulic pressure. The working device 130 is supported by the front part of the revolving body 120 so as to be able to be driven in the vertical direction. The cab 140 is a space for an operator to board and operate the working machine 100. The cab 140 is provided at the left front part of the revolving body 120. Here, the part of the revolving body 120 where the working device 130 is attached is referred to as the front part. Also, with respect to the revolving body 120, the part on the opposite side with respect to the front part is referred to as the rear part, the left side part is referred to as the left part, and the right side part is referred to as the right part.

[0011] 《Configuration of Revolving Body 120》 The revolving body 120 is provided with a plurality of cameras 121 that image the surroundings of the working machine 100. FIG. 2 is a diagram showing the imaging ranges of the plurality of cameras 121 provided in the working machine 100 according to the first embodiment. Specifically, the revolving body 120 is provided with a left rear camera 121A that images the left rear region Ra among the surroundings of the revolving body 120, a rear camera 121B that images the rear region Rb among the surroundings of the revolving body 120, a right rear camera 121C that images the right rear region Rc among the surroundings of the revolving body 120, and a right front camera 121D that images the right front region Rd among the surroundings of the revolving body 120. Note that a part of the imaging ranges of the plurality of cameras 121 may overlap with each other. The imaging ranges of the plurality of cameras 121 cover the range excluding the left front region Re visible from the driver's cab 140 among the entire circumference of the working machine 100. Note that the cameras 121 according to the first embodiment image the left rear, rear, right rear, and right front of the revolving body 120, but are not limited to this in other embodiments. For example, the number and imaging ranges of the cameras 121 according to other embodiments may be different from the examples shown in FIGS. 1 and 2.

[0012] Note that the left rear camera 121A images the left side region and the left rear region of the revolving body 120 as shown in the left rear region Ra of FIG. 2, but may image either one of these regions. Similarly, the right rear camera 121C images the right side region and the right rear region of the revolving body 120 as shown in the right rear range Rc of FIG. 2, but may image either one of these regions. Similarly, the right front camera 121D images the right front region and the right side region of the revolving body 120 as shown in the right front range Rd of FIG. 2, but may image either one of these regions. Also, in other embodiments, the entire circumference of the working machine 100 may be set as the imaging range using the plurality of cameras 121. For example, it may be provided with a left front camera that images the left front range Re and set the entire circumference of the working machine 100 as the imaging range.

[0013] 《Configuration of the working device 130》 The work machine 130 includes a boom 131, an arm 132, a bucket 133, a boom cylinder 131C, an arm cylinder 132C, and a bucket cylinder 133C.

[0014] The base end portion of the boom 131 is attached to the revolving body 120 via a boom pin 131P. The arm 132 connects the boom 131 and the bucket 133. The base end portion of the arm 132 is attached to the tip end portion of the boom 131 via an arm pin 132P. The bucket 133 includes a blade for excavating earth and sand and a storage portion for storing the excavated earth and sand. The base end portion of the bucket 133 is attached to the tip end portion of the arm 132 via a bucket pin 133P.

[0015] The boom cylinder 131C is a hydraulic cylinder for operating the boom 131. The base end portion of the boom cylinder 131C is attached to the revolving body 120. The tip end portion of the boom cylinder 131C is attached to the boom 131. The arm cylinder 132C is a hydraulic cylinder for driving the arm 132. The base end portion of the arm cylinder 132C is attached to the boom 131. The tip end portion of the arm cylinder 132C is attached to the arm 132. The bucket cylinder 133C is a hydraulic cylinder for driving the bucket 133. The base end portion of the bucket cylinder 133C is attached to the arm 132. The tip end portion of the bucket cylinder 133C is attached to a link member connected to the bucket 133.

[0016] 《Configuration of the driver's cab 140》 FIG. 3 is a diagram showing the internal configuration of the driver's cab 140 according to the first embodiment. In the driver's cab 140, a driver's seat 141, an operating device 142, and a control device 145 are provided.

[0017] The operating device 142 is a device for driving the traveling body 110, the slewing body 120, and the working machine 130 by manual operation of the operator. The operating device 142 includes a left operation lever 142LO, a right operation lever 142RO, a left foot pedal 142LF, a right foot pedal 142RF, a left travel lever 142LT, and a right travel lever 142RT.

[0018] The left operation lever 142LO is provided on the left side of the driver's seat 141. The right operation lever 142RO is provided on the right side of the driver's seat 141.

[0019] The left operation lever 142LO is an operating mechanism for performing the slewing operation of the slewing body 120 and the excavation / dumping operation of the arm 132. Specifically, when the operator of the working machine 100 tilts the left operation lever 142LO forward, the arm 132 performs a dumping operation. When the operator of the working machine 100 tilts the left operation lever 142LO backward, the arm 132 performs an excavation operation. When the operator of the working machine 100 tilts the left operation lever 142LO to the right, the slewing body 120 slews to the right. When the operator of the working machine 100 tilts the left operation lever 142LO to the left, the slewing body 120 slews to the left. In other embodiments, when the left operation lever 142LO is tilted in the front-rear direction, the slewing body 120 may slew to the right or left, and when the left operation lever 142LO is tilted in the left-right direction, the arm 132 may perform an excavation operation or a dumping operation.

[0020] The right operation lever 142RO is an operating mechanism for performing the excavation / dumping operation of the bucket 133 and the raising / lowering operation of the boom 131. Specifically, when the operator of the work machine 100 tilts the right operation lever 142RO forward, the lowering operation of the boom 131 is executed. Also, when the operator of the work machine 100 tilts the right operation lever 142RO backward, the raising operation of the boom 131 is executed. Further, when the operator of the work machine 100 tilts the right operation lever 142RO to the right, the dumping operation of the bucket 133 is performed. Also, when the operator of the work machine 100 tilts the right operation lever 142RO to the left, the excavation operation of the bucket 133 is performed. Note that in other embodiments, when the right operation lever 142RO is tilted in the front-rear direction, the bucket 133 may perform a dumping operation or an excavation operation, and when the right operation lever 142RO is tilted in the left-right direction, the boom 131 may perform a raising operation or a lowering operation.

[0021] The left foot pedal 142LF is disposed on the left side of the floor surface in front of the driver's seat 141. The right foot pedal 142RF is disposed on the right side of the floor surface in front of the driver's seat 141. The left travel lever 142LT is pivotally supported by the left foot pedal 142LF and is configured such that the tilt of the left travel lever 142LT and the depression of the left foot pedal 142LF are interlocked. The right travel lever 142RT is pivotally supported by the right foot pedal 142RF and is configured such that the tilt of the right travel lever 142RT and the depression of the right foot pedal 142RF are interlocked.

[0022] The left foot pedal 142LF and the left travel lever 142LT correspond to the rotational drive of the left crawler of the traveling body 110. Specifically, when the operator of the work machine 100 tilts the left foot pedal 142LF or the left travel lever 142LT forward, the left crawler rotates in the forward direction. Also, when the operator of the work machine 100 tilts the left foot pedal 142LF or the left travel lever 142LT backward, the left crawler rotates in the reverse direction.

[0023] The right foot pedal 142RF and the right travel lever 142RT correspond to the rotational drive of the right crawler belt of the traveling body 110. Specifically, when the operator of the work machine 100 tilts the right foot pedal 142RF or the right travel lever 142RT forward, the right crawler belt rotates in the forward direction. Also, when the operator of the work machine 100 tilts the right foot pedal 142RF or the right travel lever 142RT backward, the right crawler belt rotates in the reverse direction.

[0024] The control device 145 includes a display 145D that displays information related to a plurality of functions of the work machine 100. The control device 145 is an example of a display system. Also, the display 145D is an example of a display unit. The input means of the control device 145 according to the first embodiment is a touch panel.

[0025] 《Configuration of Control Device 145》 FIG. 4 is a schematic block diagram showing the configuration of the control device 145 according to the first embodiment. The control device 145 is a computer including a processor 210, a main memory 230, a storage 250, and an interface 270. Also, the control device 145 includes a display 145D and a speaker 145S. Also, the control device 145 according to the first embodiment is provided integrally with the display 145D and the speaker 145S, but in other embodiments, at least one of the display 145D and the speaker 145S may be provided separately from the control device 145. Note that when the display 145D and the control device 145 are provided separately, the display 145D may be provided outside the driver's cab 140. In this case, the display 145D may be a mobile display. Also, when the work machine 100 is driven by remote control, the display 145D may be provided in a remote control room provided remotely from the work machine 100. Similarly, when the speaker 145S and the control device 145 are provided separately, the speaker 145S may be provided outside the driver's cab 140. Also, when the work machine 100 is driven by remote control, the speaker 145S may be provided in a remote control room provided remotely from the work machine 100.

[0026] Note that the control device 145 may be constituted by a single computer, or the configuration of the control device 145 may be divided and arranged among a plurality of computers, and the plurality of computers may cooperate with each other to function as an obstacle notification system for the working machine. The working machine 100 may include a plurality of computers that function as the control device 145. Some of the computers constituting the control device 145 may be mounted inside the working machine 100, and other computers may be provided outside the working machine 100. Note that the above-described single control device 145 is also an example of an obstacle notification system for a working machine. In other embodiments, some of the components constituting the obstacle notification system for the working machine may be mounted inside the working machine 100, and other components may be provided outside the working machine 100. For example, the display 145D may be provided in a remote operation room provided remotely from the working machine 100 as an obstacle notification system for the working machine. In still other embodiments, all of the one or more computers constituting the obstacle notification system for the working machine may be provided outside the working machine 100.

[0027] The camera 121, the display 145D, and the speaker 145S are connected to the processor 210 via the interface 270. Examples of the storage 250 include an optical disk, a magnetic disk, a magneto-optical disk, a semiconductor memory, and the like. The storage 250 may be an internal medium directly connected to the bus of the control device 145, or may be an external medium connected to the control device 145 via the interface 270 or a communication line. The storage 250 stores a program for realizing the surrounding monitoring of the working machine 100. Further, a plurality of images including icons to be displayed on the display 145D are stored in the storage 250 in advance.

[0028] The program may be for realizing a part of the functions to be exerted by the control device 145. For example, the program may exert functions by combination with other programs already stored in the storage 250 or combination with other programs implemented in other devices. In other embodiments, in addition to or instead of the above configuration, the control device 145 may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device). Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, part or all of the functions realized by the processor 210 may be realized by the integrated circuit.

[0029] Also, the storage 250 stores obstacle dictionary data D1 for detecting obstacles. The obstacle dictionary data D1 may be, for example, dictionary data of feature amounts extracted from each of a plurality of known images in which obstacles appear. Examples of feature amounts include HOG (Histograms of Oriented Gradients) and CoHOG (Co-occurrence HOG).

[0030] By executing the program, the processor 210 includes an acquisition unit 211, an aerial image generation unit 212, an obstacle detection unit 213, an operation input unit 214, a change unit 215, a display screen generation unit 216, a display control unit 217, and an alarm control unit 218. Also, by executing the program, the processor 210 secures a storage area of the detection range storage unit 231 in the main memory 230.

[0031] The detection range storage unit 231 stores the detection range that is the target of obstacle detection by the obstacle detection unit 213. The detection range storage unit 231 according to the first embodiment stores any one of the left rear area Ra, the rear area Rb, the right rear area Rc, and the right front area Rd as the detection range.

[0032] The acquisition unit 211 acquires captured images from a plurality of cameras 121. The bird's-eye view image generation unit 212 deforms and synthesizes the plurality of captured images acquired by the acquisition unit 211 to generate a bird's-eye view image of the site as seen from above around the working machine 100. Hereinafter, the captured image deformed by the bird's-eye view image generation unit 212 is also referred to as a deformed image. The bird's-eye view image generation unit 212 may generate a bird's-eye view image by cutting out a part of each captured image after deformation and synthesizing the cut-out captured images. An image of the working machine 100 as seen from above is pre-attached to the center of the bird's-eye view image generated by the bird's-eye view image generation unit 212. That is, the bird's-eye view image is a surrounding image in which the surroundings of the working machine 100 are captured.

[0033] The obstacle detection unit 213 detects obstacles in the captured images acquired by the acquisition unit 211 that show the detection range stored in the detection range storage unit 231. That is, the obstacle detection unit 213 is an example of an obstacle determination unit that determines whether there are obstacles around the working machine 100. Examples of obstacles include people, vehicles, rocks, and the like. Note that the obstacle detection unit 213 according to other embodiments may detect obstacles in each captured image and mask obstacles that are not included in the detection range stored in the detection range storage unit 231.

[0034] The obstacle detection unit 213 detects obstacles, for example, according to the following procedure. The obstacle detection unit 213 extracts feature amounts from each captured image acquired by the acquisition unit 211. The obstacle detection unit 213 detects obstacles from the captured image based on the extracted feature amounts and the obstacle dictionary data. Examples of obstacle detection methods include pattern matching and object detection processing based on machine learning. In the first embodiment, the obstacle detection unit 213 detects a person using the feature amount of an image, but it is not limited to this. For example, in other embodiments, the obstacle detection unit 213 may detect an obstacle based on the measurement value of LiDAR (Light Detection and Ranging) or the like.

[0035] The operation input unit 214 receives an operator's touch operation input to the touch panel of the control device 145. In particular, the operation input unit 214 receives a swipe operation on the touch panel as an operation to change the detection range to the display 145D. The swipe operation refers to an operation of sliding a finger that has touched the touch panel. The operation input unit 214 specifies the start coordinate and the end coordinate related to the swipe operation on the touch panel. In this embodiment, the swipe operation is described as including a flick operation. The change unit 215 rewrites the detection range stored in the detection range storage unit 231 based on the input to the operation input unit 214.

[0036] The display screen generation unit 216 generates display screen data G1 in which a marker G12 indicating the position of the obstacle is arranged at a position corresponding to the detection position of the obstacle, superimposed on the bird's-eye view image G11 generated by the bird's-eye view image generation unit 212. The arrangement of the marker G12 on the display screen data G1 is an example of notifying the presence of the obstacle. Further, the display screen generation unit 216 arranges a frame line G13 representing the detection range stored in the detection range storage unit 231 in the display screen data G1. An example of the display screen will be described later. In other embodiments, the frame line G13 may not be displayed.

[0037] The display control unit 217 outputs the display screen data G1 generated by the display screen generation unit 216 to the display 145D. Thereby, the display 145D displays the display screen data G1. The display control unit 217 is an example of a notification unit. When the obstacle detection unit 213 detects an obstacle, the alarm control unit 218 outputs an alarm audio signal to the speaker 145S. The alarm control unit 218 is an example of a notification unit.

[0038] "Regarding the Display Screen" FIG. 5 is a diagram showing an example of a display screen according to the first embodiment. As shown in FIG. 5, the display screen data G1 includes an aerial image G11, a marker G12, a frame line G13, and a single-camera image G14. The aerial image G11 is an image of the site viewed from above in a plan view. The aerial image G11 has a left-rear area Ra where a deformed image related to the left-rear camera 121A is shown, a rear area Rb where a deformed image related to the rear camera 121B is shown, a right-rear area Rc where a deformed image related to the right-rear camera 121C is shown, a right-front area Rd where a deformed image related to the right-front camera 121D is shown, and a left-front area Re where no image is shown. Note that the boundary lines of the areas of the left-rear area Ra, rear area Rb, right-rear area Rc, right-front area Rd, and left-front area Re are not shown in the aerial image G11.

[0039] The marker G12 indicates the position of an obstacle. Examples of the shape of the marker G12 include a circle, an ellipse, a regular polygon, a polygon, etc. The frame line G13 surrounds those of the left-rear area Ra, rear area Rb, right-rear area Rc, and right-front area Rd that are within the detection range. The single-camera image G14 is a single-camera image captured by one camera 121.

[0040] "Method for Notifying Obstacles" FIG. 6 is a flowchart showing the operation of the control device 145 according to the first embodiment. When the control device 145 starts the surrounding monitoring process, the acquisition unit 211 acquires captured images from a plurality of cameras 121 (step S1). Next, the bird's-eye view image generation unit 212 generates a bird's-eye view image G11 that provides a plan view of the site from above, centered on the work machine 100, by deforming and synthesizing the plurality of captured images acquired in step S1 (step S2). Next, the obstacle detection unit 213 performs an obstacle detection process on the captured images acquired in step S1 that pertain to the detection range stored in the detection range storage unit 231, and determines whether an obstacle has been detected (step S3). For example, when the detection range is the left rear region Ra, the obstacle detection unit 213 performs an obstacle detection process on the captured image captured by the left rear camera 121A.

[0041] If an obstacle is detected in the captured image (step S3: YES), the alarm control unit 218 outputs an alarm audio signal to the speaker 145S (step S4). Further, the display screen generation unit 216 places a marker G12 at a position corresponding to the detected obstacle in the bird's-eye view image G11 generated in step S2 (step S5).

[0042] The operation input unit 214 determines whether a swipe operation has been performed on the touch panel (step S6). If a swipe operation has been performed (step S6: YES), the change unit 215 determines whether the start coordinates of the swipe operation are included in the detection range stored in the detection range storage unit 231 (step S7). If the start coordinates are not included in the detection range (step S7: NO), the change unit 215 treats the operation as not being a change operation.

[0043] When the starting coordinate is included in the detection range (step S7: YES), the changing unit 215 changes the detection range based on the starting coordinate and the ending coordinate of the swipe operation, and rewrites the detection range stored in the detection range storage unit 231 (step S8). For example, when the ending coordinate of the swipe operation is located in an area outside the detection range among the left rear area Ra, the rear area Rb, the right rear area Rc, and the right front area Rd, the operation input unit 214 sets the area as the detection range and cancels the detection range before the change. Also, for example, when the ending coordinate of the swipe operation remains within the detection range, the operation input unit 214 may set, as the detection range, those located in the direction of the swipe operation among the left rear area Ra, the rear area Rb, the right rear area Rc, and the right front area Rd, and cancel the detection range before the change.

[0044] When the target range is changed in step S8, when no swipe operation is performed in step S6 (step S7: NO), or when the starting coordinate of the swipe operation is outside the target range (step S7: NO), the display screen generation unit 216 arranges a frame line G13 surrounding the detection range stored in the detection range storage unit 231 on the bird's-eye view image G11 (step S9). Then, the display screen generation unit 216 generates display screen data G1 in which the bird's-eye view image G11 generated in step S2, the marker G12 arranged in step S5, the frame line G13 arranged in step S9, and the single camera image G14 acquired in step S1 are arranged (step S10). The display control unit 217 outputs the generated display screen data G1 to the display 145D (step S11).

[0045] When no obstacle is detected in the captured image in step S3 (step S3: NO), the alarm control unit 218 stops the output of the audio signal (step S12). Then, the display screen generation unit 216 arranges a frame line G13 surrounding the detection range stored in the detection range storage unit 231 on the bird's-eye view image G11 (step S9), and generates display screen data G1 (step S10). The display control unit 217 outputs the generated display screen data G1 to the display 145D (step S11).

[0046] By repeatedly executing the above processing, the control device 145 can change the detection range of obstacle detection according to a change operation by the operator, and can detect the detection range for the changed detection range.

[0047] Note that the flowchart shown in FIG. 6 is an example, and in other embodiments, it is not necessarily required to execute all steps. For example, in other embodiments, when not notifying by an alarm, the processes of step S4 and step S12 may not be executed. Also, for example, in other embodiments, when not notifying by the marker G12, the process of step S5 may not be executed.

[0048] 《Operation Example》 Hereinafter, an operation example of the control device 145 according to the first embodiment will be described with reference to the drawings. FIG. 7 is a diagram showing an operation example of the control device 145 according to the first embodiment. In the detection range storage unit 231 of the control device 145, the left rear area Ra is stored as the detection range. In this case, the obstacle detection unit 213 determines the presence or absence of an obstacle in the left rear area Ra in step S3. When an obstacle is detected in the left rear area Ra, the display screen generation unit 216 arranges the marker G12 at a position corresponding to the detection position of the obstacle. The operator recognizes the presence of an obstacle in the left rear area Ra by the alarm emitted from the speaker 145S or the marker G12 displayed on the display 145D.

[0049] Here, in order to change the detection range of obstacle detection, the operator touches the left rear area Ra of the display 145D and performs a swipe operation toward the rear area Rb. When the change unit 215 identifies that the start coordinate of the swipe operation is within the current detection range and the end coordinate is in the rear area Rb, the change unit 215 rewrites the detection range stored in the detection range storage unit 231 to the rear area Rb. As a result, the arrangement position of the frame line G13 switches from the left rear area Ra to the rear area Rb.

[0050] Thereafter, the obstacle detection unit 213 determines the presence or absence of an obstacle in the rear area Rb in step S3. When an obstacle is detected in the rear area Rb, the display screen generation unit 216 places a marker G12 at a position corresponding to the detection position of the obstacle. The operator recognizes that there is an obstacle in the rear area Rb by listening to the alarm emitted from the speaker 145S and visually recognizing the display 145D.

[0051] 《Operation and Effect》 The control device 145 according to the first embodiment can change the direction of the detection range of obstacle detection by a swipe operation by the operator. The change in the direction of the detection range is an example of the change in the size of the detection range. Thereby, the operator can suppress the notification related to the obstacle from being made for the range where obstacle detection is not required according to the work content or the like.

[0052] Further, the control device 145 according to the first embodiment specifies the detection range after the change based on the swipe operation. Thereby, the operator can change the detection range by an intuitive operation. In addition, the control device 145 can prevent the detection range from being changed due to a misoperation by determining whether or not the start coordinates of the swipe operation are within the detection range.

[0053] 《Modification Example》 Note that the control device 145 according to the first embodiment changes the detection range by a swipe operation on the bird's-eye view image G11, but is not limited thereto. FIG. 8 is a diagram showing an operation example of the control device 145 according to a modification example of the first embodiment. For example, as shown in FIG. 8, the display screen data G1 according to the first modification example may include a plurality of single camera images G14. And the single camera image G14 related to the detection range may be displayed larger than the other single camera images G14. In this case, the control device 145 may change the detection range by swiping from the single camera image G14 related to the detection range to the other single camera images G14.

[0054] 〈Second Embodiment〉 The control device 145 according to the first embodiment switches the detection range to any one of a left rear region Ra, a rear region Rb, a right rear region Rc, and a right front region Rd. On the other hand, the control device 145 according to the second embodiment sets an arbitrary range by the operator as the detection range.

[0055] The configuration of the control device 145 according to the second embodiment is the same as that of the first embodiment. The control device 145 according to the second embodiment differs from the first embodiment in the information stored in the detection range storage unit 231, and the operations of the obstacle detection unit 213, the change unit 215, and the display screen generation unit 216.

[0056] The detection range storage unit 231 according to the second embodiment stores overhead range data indicating the shape of the detection range in the overhead image G11. The shape of the detection range may be represented by, for example, a polygon composed of a plurality of vertices, or may be a closed region drawn by a curve. The closed region may be represented by, for example, a Bézier curve. The Bézier curve is represented by the coordinates of the vertices and the control points. The overhead range data is represented by the coordinates of a plurality of vertices. In addition, the detection range storage unit 231 stores image range data indicating the detection range in each captured image. Each image range data indicates a portion of the captured image that corresponds to the detection range indicated by the overhead range data. The image range data is data obtained by being converted from the overhead range data based on a predetermined pixel correspondence relationship between the captured image and the overhead image. That is, it is data obtained by performing an inverse transformation on the overhead range data with respect to the image conversion by the overhead image generation unit 212.

[0057] The obstacle detection unit 213 according to the second embodiment detects obstacles in each captured image acquired by the acquisition unit 211. The change unit 215 changes the shape of the detection range stored in the detection range storage unit 231 based on the swipe operation received by the operation input unit 214.

[0058] 《Regarding the display screen》 FIG. 9 is a diagram showing an example of a display screen according to the second embodiment. Since the shape of the detection range according to the second embodiment is arbitrarily set by the operator, as shown in FIG. 9, the frame line G13 indicating the shape of the detection range does not necessarily follow along each of the regions Ra - Re. Further, handles G15 are drawn at each vertex portion of the frame line G13 of the display screen data G1. The operator changes the shape of the frame line G13, that is, the shape of the detection range, by moving the handle G15 by a swipe operation. Note that in other embodiments, the frame line G13 or the handle G15 may not be displayed. When the handle G15 is not displayed, the operator changes the shape of the detection range by moving the vertex portion of the frame line G13 by a swipe operation. Also, in other embodiments, the frame line G13 or the handle G15 may be non - displayed until a touch operation is performed and may be displayed by the touch operation.

[0059] 《Method for Notifying Obstacles》 FIG. 10 is a flowchart showing the operation of the control device 145 according to the second embodiment. When the control device 145 starts the surrounding monitoring process, the acquisition unit 211 acquires captured images from a plurality of cameras 121 (step S1). Next, the bird's - eye view image generation unit 212 generates a bird's - eye view image G11 that views the work site from above around the work machine 100 by deforming and synthesizing the plurality of captured images acquired in step S1 (step S2). The obstacle detection unit 213 performs an obstacle detection process on each of the captured images acquired in step S1 and determines whether an obstacle has been detected (step S3).

[0060] When an obstacle is detected in the captured image (step S3: YES), the obstacle detection unit 213 determines whether the detected obstacle exists within the detection range indicated by the image range data stored in the detection range storage unit 231 (step S21). When the detection position of at least one obstacle is within the detection range (step S21: YES), the alarm control unit 218 outputs an alarm audio signal to the speaker 145S (step S4). Also, the display screen generation unit 216 places a marker G12 at the position corresponding to the detected obstacle in the bird's - eye view image G11 generated in step S2 (step S5).

[0061] The operation input unit 214 determines whether a swipe operation has been performed on the touch panel (step S6). When a swipe operation has been performed (step S6: YES), the change unit 215 determines whether the start coordinate of the swipe operation is near the vertex of the overhead view range data stored in the detection range storage unit 231 (step S22). When the start coordinate is not near the vertex of the overhead view range data (step S22: NO), the change unit 215 treats the operation as not being a change operation.

[0062] When the start coordinate is near the vertex of the overhead view range data (step S22: YES), the change unit 215 changes the position of the vertex near the start coordinate of the swipe operation to the end coordinate of the swipe operation, and rewrites the overhead view range data stored in the detection range storage unit 231 (step S23). Further, the change unit 215 divides the rewritten overhead view range data for each of the regions Ra - Rd, deforms each of them, generates new image range data, and overwrites the image range data stored in the detection range storage unit 231 (step S24).

[0063] When the target range is changed in step S24, when no swipe operation is performed in step S6 (step S6: NO), or when the start coordinate of the swipe operation is not near the handle (step S22: NO), the display screen generation unit 216 arranges the frame line G13 and the handle G15 indicating the outer shape of the overhead view range data stored in the detection range storage unit 231 on the overhead view image G11 (step S9). Then, the display screen generation unit 216 generates display screen data G1 in which the overhead view image G11 generated in step S2, the marker G12 arranged in step S5, the frame line G13 and the handle G15 arranged in step S9, and the single camera image G14 acquired in step S1 are arranged (step S10). The display control unit 217 outputs the generated display screen data G1 to the display 145D (step S11).

[0064] When no obstacle is detected in the captured image in step S3 (step S3: NO), the alarm control unit 218 stops the output of the audio signal (step S12). Then, the display screen generation unit 216 arranges the frame line G13 and the handle G15 indicating the outer shape of the bird's-eye view range data stored in the detection range storage unit 231 on the bird's-eye view image G11 (step S9), and generates the display screen data G1 (step S10). The display control unit 217 outputs the generated display screen data G1 to the display 145D (step S11).

[0065] By repeatedly executing the above processing, the control device 145 can change the detection range of obstacle detection to an arbitrary shape according to the change operation by the operator, and detect the detection range for the changed detection range.

[0066] Note that the flowchart shown in FIG. 10 is an example, and in other embodiments, it is not necessarily required to execute all steps. For example, in other embodiments, when not notifying by an alarm, the control device 145 does not have to execute the processing of steps S4 and S12. Also, for example, in other embodiments, when not notifying by the marker G12, the control device 145 does not have to execute the processing of step S5. Also, in the flowchart shown in FIG. 10, the control device 145 executes the obstacle detection process for the entire range of each captured image in step S3, and then masks what exists outside the detection range in step S21. However, the obstacle detection process may be executed only for the inside of the detection range in step S3. In this case, the control device 145 does not have to execute the determination in step S21.

[0067] <<Operation Example>> Hereinafter, an operation example of the control device 145 according to the second embodiment will be described with reference to the drawings. FIG. 11 is a diagram showing an operation example of the control device 145 according to the second embodiment. When the obstacle detection unit 213 of the control device 145 detects an obstacle in the rear area Rb and the right front area Rd in step S3, in step S21, it determines whether the detected obstacle is within the detection range. Here, it is determined that any obstacle in the rear area Rb and the right front area Rd is within the detection range. Thereby, the control device 145 attaches a marker G12 to each obstacle in the rear area Rb and the right front area Rd.

[0068] After that, the operator narrows the detection ranges of the right rear area Rc and the right front area Rd by a swipe operation. The control device 145 moves the position of the steering wheel G15 and changes the shape of the frame line G13 in response to the swipe operation.

[0069] After that, when the obstacle detection unit 213 of the control device 145 detects an obstacle in the rear area Rb and the right front area Rd again in step S3, in step S21, it determines whether the detected obstacle is within the detection range. At this time, due to the change in the detection range, the right front area Rd is located outside the detection range. Therefore, the control device 145 attaches a marker G12 only to the obstacle existing in the rear area Rb.

[0070] 《Operation and Effect》 The control device 145 according to the second embodiment changes the contour shape of the detection range by a swipe operation. That is, the size of the detection range can be changed. The change in the size of the detection range is an example of the change in the size of the detection range. The control device 145 determines whether to leave the obstacle detected based on the contour shape of the detection range. Thereby, the operator can intuitively and with high freedom set the detection range for obstacle detection.

[0071] 〈Other Embodiments〉 As described above, one embodiment has been described in detail with reference to the drawings, but the specific configuration is not limited to the above, and various design changes and the like are possible. That is, in other embodiments, the order of the above-described processes may be appropriately changed. Also, some processes may be executed in parallel.

[0072] The control device 145 according to the above-described embodiment performs notification of an obstacle by displaying the marker G12 on the display 145D, displaying the alarm icon G13, and sounding an alarm from the speaker 145S, but is not limited thereto in other embodiments. For example, the control device 145 according to another embodiment may perform notification of an obstacle by intervention control of the work machine 100.

[0073] Also, the work machine 100 according to the above-described embodiment is a hydraulic excavator, but is not limited thereto. For example, the work machine 100 according to another embodiment may be another work machine such as a dump truck, a bulldozer, or a wheel loader.

[0074] Also, the control device 145 according to the above-described embodiment accepts a change operation by a swipe operation, but is not limited thereto. For example, the control device 145 according to another embodiment may accept a change operation by a tap operation.

[0075] Also, in the example of the display screen shown in FIG. 5 and the like, it has been described that the boundary lines of the regions of the left rear region Ra, the rear region Rb, the right rear region Rc, the right front region Rd, and the left front region Re are not displayed on the display screen, but is not limited thereto, and in other embodiments, the boundary lines of the regions may be displayed on the display screen.

[0076] The obstacle detection unit 213 of the control device 145 according to the above-described embodiment identifies the region where the obstacle exists, but is not limited thereto. For example, the control device 145 according to another embodiment may not identify the region where the obstacle exists. In this case, the control device 145 may identify the obstacle closest to the coordinates where the contact occurred based on an enlargement instruction or a type display instruction, and enlarge around the obstacle, or may display the type of the obstacle in the vicinity of the obstacle.

[0077] In addition, the control device 145 according to the above-described embodiment accepts a change operation by a touch operation on a touch panel such as a swipe operation or a tap operation, but is not limited thereto. For example, the control device 145 according to another embodiment may be provided with a hard key on the display 145D or outside the display 145D and accept a change operation by operating the hard key.

[0078] 《First Modification Example》 FIG. 12 is a diagram showing an operation example of the control device 145 according to the first modification example. For example, the control device 145 according to another embodiment may switch whether or not to include each region Ra-Rd in the detection range by a touch operation. In this case, the detection range storage unit 231 stores a target flag indicating whether or not the region is included in the detection range in association with each region Ra-Rd. When the operator taps the bird's-eye view image G11, the ON / OFF of the value of the target flag associated with the region corresponding to the tapped coordinates is switched. For example, as shown in FIG. 12, when only the left rear region Ra is set in the detection range and the rear region Rb is tapped, the rear region Rb is switched from the non-detection range to the detection range, and the left rear region Ra and the rear region Rb are in the detection range. Thereafter, when the left rear region Ra is tapped, the left rear region Ra is switched from the detection range to the non-detection range, and only the rear region Rb is in the detection range. In this way, the size of the detection range can be changed by a touch operation on each region.

[0079] 《Second Modification Example》 FIG. 13 is a diagram showing an operation example of the control device 145 according to the second modification example. For example, the control device 145 according to another embodiment may sequentially switch the detection range among regions Ra - Rd for each touch operation. In this case, similar to the first embodiment, the detection range storage unit 231 stores which of the regions Ra - Rd is the detection range. When the operator taps the bird's-eye view image G11, the control device 145 sets another region adjacent to the currently detected range region as the detection range. For example, as shown in FIG. 13, the control device 145 may switch the detection range counterclockwise for each tap operation. In this way, the size of the detection range can be changed for each tap operation.

[0080] 《Third Modification Example》 FIG. 14 is a diagram showing an operation example of the control device 145 according to the third modification example. For example, the control device 145 according to another embodiment may set combinations of a plurality of regions in advance as detection range candidates, and sequentially switch the detection range among the candidates for each touch operation. In the example shown in FIG. 14, as candidates for the size of the detection range, only the left - rear region Ra, the combination of the left - rear region Ra, the rear region Rb, the right - rear region Rc, and the right - front region Rd, the combination of the right - rear region Rc and the right - front region Rd, and only the right - front region Rd are set.

[0081] 《Fourth Modification Example》 FIG. 15 is a diagram showing an operation example of the control device 145 according to the fourth modification example. Also, in the first modification example, it is switched whether each of the regions Ra - Rd is set as the detection range by a touch operation on the bird's - eye view image G11, but it is not limited to this. For example, the control device 145 may change the size of the detection range by switching whether to set it as the detection range by a tap operation on the single - camera image G14.

[0082] 《Fifth Modification Example》 FIG. 16 is a diagram showing an operation example of the control device 145 according to the fifth modification example. In addition, in the second embodiment, the position of the vertex of the frame line G13 shown in the bird's-eye view image G11 is moved by a touch operation, but it is not limited to this. For example, the control device 145 may change the size of the detection range of each single-camera image G14 by a touch operation on the single-camera image G14. For example, in the example shown in FIG. 16, a frame line G13 extending in the horizontal direction is displayed in each single-camera image G14. The operator can set the detection range of each single-camera image G14 by moving the frame line G14 in the vertical direction by a swipe operation. In the example shown in FIG. 14, when there is an obstacle below the frame line in the single-camera image G14, a mark G12 is displayed. In this case, the detection range storage unit 231 may store the height of the detection range related to each single-camera image as image range data.

[0083] In the above-described embodiment, the touch operation such as the tap operation described may be performed with a finger or may be performed using a touch pen or the like.

Explanation of Reference Numerals

[0084] 100... working machine 110... traveling body 120... slewing body 121... camera 145... control device 211... acquisition unit 212... bird's-eye view image generation unit 213... obstacle detection unit 214... operation input unit 215... change unit 216... display screen generation unit 217... display control unit 218... alarm control unit 231... detection range storage unit

Claims

1. an obstacle determination unit that determines whether an obstacle is present within a detection range of an obstacle detection target; a notification unit that issues a notification indicating the obstacle when it is determined that the obstacle is present; an operation input unit that receives an operation on a display unit in response to a change in the detection range; a change unit that changes the size of the detection range based on the operation; An obstacle warning system for a work machine comprising:

2. The change in the size of the detection range is the change in the direction of the detection range.

2. An obstacle warning system for a work machine according to claim 1.

3. The obstacle warning system for a work machine according to claim 1 , wherein the change in the size of the detection range is a change in the size of the detection range.

4. The notification unit displays information indicating the obstacle on a touch panel display, 4. The obstacle warning system for a work machine according to claim 1, wherein the operation input unit receives a touch operation on the touch panel display.

5. The change unit changes the size of the detection range by switching the detection range to one of a plurality of detection range candidates based on the operation.

5. An obstacle warning system for a work machine according to any one of claims 1 to 4.

6. the detection range is constituted by imaging ranges of a plurality of imaging devices which image the surroundings of the work machine, The obstacle determination unit determines whether or not the obstacle is present based on image data captured by a plurality of imaging devices that capture images of the surroundings of the work machine, the operation input unit accepts an operation to select the detection range corresponding to at least one of the plurality of imaging devices; The change unit switches whether or not the selected detection range is used to determine the obstacle.

2. An obstacle warning system for a work machine according to claim 1.

7. A step of determining whether an obstacle is present within a detection range of an obstacle detection target; a step of notifying the presence of the obstacle when it is determined that the obstacle is present; and a step of accepting an operation on a display unit to change the detection range. changing a size of the detection range based on the operation; An obstacle warning method for a work machine having the above-mentioned configuration.

Images