Surroundings monitoring system for excavators

Abstract

To provide a periphery monitoring system for a shovel which enables a space that cannot be captured by a camera to be displayed on a bird's-eye video image.SOLUTION: A periphery monitoring system for a shovel calculates relative positions between a work device 307 and a first camera and between the work device 307 and a second camera on the basis of output from a plurality of attitude sensors 106 by a controller 110, calculates a shielding region 703 that is a region shielded by the work device on the first camera video image on the basis of the relative position between the first camera and the work device and the angle of the first camera, extracts a shielding region video image 708 that is a video image in a part corresponding to the shielding region in a second camera video image 702 photographed by the second camera on the basis of the relative position between the second camera 312 and the work device, and the angle of the second camera, and synthesizes the shielding region video image in the shielding region on the first camera video image when generating a bird's-eye video image 803.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a surroundings monitoring system for an excavator that displays an image of the surroundings of an excavator operating at a construction site or the like. [Background technology]

[0002] A surroundings monitoring system for an excavator includes a plurality of cameras that capture the surrounding conditions of the work machine (excavator), a surroundings monitoring monitor that displays overhead images and single camera images captured by the plurality of cameras, and a display control unit that displays, on the overhead image and single camera image, direction guide information that extends in a straight line from the center of rotation of the upper rotating body toward the periphery of the upper rotating body (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2016 / 159012 Summary of the Invention [Problem to be solved by the invention]

[0004] Generally, in a surroundings monitoring system for an excavator, the surroundings of the vehicle are photographed using a camera mounted on the vehicle (excavator), and then the captured camera video (images) is subjected to image transformation to generate video that can be used to monitor the surroundings of the vehicle. For example, this system can convert and combine video from multiple cameras to create a video that looks like a view of the surroundings of the vehicle from above, i.e., a bird's-eye view video. Here, the process for creating a bird's-eye view video is called bird's-eye view synthesis processing.

[0005] In a perimeter monitoring system for excavators, the front work implement (work implement), which includes a boom, arm, and attachment (e.g., a bucket), may appear in the camera's image. In this case, the presence of the work implement results in the presence of a space within the field of view that cannot be captured by the camera (e.g., the ground surface, or the space on the opposite side of the camera across from the work implement (hereinafter referred to as the "out-of-capture space"). In this paper, the area of ​​the camera's image that is blocked by the work implement (the area where the work implement is located) is referred to as the "occluded area." If an occluded area exists in the camera image, the occluded area will also exist in the overhead image obtained by applying overhead synthesis processing to the camera image. Therefore, if an occluded area exists in the camera image and a worker is present in the out-of-capture space corresponding to that occluded area, it becomes difficult to determine the worker's presence from the overhead image.

[0006] An object of the present invention is to provide a surroundings monitoring system for an excavator that can display the space outside the camera's capture area on an overhead image. [Means for solving the problem]

[0007] The present application includes a plurality of means for solving the above-mentioned problems. One example of such means is a system including a plurality of cameras that capture images of a working device of a shovel and its surroundings from different directions, and a system for generating an overhead image of the surroundings of the shovel by synthesizing images captured by the plurality of cameras. pictureand a controller that displays on a monitor, a data output device that outputs data used by the controller when calculating the relative position between each of the plurality of cameras and the work device, wherein the controller calculates the relative position between each of the plurality of cameras and the work device based on the data output from the data output device, calculates a blocked area that is an area blocked by the work device in the first image captured by the first camera based on the relative position between a first camera of the plurality of cameras and the work device and the mounting angle of the first camera, extracts a blocked area image that is an image of a portion of the second image captured by the second camera based on the relative position between a second camera of the plurality of cameras that captures the blocked area and the work device and the mounting angle of the second camera, and combines the blocked area image with the blocked area in the first image when generating the overhead image. [Effects of the Invention]

[0008] According to the present invention, the space outside the capture range of the first camera is photographed by the second camera, and the images from the first camera and the second camera are combined to generate an overhead image, so that the space outside the capture range of the first camera can be displayed on the overhead image. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic configuration diagram of a hydraulic excavator according to a first embodiment. [Figure 2] FIG. 1 is a configuration diagram of a surroundings monitoring system for an excavator according to a first embodiment. [Figure 3] FIG. 10 is a diagram showing the flow of a bird's-eye view video generation flow executed by the controller 110 (processor 110a). [Figure 4] 4 is an explanatory diagram of a boom angle 401, an arm angle 402, and a bucket angle 403. [Figure 5] 10A and 10B are explanatory diagrams of a masked area, an out-of-capture area, and a masked area image in a camera image. [Figure 6] FIG. 7 is an explanatory diagram (first embodiment) of a process for generating an overhead video by combining two camera videos 701 and 702. [Figure 7] FIG. 2 is a diagram showing an example of an overhead image displayed on a monitor 104. [Figure 8] FIG. 7 is an explanatory diagram (second embodiment) of a process for generating an overhead video by combining two camera videos 701 and 702 when there is an area within the blocked area where no blocked area video exists. [Figure 9] FIG. 10 is a configuration diagram of a surroundings monitoring system for an excavator according to a third embodiment. [Figure 10] 10 is a diagram showing an example of the correspondence between the occluded area of ​​a certain camera 310 and the posture of a working device 307, which is stored in the area candidate database 105. FIG. [Figure 11] FIG. 10 is a diagram showing a three-dimensional model of a shovel used in the fourth embodiment. [Figure 12] FIG. 10 is a diagram showing an example of a camera image 1001 of a first virtual camera 905. [Figure 13] FIG. 10 is a configuration diagram of a surroundings monitoring system for an excavator according to a fifth embodiment. [Figure 14] FIG. 11 is a left side view of a shovel 301 according to a fifth embodiment. [Figure 15] FIG. 10 is a configuration diagram of a surroundings monitoring system for an excavator according to a sixth embodiment. [Figure 16] FIG. 12 is a diagram showing a portion of a plurality of matching images 1201, 1202, and 1203 captured by the first camera 311 and stored in the matching image database 118. [Figure 17] FIG. 13 is a configuration diagram of a surroundings monitoring system for an excavator according to a seventh embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] First Embodiment In this embodiment, an example of a surroundings monitoring system for an excavator that can suppress a decrease in visibility of obstacles present in an obstructed area even if an obstructed area occurs in a camera video (image) capturing the area around the hydraulic excavator is described.

[0012] Fig. 1 is a schematic diagram of a hydraulic excavator according to this embodiment. The excavator 301 in Fig. 1 includes a lower traveling body 302, an upper rotating body 303 rotatably attached to the upper part of the lower traveling body 302, a working implement (front working implement) 307 attached to the upper rotating body 303, and multiple cameras 310 (311, 312, 313, 314, 315) attached to the upper rotating body 303 for capturing images of the surrounding area from different directions.

[0013] The work device 307 includes a boom 304 whose base end is rotatably attached to the upper rotating body 303, an arm 305 rotatably attached to the tip of the boom 304, and a bucket (attachment) 306 rotatably attached to the tip of the arm 305. An attitude sensor (boom angle sensor) 106a that detects the angle of the boom 304 is provided on the rotation shaft on the base end side of the boom 304, an attitude sensor (arm angle sensor) 106b that detects the angle of the arm 305 is provided on the rotation shaft on the tip side of the boom 304, and an attitude sensor (bucket angle sensor) 106c that detects the angle of the bucket 306 is provided on the rotation shaft on the tip side of the arm 305. Hereinafter, the three attitude sensors 106a, 106b, and 106c may be collectively referred to as attitude sensor 106.

[0014] A plurality of cameras 310, numbered a first camera to an n-th camera, are mounted on the upper rotating body 303. Each camera is mounted on the upper surface of the upper rotating body 303 and captures the surroundings of the upper rotating body 303. In order to synthesize an overhead image, the positions and lens directions of the mounted cameras 310 are determined according to the body shape of the excavator 301. In the example of FIG. 1 , a total of five cameras 311, 312, 313, 314, and 315 are mounted on the upper surface of the upper rotating body 303 as the cameras 310. Of these, the first camera 311 is mounted on the front left side of the upper rotating body 303, the second camera 312 is mounted on the front right side of the upper rotating body 303, the third camera 313 is mounted in the left central part of the upper rotating body 303, the fourth camera 314 is mounted in the right central part of the upper rotating body 303, and the fifth camera 315 is mounted in the rear central part of the upper rotating body 303. However, to implement this embodiment, it is sufficient that the first camera 311 and the second camera 312 capable of photographing the left and right sides of the working device 307 are mounted.

[0015] 2 is a configuration diagram of a perimeter monitoring system for an excavator according to this embodiment. Note that the same components as those in the previous drawings are designated by the same reference numerals and their explanations may be omitted, and this also applies to subsequent drawings. The perimeter monitoring system according to this embodiment includes multiple cameras 310, a controller 110, multiple attitude sensors 106, and a monitor 104.

[0016] Each camera 310 (cameras 311, 312, 313, 314, 315) outputs the captured images (camera images) to the controller 110 one by one.

[0017] 2 (for example, the multiple cameras 310, the multiple attitude sensors 106, and the monitor 104) via communication lines, a processor 110a such as a CPU that executes various calculations, and a storage device 110b (for example, a semiconductor memory such as a ROM or RAM, or a magnetic storage device such as an HDD (neither of which are shown)) that stores programs that can be executed by the processor 110a and various data. A microcomputer, for example, can be used as the controller 110.

[0018] The controller 110 (processor 110a) synthesizes images captured by multiple cameras 310 (five cameras 311, 312, 313, 314, and 315 in this embodiment) to generate an overhead image 880 (see the lower diagram in Figure 7 described below) of the area around the shovel 301, and displays the generated overhead image 880 on the monitor 104.

[0019] For example, processor 110a applies viewpoint conversion processing to each camera image to convert the viewpoint (image) from which each camera 310 views the area around shovel 301 from diagonally above into a viewpoint (image) from directly above the area, thereby generating the same number of overhead images as cameras 310, and can generate overhead image 880 of the area around the shovel by synthesizing the overhead images of each camera 310. The processing of generating a plurality of overhead images by converting the viewpoint of the image of each camera 310 in this way and synthesizing the plurality of overhead images to generate overhead image 880 of the area around the shovel is called overhead synthesis processing.

[0020] (Data output device 120A) The multiple attitude sensors 106 constitute a data output device 120A that outputs data (hereinafter sometimes referred to as "relative position calculation data") used by the controller 110 when calculating the relative position between each of the multiple cameras 310 and the working implement 307. The controller 110 calculates the relative position between each of the multiple cameras 310 and the working implement 307 based on the data output from the data output device 120A. That is, the controller 110 inputs detection signals (relative position calculation data) from the three attitude sensors 106a, 106b, and 106c to calculate the angles of the boom 304, arm 305, and bucket 306 (boom angle 401, arm angle 402, bucket angle 403 (see FIG. 4)), and calculates the attitude of the working implement 307 relative to the upper rotating body 303 based on each calculated angle (for example, the attitude of the working implement 307 in a coordinate system set for the upper rotating body 303 (sometimes referred to as a vehicle body coordinate system)). Furthermore, controller 110 calculates the relative position between working device 307 and first camera 311 and the relative position between working device 307 and second camera 312 based on the dimensions of boom 304, arm 305, and bucket 306, and the mounting positions and mounting angles of first camera 311 and second camera 312 relative to upper rotating body 303 (for example, the mounting positions and mounting angles of each camera 311, 312 in the vehicle body coordinate system), which are stored in a storage device within controller 110. The relative positions include the distances from each camera 311, 312 to each front member (boom 304, arm 305, and bucket 306) that constitutes working device 307.

[0021] (flowchart) 3 is a diagram showing the flow of a bird's-eye view image generation flow executed by the controller 110 (processor 110a). Note that, to simplify the explanation, the following describes the part where an bird's-eye view image of the front of the shovel 301 is generated using images from two cameras (first camera 311 and second camera 312) attached to the front of the upper rotating body 303 and on the left and right sides of the work implement 307, but an bird's-eye view image 880 (see FIG. 7) of the entire circumference of the shovel 301 may be generated using images from the remaining camera 310.

[0022] When the process starts, in step 210, the processor 110a acquires camera images from each camera attached to the upper rotating body 303 (for example, the first camera 311 and the second camera 312).

[0023] In step 215, the processor 110a calculates the current attitude of the work implement 307 based on the detection signals of the three attitude sensors 106a, 106b, and 106c. Specifically, the processor 110a calculates the angles of the boom 304, the arm 305, and the bucket 306 about their rotation axes (boom angle 401, arm angle 402, and bucket angle 403). Examples of these angles are shown in FIG. 4. This figure shows the excavator 301 as seen from the left side. The boom 304 is attached to the upper rotating body 303 and rotates vertically. The angle formed between the upper rotating body 303 and the boom 304 during this operation is called the boom angle 401. Similarly, the arm 305 is attached to the boom 304 and rotates vertically. The angle formed between the boom 304 and the arm 305 during this operation is called the arm angle 402. Furthermore, the bucket 306 is attached to the arm 305 and rotates vertically. The angle formed between the arm 305 and the bucket 306 during this operation is called the bucket angle 403. Note that depending on the type of shovel 301, there may be models in which the angles and distances change in other parts. For these models, it is sufficient to detect the changeable angles and distances using an appropriate attitude sensor 106.

[0024] After calculating boom angle 401, arm angle 402, and bucket angle 403 (i.e., posture data of working implement 307), processor 110a calculates the relative position of first camera 311 and working implement 307 (i.e., the position of working implement 307 with first camera 311 as the reference) and the relative position of second camera 312 and working implement 307 (i.e., the position of working implement 307 with second camera 312 as the reference). After this, processor 110a moves the process to step 220.

[0025] In step 220, the processor 110a calculates an occluded area (referred to as the first occluded area) 703 caused by the working device 307 in the first camera image 701 captured by the first camera 311, and an occluded area (referred to as the second occluded area) 704 caused by the working device 307 in the second camera image 702 captured by the second camera 312, based on the relative position data between the first and second cameras 311, 312 and the working device 307 calculated in step 215.

[0026] This pattern is shown in Figure 5. Arm 305 and bucket 306 are reflected in first camera image 701 captured by first camera 311. Based on the relative positions of the first camera and work implement 307 calculated in step 215 and the mounting angle of first camera 311 with respect to upper rotating body 303 (and the angle of view of first camera 311 (i.e., the focal length of the lens) may also be added), processor 110a calculates this area on first camera image 701 as first shielded area 703. Note that first shielded area 703 may be defined by the outermost contour (outermost contour) of the contours of work implement 307 appearing on first camera image 701, or may be any area on first camera image 701 that includes the outermost contour of work implement 307.

[0027] The space shielded by the working device 307 (arm 305 and bucket 306 in the example of FIG. 5) located in the first shielded area 703, i.e., the out-of-capture space (first out-of-capture space) 706, cannot be seen in the first camera image 701. In contrast, unlike the first camera 311 located on the left side of the working device 307, the second camera 312 is located on the right side of the working device 307, and therefore the first out-of-capture space 706 is captured in the second camera image 702. This is referred to as the out-of-capture space (first out-of-capture space) 706 derived from the first camera. Processor 110a calculates the position of first out-of-capture space 706 on the ground surface based on the relative positions of first camera 311 and working implement 307, the mounting angle of first camera 311, and the shape of the terrain in front of upper rotating body 303 (for example, a plane parallel to the bottom surface of lower running body 302 may be assumed to be the terrain, or three-dimensional shape data of the terrain obtained by three-dimensional surveying or the like may be input). First out-of-capture space 706 can be calculated by a known method, but for example, it can be defined by a set of points where a straight line passing through each point on the outermost contour of working implement 307 visible from first camera 311 and the position of first camera 311 intersects with the terrain.

[0028] Furthermore, processor 110a calculates the area in which first uncaptured space 706 is captured in second camera image 702 (i.e., first shielded area image 708, which is an image of the part of second camera image 702 corresponding to first shielded area 703) based on the calculated position of first uncaptured space 706, the relative position of the second camera and work device 307 calculated in step 215, and the mounting angle of second camera 312 relative to upper rotating body 303 (and may also include the angle of view of second camera 312 (i.e., the focal length of the lens)).

[0029] In addition, by processing similar to that described above, processor 110a calculates a second shielded area 704 in second camera image 702, a second non-captured space 705 on the ground surface corresponding to second shielded area 704, and the area in first camera image 701 in which second non-captured space 705 is captured (i.e., second shielded area image 707, which is an image of the part of first camera image 701 corresponding to second shielded area 704).

[0030] In step 225, processor 110a converts the camera images input from each camera 310 into overhead images. This conversion process includes applying projective transformation to each camera image to create an image (overhead image) that looks like it's seen from directly above. An example of this process is shown in FIG. 6. FIG. 6 is a diagram showing an example of a process for generating a portion of an overhead image (a portion obtained by combining two images 803 and 804) from camera images 701 and 702 of two cameras 311 and 312. By applying projective transformation to first camera image 701 and second camera image 702, respectively, shown in the first row of the figure, first camera overhead image 801 and second camera overhead image 802 are obtained (see the second row of the figure). These overhead images 801 and 802 constitute images equivalent to those seen from above in their respective ranges. The contour shapes of the blocked areas 703 and 704 and the blocked area images 707 and 708 in the camera overhead images 801 and 802 also change from the camera images 701 and 702 before conversion.

[0031] In step 230, processor 110a extracts a second shielded area image (second alternative image) 707 from the first camera overhead image 801 and a first shielded area image (first alternative image) 708 from the second camera overhead image 802, and in step 235, combines the extracted second shielded area image 707 with the second shielded area 704 and combines the extracted first shielded area image 708 with the first shielded area 703.

[0032] When combining the masked area image 708 (707) and the masked area 703 (704), the following rendering process may be performed. That is, in this rendering process, the masked area image 708 (707) and the masked area 703 (704) are blended at a desired ratio and combined. For example, if the blending ratio of the first masked area image 708 is r, the blending ratio of the first masked area 703 can be (1 - r). For example, r may be set to a value greater than 0.5 (e.g., r = 0.8) to increase the weight of the first masked area image 708 (alternative image). Note that r can take a value between 0 and 1, and, for example, r = 1 may be set to render only the masked area image (alternative image). The result of blending the masked area image 707 (708) with the masked area 703 (704) in this way is shown as the masked areas 703 and 704 in the cut-out images 803 and 804 in the third row of Figure 6, which will be described later. That is, the images of the arm 305 and bucket 306 in the shielded areas 703 and 704 are drawn lighter than those in the overhead camera images 801 and 802 on the second stage, and instead, the shielded area images 707 and 708 depict the state of the outside captured space 705 and 706 that was shielded by the work device 307 on the overhead image.

[0033] In step 240, the processor 110a cuts out portions of the first and second camera overhead images 801 and 802 combined in step 235 to create, for example, rectangular or square first and second cropped images 803 and 804, and combines these images by arranging them in accordance with the coordinate system of the ground surface of the work site, as shown in the fourth row of FIG. 6. This makes it possible to generate an overhead image that displays a wider range around the shovel than an overhead image from a single camera. Note that, as shown in FIG. 6, when combining camera overhead images generated from images from two cameras 310 adjacent to each other around the shovel 301, such as the first camera 311 and the second camera 312, there is a possibility that there will be an overlapping area between the two camera overhead images (overlap area 805 in the example of FIG. 6). Regarding this overlapping area, it is possible to apply a method of cutting out and displaying one of the two camera overhead images, or blending and displaying both camera overhead images at a desired ratio, as described above. In step 240, processor 110a combines the camera overhead images from all cameras 310 to generate overhead image 880 as shown in Fig. 7, and then proceeds to step 245. An image (icon) 806 of a top view of shovel 301 is placed in the center of overhead image 880 in Fig. 7. Note that the outlines (broken lines) of shielded areas 704 and 703 shown in Figs. 5 and 6 are not displayed in the overhead image in the first row of Fig. 7.

[0034] In step 245, the processor 110a displays the overhead image 880 of the surroundings of the shovel 301, which was synthesized in step 240, on the monitor 104.

[0035] (effect) By performing the above-described processing, by combining the images 701 and 702 from the two cameras 311 and 312 located on the left and right of the work device 307, it becomes possible to display images of the uncaptured spaces 705 and 706 blocked by the work device 307 (blocked area images 708 and 707) on the overhead image 880. This makes it possible to accurately and quickly grasp the situation around the shovel 301.

[0036] In particular, with shovel 301, the dimensions of each part 304, 305, 306 of working device 307 and the mounting position and specifications of each camera 311, 312 are known, and the position of working device 307 based on the positions of each camera 311, 312 can be accurately calculated using data output device 120A (attitude sensor 106). This makes it possible to accurately calculate the positions of out-of-capture spaces 705, 706, and ultimately to accurately extract shielded area images 707, 708 from each camera image 701, 702. In other words, highly accurate shielded area images 707, 708 can be displayed on overhead image 880.

[0037] 7, the upper revolving body 303 and lower traveling body 302 of the shovel 301 are shown as images (icons) 806, but the work equipment 307 is displayed as it is captured in the camera overhead images 801, 802 (cut-out images 803, 804) on the overhead image 880. Displaying the work equipment 307 on the overhead image 880 in this way makes it easy to grasp the position and movement of the work equipment 307 via the overhead image 880.

[0038] The data output device that outputs relative position calculation data to the controller 110 is not limited to the data output device 120A that uses the attitude sensor 106 described above, but may be any of a number of data output devices described below.

[0039] In the above description, the calculation of the occluded area and the identification of the position of the occluded area image are performed in step 220 before the generation of the overhead image of each camera in step 225, but this order may be reversed. That is, the calculation of the occluded area and the identification of the position of the occluded area image may be performed on the overhead image after the overhead image is generated. This is also true for other embodiments.

[0040] Second Embodiment In the first embodiment, a case where a shielded area image exists that covers the entire shielded area (for example, a case where a first shielded area image 708 exists that covers the entire first shielded area 703 as shown in FIG. 6) has been described, but in reality, there may be cases where a shielded area image does not exist in part of the shielded area. In this case, the part of the shielded area where the shielded area image does not exist may be displayed differently from the other parts. This embodiment will be described using FIG. 8.

[0041] FIG. 8 is a diagram showing an example of a process for generating a portion of an overhead image 880 (a portion obtained by combining two images 803 and 804) from camera images 701 and 702 of two cameras 311 and 312 (however, there is an area 709 within the blocked area where no blocked area image exists). In this diagram, the area 709 indicated by diagonal lines within the first and second blocked areas 703 and 704 of the first and second camera images 701 and 702 is an area where no blocked area image exists. In this embodiment, the area 709 is highlighted by hatching to differentiate it from other areas. Other methods for highlighting the area 709 include drawing an outline with a thicker line than the other areas or adding color. Highlighting the area 709 within the blocked area where no blocked area image exists can prompt a viewer of the overhead image 880 to recognize the existence of the area 709 and to recognize that sufficient caution is required when operating the shovel 301.

[0042] It goes without saying that this embodiment can also be applied to embodiments other than the first embodiment described below.

[0043] Third Embodiment In the first embodiment, the blocked area on each camera image is calculated using the relative positions of the working device 307 and each camera 310 (first and second cameras 311, 312), but the blocked area on each camera image may also be calculated from the attitude of the working device 307 calculated from the detection signal of the attitude sensor 106. This embodiment will be described with reference to FIGS. 9 and 10.

[0044] 9 is a configuration diagram of a perimeter monitoring system for an excavator according to this embodiment. In addition to the configuration shown in FIG. 2, the perimeter monitoring system according to this embodiment includes an area candidate database 105 that defines the correspondence between the blocked area in the camera image of each camera 310 and the posture of the work implement 307. The area candidate database 105 may be provided in a storage device (not shown) of the controller 110, or in an external storage device of the controller 110 (for example, a storage medium such as a flash memory or a storage device in a server), or in a storage device in another terminal (for example, a server) that can communicate with the controller 110.

[0045] The processor 110a selects from the area candidate database 105 an area blocked by the work device 307 in the selected camera image, the area corresponding to the attitude of the work device 307 calculated using the attitude sensor 106, and sets the selected area as the blocked area of ​​the selected camera image.

[0046] An example of the correspondence between the shielding area of a certain camera 310 held by the area candidate database 105 and the posture of the working device 307 is shown in FIG. 10. The area candidate database 105 outputs one that corresponds to the posture of the working device 307 at that time (a combination of values of the boom angle 401, arm angle 402, and bucket angle 403) from among a plurality of candidate shielding areas corresponding to combinations of values of the boom angle 401, arm angle 402, and bucket angle 403. In the figure, the boom angle 401 is described as Ab, the arm angle 402 as Aa, and the bucket angle 403 as Ak. In the figure, the boom angle Ab is divided by four constants Ab0, Ab1, Ab2, Ab3, the arm angle Aa is divided by four constants Aa0, Aa1, Aa2, Aa3, and the bucket angle Ak is divided by three constants Ak0, Ak1, Ak2. The matrix on the left side in the figure is used when the bucket angle Ak is less than Ak1, and the matrix on the right side is used when the bucket angle Ak is greater than or equal to Ak1. In the example of FIG. 10, two matrices are used according to the bucket angle Ak, but the number of matrices used is not limited. For example, three matrices may be used by adding Ak3.

[0047] For example, when the boom angle (Ab) 401 is Ab0 ≦ Ab < Ab1, the arm angle (Aa) 402 is Aa0 ≦ Aa < Aa1, and the bucket angle (Ak) 403 is Ak0 ≦ Ak < Ak1, the shielding area 502a stored in the upper left cell within the matrix located on the left side of FIG. 10 is selected as the shielding area of the camera image 501.

[0048] Depending on the combination of the boom angle (Ab) 401, the arm angle (Aa) 402, and the bucket angle (Ak) 403, various-shaped shielding areas are output from the area candidate database 105. Also, for example, in the case of the state of the upper right position within the matrix located on the right side of the figure, that is, the state where Ab2 ≤ Ab ≤ Ab3, Aa0 ≤ Aa < Aa1, and Ak1 ≤ Ak ≤ Ak2, a result indicating that there is no shielding area is output. The situation where there is no shielding area means that the working device 307 is not reflected in the camera image 501 due to the combination of the boom angle 401, the arm angle 402, and the bucket angle 403. Therefore, shielding by the working device 307 does not occur, and no shielding area is generated. When there is no shielding area, the extraction and synthesis of the shielding area image are also unnecessary, so the generation process of the bird's-eye view image can be accelerated.

[0049] Note that the matrix shown in FIG. 10 exists for each camera 310, and the matrix is selected according to the camera 310 for which the shielding area is to be calculated.

[0050] In the present embodiment, by using the posture of the working device 307 calculated using the posture sensor 106 as described above and the area candidate database 105, it is possible to easily calculate a shielding area suitable for the posture of the working device 307 at that time.

[0051] <Fourth Embodiment> In the present embodiment, the controller 110 stores a three-dimensional model of the excavator 301, makes the three-dimensional model take the same posture as the actual excavator 301 (specifically, the working device 307), and shoots the surroundings with a virtual camera having the same mounting position, mounting angle, and viewing angle as the actual camera 310, and calculates the shielding area from the position of the working device 307 appearing in the camera image of the virtual camera.

[0052] The hardware configuration of the surrounding monitoring system for the excavator according to the present embodiment is the same as that in FIG. 2. The controller 110 of the present embodiment holds a three-dimensional model 901 of the excavator 301 in the storage device 110b.

[0053] A three-dimensional model 901 in this embodiment is shown in Fig. 11. The three-dimensional model 901 represents the three-dimensional shape of the shovel 301 as data, and the shapes and dimensions of each part of the three-dimensional model 901, such as the lower traveling body 302, upper rotating body 303, boom 304, arm 305, and bucket 306, match those of the actual shovel 301. Furthermore, the three-dimensional model 901 also holds position data of points that serve as references when the actual shovel 301 operates, such as a boom rotation center 902, an arm rotation center 903, and a bucket rotation center 904. In addition, data relating to the position, angle, and angle of view of each camera 310 mounted on the actual shovel 301 is also held, and for example, a first virtual camera 905 is provided in the three-dimensional model 901 as a virtual camera corresponding to an actual first camera 311.

[0054] In this shovel three-dimensional model 901, the posture of each part 304, 305, 306 of the working device 307 is set based on posture data of the actual shovel 301 input from posture sensor 106 (data output device 120A), and the posture of the actual shovel 301 is reproduced in shovel three-dimensional model 901. A part of the shape of shovel three-dimensional model 901 may be reflected in an image obtained by capturing this with a virtual camera such as first virtual camera 905. Processor 110a sets the range in which the working device 307 is reflected in the camera image of the virtual camera as a blocked area, and calculates this blocked area as blocked area 703 (see FIG. 5, etc.) in the image of actual camera 310 (e.g., first camera 311) corresponding to the virtual camera (e.g., first virtual camera 905).

[0055] FIG. 12 shows an example of camera image 1001 of first virtual camera 905. Arm 305 and bucket 306, which are part of three-dimensional model 901, are reflected in this camera image 1001. The shapes and locations of these components can be obtained as numerical values, and it is also possible to calculate which area in first virtual camera image 1001 they occupy (i.e., the position of masked area 1002). Based on these calculations, processor 110a calculates the position of masked area 1002 in image 1001 of first virtual camera 905, and sets first masked area 703 in actual first camera image 701 at the same position as masked area 1002. Note that although the method using first virtual camera 905 corresponding to first camera 311 has been described here, masked areas can also be set in a similar manner when a virtual camera corresponding to another camera 310 is used.

[0056] According to the present embodiment configured as described above, the blocked area in the camera image can be calculated with higher accuracy, so that the situation around the shovel 301 can be grasped more accurately.

[0057] In this embodiment, the position of the obstructed area in the actual camera image is calculated using three-dimensional model 901, but it is also possible to use three-dimensional model 901 only to calculate the relative position between working device 307 and each camera 310. In this case, processor 110a calculates the attitude of actual working device 307 based on the detection signal (relative position calculation data) of attitude sensor 106, makes working device 307 of three-dimensional model 901 take the calculated attitude of working device 307, and calculates the position between first virtual camera 905 attached to three-dimensional model 901 at the same position as first camera 311, for example, and working device 307 of three-dimensional model 901, thereby calculating the relative position between actual working device 307 and first camera 311.

[0058] Fifth Embodiment In this embodiment, the relative positions of the working device 307 and the multiple cameras 310 are calculated by taking images of markers 1100 attached to the respective parts 304, 305, and 306 of the working device 307 with the cameras 310.

[0059] The surroundings monitoring system for excavators of this embodiment shown in Figure 13 is equipped with a data output device 120B that outputs data for relative position calculation to the controller 110, and includes multiple markers 1100 (a boom marker 1101, an arm marker 1102, and a bucket marker 1103) attached to each part 304, 305, and 306 of the work device 307, and a first camera 311 that photographs the multiple markers 1100.

[0060] 14 shows a left side view of a shovel 301 according to this embodiment. Here, a boom marker 1101, an arm marker 1102, and a bucket marker 1103 are affixed to the boom 304, the arm 305, and the bucket 306, respectively. These markers 1100 are patterns drawn on a plane, and different patterns are drawn so that each marker 1100 can be uniquely identified. In the example shown, the letter M is drawn as a pattern on the boom marker 1101, the letter A is drawn as a pattern on the arm marker 1102, and the letter B is drawn as a pattern on the bucket marker 1103. Each marker 1100 is affixed to a location on the surface of the corresponding portion 304, 305, 306 of the work device 307 that is likely to be captured by the first camera 311.

[0061] When these markers 1100 are photographed by the first camera 311, the alphabet (pattern) on each marker 1100 may appear to expand or contract or deform in the first camera image (relative position calculation data). This occurs when the parts 304, 305, and 306 to which the markers 1100 are attached approach or move away from the first camera 311, or when the parts 304, 305, and 306 are tilted due to changes in the boom angle 401, arm angle 402, and bucket angle 403. Therefore, the processor 110a can calculate the attitudes of the boom 304, arm 305, and bucket 306 based on the shape of the alphabet of each marker 1100 captured in the first camera image (relative position calculation data). This processing allows the relative position of the work device 307 and each camera 310 to be calculated even without the attitude sensor 106, thereby enabling more efficient calculation of the blocked area in each camera image.

[0062] Although the above description has been given of the data for relative position calculation when the marker 1100 is photographed by the first camera 311, the same applies when the marker 1100 is photographed by the other camera 310. If it is difficult to photograph the marker 1100 by the other camera 310, the marker may be added appropriately to a position on the working device 307 where it is easy to photograph the marker by the other camera 310. Alternatively, a separate camera dedicated to photographing the marker may be prepared to photograph the marker 1100.

[0063] Furthermore, once the calculation of the attitudes of the boom 304, arm 305, and bucket 306 is completed, the shielded area corresponding to the attitudes may be selected from the area candidate database 105 described in the third embodiment and used.

[0064] Sixth Embodiment This embodiment includes a data output device 120C different from that described above. The surroundings monitoring system for an excavator of this embodiment shown in Fig. 15 includes, as data output device 120C that outputs relative position calculation data to controller 110, a comparison video database 118 that stores a plurality of comparison videos captured by cameras 310 of working implement 307 in various postures, the plurality of comparison videos being linked to the postures of working implement 307 at the time of capture, and cameras 310 that capture images of working implement 307.

[0065] 16 is a diagram showing some of the multiple matching videos 1200 (matching videos 1201, 1202, 1203) for the first camera 311 stored in the matching video database 118. The matching video database 118 stores the multiple matching videos 1200 in association with the posture (angle) of each part of the working device 307 in the matching video.

[0066] In this example, when performing calculation processing of the relative positions of each camera 310 and the work device 307, it is assumed that the first camera image 701 in FIG. 16 is input, and that the input first camera image 701 reflects the arm 305 and bucket 306 in the orientation shown in the figure. The processor 110a selects the matching image that is most similar to the input first camera image 701 (relative position calculation data) from among the multiple matching images (relative position calculation data) for the first camera 311 stored in the matching image database 118, and calculates the orientation (boom angle, arm angle, bucket angle) of the work device 307 linked to the selected matching image as the orientation of the work device 307 at that time. In the example of FIG. 16, the second matching image 1202 is selected as the most similar matching image. As a result, the processor 110a calculates the orientation (boom angle, arm angle, bucket angle) of the work device 307 linked to the second matching image 1202 as the orientation of the work device 307 at that time. The comparison of the similarity between the camera image 701 and the collation image may be limited to the operation device 307 in the image.

[0067] By this processing, even if the working device 307 does not have a marker 1100 attached and does not have an orientation sensor 106, the relative position between the working device 307 and each camera 310 can be calculated, and the calculation of the occluded area on each camera image can be made more efficient.

[0068] 16, each comparison image 1200 may be stored in association with a masked area 1220 (masked areas 1221, 1222, 1223). In the example of Fig. 16, the first comparison image 1201 is associated with the first masked area 1221, the second comparison image 1202 is associated with the second masked area 1222, and the third comparison image 1203 is associated with the third masked area 1223. This means that when processor 110a determines by image recognition that the posture pattern of working device 307 in first camera image 701 is similar to the posture pattern of working device 307 in first comparison image 1201, processor 110a outputs first masked area 1221 as first masked area 703 of first camera image 701.

[0069] In the example of Figure 16, the first camera image 701 is determined to be most similar to the second comparison image 1202 as described above, so the processor 110a calculates the second occlusion area candidate 1205 linked to the second comparison image 1202 as the occlusion area 703 of the first camera image 701.

[0070] By performing such processing, even if the marker 1100 is not attached to the working device 307 and the orientation sensor 106 is not present, the blocked area can be calculated, and the surrounding situation can be grasped more efficiently.

[0071] Seventh Embodiment In each of the above embodiments, all of the cameras 310 were attached to the upper rotating body 303 of the shovel 301, but the camera that photographs the work equipment 307 and its surroundings may be installed in a location away from the shovel 301.

[0072] 17 is a configuration diagram of a surroundings monitoring system for an excavator of this embodiment. The surroundings monitoring system of this embodiment includes a communication device 119 communicably connected to the controller 110, a first position sensor 116 for detecting the position of the upper rotating body 303, a first direction sensor 117 for detecting the direction (pitch angle, yaw angle, roll angle) of the upper rotating body 303, a remote camera 1302 installed at a location remote from the excavator 301, a second position sensor 1304 for detecting the position of the remote camera 1302, a second direction sensor 1303 for detecting the direction of the remote camera 1302, and a communication device 1301 for transmitting detection signals of the second position sensor 1304 and the second direction sensor 1303 to the controller 110 via a network.

[0073] Of these, the communication device 119, the first position sensor 116, the first direction sensor 117, the communication device 1301, the second position sensor 1304, and the second direction sensor 1303 constitute a data output device 120D that outputs data used by the controller 110 when calculating the relative position between the first camera 311 or the remote camera 1302 and the working device 307.

[0074] The communication device 119 receives signals transmitted to the controller 110 via the network (for example, camera images from the remote camera 1302 transmitted from the communication device 1301, and detection signals (data for calculating relative position) from the second position sensor 1304 and the second direction sensor 1303) and outputs them to the controller 110.

[0075] The first position sensor 116 may, for example, be a first GNSS antenna attached to the upper rotating body 303 that receives satellite signals from multiple positioning satellites, and a first receiver that calculates the position of the first GNSS antenna (data for calculating relative position) based on the satellite signals received by the first GNSS antenna.

[0076] The first direction sensor 117 can be, for example, a second GNSS antenna that is attached to a different location on the upper rotating structure 303 from the first GNSS antenna and receives satellite signals from multiple positioning satellites. The satellite signals received by the second GNSS antenna are transmitted to the first receiver, and the positions of the first GNSS antenna and the second GNSS antenna are calculated, thereby making it possible to calculate the direction of the upper rotating structure 303 (pitch angle, yaw angle, roll angle (data for calculating relative position)). Note that a geomagnetic sensor or an acceleration sensor can also be used as the first direction sensor 117.

[0077] Remote camera (second camera) 1302 is installed at a location away from shovel 301, and captures images of work equipment 307 and its surroundings from a different direction than first camera 311. The camera image (data for calculating relative position) of remote camera 1302 captures a space 706 outside the capture range of first camera 311, and includes first shielded area image 708.

[0078] The second position sensor 1304 may, for example, be a third GNSS antenna attached to an imaging device 1305 equipped with a remote camera 1302, which receives satellite signals from multiple positioning satellites, and a second receiver which calculates the position of the third GNSS antenna (data for calculating relative position) and, ultimately, the position of the remote camera 1302 (data for calculating relative position) based on the satellite signals received by the third GNSS antenna.

[0079] The second direction sensor 1303 can be, for example, a fourth GNSS antenna that is attached to the imaging facility 1305 at a different location from the third GNSS antenna and receives satellite signals from multiple positioning satellites. The satellite signals received by the fourth GNSS antenna are transmitted to the second receiver, and the positions of the third GNSS antenna and the fourth GNSS antenna are calculated, thereby calculating the direction (pitch angle, yaw angle, roll angle (data for calculating relative position)) of the remote camera 1302. Note that a geomagnetic sensor or an acceleration sensor can also be used as the second direction sensor 1303.

[0080] The communication device 1301 sequentially transmits camera image data from the remote camera 1302, a detection signal from the second position sensor 1304, and a detection signal from the second direction sensor 1303 to the controller 110 via the network.

[0081] The processor 110a of the controller 110 calculates the relative position between the working device 307 and the first camera 311 and the relative position between the working device 307 and the remote camera (second camera) 1302 based on the detection signals of the first position sensor 116 and the first direction sensor and the detection signals of the second position sensor 1304 and the second direction sensor 1303. The processing that follows this has already been explained and will not be described again.

[0082] By performing this processing, when it is difficult to obtain an image of the occluded area using only the camera 310 mounted on the shovel 301, it is possible to draw an image of the occluded area on the overhead image 880 by using the image of the space outside the capture area (image of the occluded area) reflected in the camera image of the remote camera 1302, so that the situation around the shovel can be grasped more accurately using the overhead image 880.

[0083] In the above, the relative position between the work device 307 and the cameras 311, 1302 is calculated by having the shovel 301 calculate the position (absolute position) and orientation of the work device 307 and the camera 311, and having the photographing equipment 1305 calculate the position (absolute position) and orientation of the remote camera 1302. However, for example, the relative position between the work device 307 and the cameras 311, 1302 may be calculated by calculating the distance (relative distance) and direction from the shovel 301 to the photographing equipment 1305 based on the position, posture, and size of the shovel 301 reflected in the camera image of the remote camera 1302.

[0084] Furthermore, although the surroundings monitoring device in FIG. 17 calculates the attitude of the working device 307 using the attitude sensor 106, the attitude of the working device 307 may be calculated using the alternative means described in the above embodiment.

[0085] Furthermore, the components of the controller 110 described above and the functions and execution processes of the components may be partially or entirely implemented by hardware (for example, by designing logic for executing the functions as an integrated circuit). The components of the controller 110 described above may be implemented as a program (software) that is read and executed by the processor 110a (for example, a CPU) to implement the functions of the controller 110. Information related to the program can be stored in, for example, a semiconductor memory (flash memory, SSD, etc.), a magnetic storage device (hard disk drive, etc.), a recording medium (magnetic disk, optical disk, etc.), etc.

[0086] In addition, in the above description of each embodiment, the control lines and information lines are those that are considered necessary for the description of the embodiment, but they do not necessarily represent all the control lines and information lines related to the product. In reality, it can be considered that almost all components are interconnected. [Explanation of symbols]

[0087] 104...monitor, 105...area candidate database, 106...posture sensor, 110...controller, 110a...processor, 110b...storage device, 116...first position sensor, 117...first direction sensor, 118...matching image database, 119...communication device, 120...data output device, 301...shovel, 302...lower traveling body, 303...upper rotating body, 304...boom, 305...arm, 306...bucket (attachment), 307...working device (front working device), 310...camera, 311...first camera, 312...second camera, 401...boom angle (Ab), 402...arm angle (Aa), 403...bucket angle (Ak), 701...first camera image, 702...second camera image, 703...first shielded area, 704...second shielded area, 705... Second non-captured space, 706...first non-captured space, 707...second shielded area image (second alternative image), 708...first shielded area image (first alternative image), 801...first camera overhead image, 802...second camera overhead image, 880...overhead image, 901...3D model of shovel, 905...first virtual camera, 1001...first virtual camera image, 1002...shielded area, 1100...marker, 110 1...Boom marker, 1102...Arm marker, 1103...Bucket marker, 1201...First matching image, 1202...Second matching image, 1203...Third matching image, 1221...First shielded area, 1222...Second shielded area, 1223...Third shielded area, 1301...Communication device, 1302...Remote camera (second camera), 1303...Second direction sensor, 1304...Second position sensor

Claims

1. a plurality of cameras that photograph the working equipment of the shovel and its surroundings from different directions; a controller that synthesizes images captured by the plurality of cameras to generate an overhead image of the periphery of the shovel and displays the overhead image on a monitor, a data output device that outputs data used when the controller calculates the relative position between each of the plurality of cameras and the working device; The controller calculating a relative position between each of the plurality of cameras and the working device based on the data output from the data output device; calculating a blocked area, which is an area blocked by the work device in the first image captured by the first camera, based on a relative position between a first camera of the plurality of cameras and the work device and an attachment angle of the first camera; extracting a shielded area image, which is an image of a portion of a second image captured by the second camera corresponding to the shielded area, from the second image based on a relative position between the working device and a second camera among the plurality of cameras that captures the shielded area and an attachment angle of the second camera; A surroundings monitoring device for an excavator, characterized in that when generating the overhead image, the blocked area image is synthesized onto the blocked area on the first image.

2. The surroundings monitoring device for a shovel according to claim 1, The controller calculating a position of an out-of-capture space corresponding to the occluded area on the surface of the terrain based on the relative positions of the first camera and the work implement, the mounting angle of the first camera, and the terrain around the shovel; A surroundings monitoring device for an excavator, characterized in that the shielded area image is calculated based on the relative position of the second camera and the work device, the mounting angle of the second camera, and the position of the non-captured space.

3. The surroundings monitoring device for a shovel according to claim 1, The controller displays on the monitor the overhead image in which an area within the blocked area where the blocked area image does not exist is highlighted.

4. The surroundings monitoring device for a shovel according to claim 1, the plurality of cameras are attached to an upper rotating body of the shovel, the data output device includes an attitude sensor for detecting the attitude of the working device; the controller calculates the attitude of the work device based on the detection signal of the attitude sensor, and calculates the relative position of each of the plurality of cameras and the work device based on the calculated attitude of the work device.

5. The surroundings monitoring device for a shovel according to claim 4, the controller selects a blocked area corresponding to the attitude of the work device calculated using the attitude sensor from an area candidate database in which a correspondence relationship between the blocked area in the first video and the attitude of the work device is defined, and sets the blocked area selected from the area candidate database as the blocked area of ​​the first video.

6. The surroundings monitoring device for a shovel according to claim 1, the plurality of cameras are attached to an upper rotating body of the shovel, the data output device includes an attitude sensor for detecting the attitude of the working device; the controller calculates the attitude of the work device based on the detection signal of the attitude sensor, applies the calculated attitude of the work device to a three-dimensional model of the shovel, and calculates the position of a virtual camera attached to the three-dimensional model at the same position as the first camera, and the position of the work device in the three-dimensional model, thereby calculating the relative position of each of the multiple cameras and the work device.

7. The surroundings monitoring device for a shovel according to claim 1, the plurality of cameras are attached to an upper rotating body of the shovel, the data output device includes a plurality of markers attached to the working device; the controller calculates an attitude of the work device based on the shapes of the plurality of markers captured in the first image, and calculates a relative position between the work device and each of the plurality of cameras based on the calculated attitude of the work device.

8. The surroundings monitoring device for a shovel according to claim 1, the plurality of cameras are attached to an upper rotating body of the shovel, the data output device includes a matching video database storing a plurality of matching videos obtained by capturing images of the working device in various postures with the first camera, the plurality of matching videos being linked to the postures of the working device at the time of capturing the images; the controller selects a comparison video that is most similar to the first video from the plurality of comparison videos stored in the comparison video database, and calculates a relative position between each of the plurality of cameras and the work device based on the posture of the work device linked to the comparison video that is most similar to the first video.

9. The surroundings monitoring device for a shovel according to claim 1, the first camera is attached to an upper rotating body of the shovel, The surroundings monitoring device for an excavator is characterized in that the second camera is installed at a location away from the excavator.

10. The surroundings monitoring device for a shovel according to claim 9, the data output device includes a first position sensor that detects the position of the upper rotating body, a first direction sensor that detects the direction of the upper rotating body, a second position sensor that detects the position of the second camera, a second direction sensor that detects the direction of the second camera, and a communication device that transmits detection signals of the second position sensor and the second direction sensor to the shovel, the controller calculates a relative position between the first camera and the work device and a relative position between the second camera and the work device based on detection signals from the first position sensor and the first direction sensor and detection signals from the second position sensor and the second direction sensor.

Images