Apparatus for generating surrounding image of work machine, and program for generating surrounding image of work machine

Abstract

To make it possible to keep continuity at boundary portions of images without making a boundary between first and second images remarkable.SOLUTION: A first image and a second image are synthesized such that a first effective pixel region of a first overlapped image region is overlapped with a second non-effective pixel region of a second overlapped image region, and a first non-effective pixel region of the first overlapped image region is overlapped with a second effective pixel region of the second overlapped image region.SELECTED DRAWING: Figure 16

Description

Technical Field

[0001] The present invention relates to a surrounding image generation device for a work machine and a surrounding image generation program for a work machine.

Background Art

[0002] In order for an operator of a work machine including a construction machine to visually confirm the situation around his or her own vehicle, images captured by a plurality of in-vehicle cameras mounted on the work machine are converted and synthesized to generate an aerial image viewed from above, and the generated aerial image is displayed on a monitor screen in the cab. Such a technique is already known. According to this technique, the operator can recognize the surrounding situation with an aerial view.

[0003] Also, a technique is already known in which two-dimensional images captured by a plurality of similar in-vehicle cameras are projected and mapped onto a three-dimensional space model to generate spatial data, and based on the spatial data, the three-dimensional space model is converted into an image viewed from an arbitrary viewpoint, for example, from above, and displayed on a monitor screen. Here, as the three-dimensional space model for projecting the two-dimensional image captured by the in-vehicle camera, there are three-dimensional shapes such as a cylindrical shape and a bowl shape composed of a curved surface and a flat surface. According to this technique, the operator of the work machine can simultaneously confirm not only an aerial image of the road surface near the work machine but also a background image including a background such as an empty space continuous with the road surface.

[0004] Patent Document 1 (Japanese Patent No. 6873889) describes an invention in which two-dimensional images captured by a plurality of in-vehicle cameras are projected onto a semi-cylindrical three-dimensional space model, and a process of forming a lattice pattern that causes an identical-color optical illusion is performed on an overlapping region of the imaging ranges of the two in-vehicle cameras.

[0005] Patent Document 2 (Japanese Patent No. 5959264) describes an invention in which two-dimensional images captured by a plurality of in-vehicle cameras are projected onto a bowl-shaped three-dimensional space model, and two adjacent images at the boundary of the captured images of each of the two in-vehicle cameras are combined by semi-transparent synthesis (alpha blending) using a coefficient (alpha value).

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] An object of the present invention is to make the boundary between a first image and a second image less conspicuous and to maintain the continuity of the images at the boundary as compared with the prior art.

Means for Solving the Problems

[0008] The first aspect is a first imaging unit that captures a first image around a working machine, a second imaging unit that captures a second image around the working machine adjacent to the first image such that a boundary image area overlaps with the first image, a first overlapping image area of the first image corresponding to the boundary image area is divided into a first valid pixel area where the pixel information of the first image is valid and a first invalid pixel area where the pixel information of the first image becomes invalid and the pixel information of the second image becomes valid, a second overlapping image area of the second image corresponding to the boundary image area is divided into a second valid pixel area where the pixel information of the second image is valid and a second invalid pixel area where the pixel information of the second image becomes invalid and the pixel information of the first image becomes valid, and an image synthesizing unit that synthesizes the first image and the second image such that the first valid pixel area of the first overlapping image area overlaps with the second invalid pixel area of the second overlapping image area and the first invalid pixel area of the first overlapping image area overlaps with the second valid pixel area of the second overlapping image area. A device for generating an image around a working machine.

[0009] The second aspect is, in the first aspect, the first overlapping image area has the first valid pixel area and the first invalid pixel area adjacent to each other alternately in a checkered pattern, and the second overlapping image area has the second valid pixel area and the second invalid pixel area adjacent to each other alternately in a checkered pattern. A device for generating an image around a working machine.

[0010] The third aspect is, in the first aspect, the composite image of the first image and the second image is an aerial image of the surroundings of the working machine as seen from above. A device for generating an image around a working machine.

[0011] The fourth aspect is, in the first aspect, further comprising a third imaging unit that captures a third image adjacent to the second image, and the first mask processing unit, the second mask processing unit, and the image synthesizing unit perform processing by replacing the first image and the second image with the second image and the third image. A device for generating an image around a working machine.

[0012] The fifth aspect is a peripheral image generation program for a working machine that causes a computer to execute the following processes to generate a peripheral image of the working machine, including a first image acquisition process for acquiring a first image around the working machine, a second image acquisition process for acquiring a second image around the working machine adjacent to the first image such that a boundary image area overlaps, a first masking process for dividing a first overlapping image area of the first image corresponding to the boundary image area into a first valid pixel area where pixel information of the first image is valid and a first invalid pixel area where pixel information of the first image is invalid and pixel information of the second image is valid, a second masking process for dividing a second overlapping image area of the second image corresponding to the boundary image area into a second valid pixel area where pixel information of the second image is valid and a second invalid pixel area where pixel information of the second image is invalid and pixel information of the first image is valid, and an image synthesis process for synthesizing the first image and the second image such that the first valid pixel area of the first overlapping image area overlaps the second invalid pixel area of the second overlapping image area and the first invalid pixel area of the first overlapping image area overlaps the second valid pixel area of the second overlapping image area.

[0013] The sixth aspect is the peripheral image generation program for a working machine according to the fifth aspect, wherein the first overlapping image area has the first valid pixel area and the first invalid pixel area adjacent to each other alternately and arranged in a checkered pattern, and

[0014] the second overlapping image area has the second valid pixel area and the second invalid pixel area adjacent to each other alternately and arranged in a checkered pattern.

[0015] The seventh aspect is the peripheral image generation program for a working machine according to the fifth aspect, wherein the composite image of the first image and the second image is an aerial image of the periphery of the working machine viewed from above.

[0016] The eighth aspect is, in the fifth aspect, further including a third image acquisition process for acquiring an image of a third image adjacent to the second image, and replacing the first image and the second image with the second image and the third image, and performing the first masking process, the second masking process, and the image synthesis process. It is a surrounding image generation program for a working machine.

Effects of the Invention

[0017] According to the first aspect to the eighth aspect, compared with the prior art, the boundary between the first image and the second image is not prominent, and the continuity of the image can be maintained at the boundary portion.

Brief Description of the Drawings

[0018]

Figure 1A

Figure 1B

Figure 1C

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments of a surrounding image generation device for a working machine and a surrounding image generation program for a working machine according to the present invention will be described with reference to the drawings.

[0020] FIGS. 1A and 1B are diagrams showing the appearance of a working machine 100. The working machine 100 is assumed to be a hydraulic excavator. FIG. 1A is a bird's-eye view of the working machine 100 seen from an overhead viewpoint. FIG. 1B is a view of the working machine 100 seen from the side.

[0021] The work machine 100 includes a lower traveling body 110, an upper slewing body 120, and a work implement 130.

[0022] The lower traveling body 110 includes left and right crawler belts 111L and 111R wound around left and right drive wheels (not shown), respectively. In response to the drive of the drive wheels (not shown), the left and right crawler belts 111L and 111R are rotationally driven, and the work machine 100 is steered.

[0023] The upper slewing body 120 is located above the lower traveling body 110 and is rotatably attached to the lower traveling body 110.

[0024] A cab 121 is provided in the upper front of the upper slewing body 120. A display unit 250 described later is provided in the cab 121. The display unit 250 is a device for displaying images and includes, for example, a liquid crystal display. The display screen 251 of the display unit 250 is provided in a manner that can be visually recognized by an operator seated on the driver's seat 122 in the cab 121.

[0025] The work implement 130 is attached to the front of the upper slewing body 120 so as to be drivable in response to the operation of an operation lever 123 in the cab 121. The work implement 130 includes a boom 131 attached to the front of the upper slewing body 120 so as to be drivable in the vertical direction, an arm 132 attached to the tip of the boom 131 so as to be drivable in the vertical direction with respect to the boom 131, and a bucket (or attachment) 133 attached to the tip of the arm 132 so as to be drivable in the vertical direction with respect to the arm 132.

[0026] Note that the boom 131, the arm 132, the bucket 133, the upper slewing body 120, and the left and right crawler belts 111L and 111R of the lower traveling body 110 are driven by a boom cylinder 131C, an arm cylinder 132C, a bucket cylinder 133C, a slewing motor (not shown), and left and right traveling motors, respectively.

[0027] There may be, for example, operators or obstacles in the areas behind the work machine 100 or in blind spots from the driver's seat 122. In order to confirm the presence of these operators or obstacles, an aerial view image 360, which will be described later, is displayed on the display screen 251 of the display unit 250.

[0028] In order to acquire the aerial view image 360, the work machine 100 is equipped with, for example, four imaging units 211, 212, 213, and 214. Each of the imaging units 211, 212, 213, and 214 includes an imaging element such as a CCD or a CMOS, and is equipped with a wide-angle or fish-eye lens having an angle of view of, for example, 180 degrees. The light condensed by the lenses of the imaging units 211, 212, 213, and 214 forms an image on the imaging element and is output as an imaging signal via the image processing units in the imaging units 211 to 214. Hereinafter, the imaging units 211, 212, 213, and 214 will be referred to as the first imaging unit 211, the second imaging unit 212, the third imaging unit 213, and the fourth imaging unit 214, respectively. However, "first, second, third, and fourth" are used to identify the installation locations of the individual imaging units 211, 212, 213, and 214 in one embodiment, and the descriptions of the first imaging unit, the second imaging unit, and the third imaging unit in the claims are not limited by this description. Hereinafter, the first imaging unit 211, the second imaging unit 212, the third imaging unit 213, and the fourth imaging unit 214 will be collectively referred to as the imaging unit 210.

[0029] A first imaging unit 211 for forward imaging is provided at the front of the upper swing body 120. The first imaging unit 211 for forward imaging images the imaging range 211A at the front of the upper swing body 120 according to its angle of view (for example, 180 degrees).

[0030] A second imaging unit 212 for imaging the right side is provided on the right side of the upper swing body 120. The second imaging unit 212 for imaging the right side images the imaging range 212A on the right side of the upper swing body 120 according to its angle of view.

[0031] At the rear of the upper revolving body 120, a third imaging unit 213 for rear imaging is provided. The third imaging unit 213 for rear imaging images an imaging range 213A behind the upper revolving body 120 according to its angle of view.

[0032] On the left side of the upper revolving body 120, a fourth imaging unit 214 for left-side imaging is provided. The fourth imaging unit 214 for left-side imaging images an imaging range 214A on the left side of the upper revolving body 120 according to its angle of view.

[0033] FIG. 1C shows a configuration example of another embodiment in which three imaging units 212, 213, and 214 are mounted on the working machine 100. It is also possible to implement an embodiment in which at least two adjacent imaging units, for example, the imaging units 212 and 213, are mounted.

[0034] FIG. 2 is a block diagram showing the functions of the surrounding image generation device 200 according to the embodiment.

[0035] The surrounding image generation device 200 includes a first imaging unit 211, a second imaging unit 212, a third imaging unit 213, a fourth imaging unit 214, a distortion correction processing unit 220, a viewpoint conversion processing unit 230, an image synthesis processing unit 240, a display unit 250, a calibration data storage unit 260, a calibration data reading processing unit 270, and an aerial view drawing processing unit 280.

[0036] The image synthesis processing unit 240 includes a first mask processing unit 241, a second mask processing unit 242, and an image synthesis unit 243.

[0037] The functions in FIG. 2 can be realized by a computer terminal (controller) 101 provided in the working machine 100.

[0038] Also, the functions in FIG. 2 can be realized by a combination of a computer terminal (controller) 101 provided in the working machine 100, a server, and a network that communicably connects them.

[0039] FIG. 3 is a diagram illustrating the hardware configuration of the computer terminal 101 (controller) and the server provided in the working machine 100, and is a hardware configuration diagram for realizing the functional configuration of FIG. 2.

[0040] As shown in FIG. 3, the computer terminal (controller) 101 and the server provided in the working machine 100 are configured such that the CPU (Central Processing Unit) 11, ROM (Read Only Memory) 12, RAM (Random Access Memory) 13, storage 14, input unit 16, display unit 250, communication interface 18, and external storage unit 19 are connected to be communicable with each other via the system bus 15. The CPU 11 is a central arithmetic processing unit that executes various programs and controls each device connected to the system bus 15. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a working area. The CPU 11 controls each device connected to the system bus 15 and performs various arithmetic processes according to the programs recorded in the ROM 12 or the storage 14.

[0041] The ROM 12 or the storage 14 holds a BIOS (Basic Input / Output System), an OS (Operating System), which are control programs executed by the CPU 11, various programs readable and executable by a computer for realizing the present embodiment, and various necessary data.

[0042] The RAM 13 functions as the main memory, work area, etc. of the CPU 11 and temporarily stores programs or data as a working area. The storage 14 is composed of an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including the BIOS and the OS and various data.

[0043] The input unit 16 includes the input of the imaging signal of the imaging unit 210, a pointing device such as a mouse, and a reading device such as a keyboard and a scanner, and is used to perform various inputs.

[0044] The display unit 250 is, for example, a liquid crystal display, and displays various information including the surrounding image. The display unit 250 may adopt a touch panel method and function as the input unit 16.

[0045] The communication interface 18 is an interface for communicating with other devices. For example, standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark) are used. The communication interface 18 connects to a network and controls the transmission and reception of data.

[0046] The external storage unit 19 is composed of various memory cards such as USB memories, and removable external storage media such as HDDs and SSDs.

[0047] FIG. 4A and FIG. 4B are flowcharts showing the processing procedures of the surrounding image generation program of the working machine performed by the surrounding image generation device 200 of the embodiment.

[0048] The processing of the embodiment mainly consists of a calibration process S100 and an aerial view display process S200.

[0049] The calibration process S100 is executed in advance as a pre-process of the aerial view display process S200.

[0050] The calibration process S100 is performed according to the processing procedure shown in FIG. 4A. The calibration process S100 consists of a distortion correction process S110, a viewpoint conversion process S120, and an image composition process S130.

[0051] The distortion correction process S110 is a process executed by the distortion correction processing unit 220. The distortion correction processing unit 220 performs distortion correction on the two-dimensional captured images 311, 312, 313, and 314 respectively acquired by the first imaging unit 211, the second imaging unit 212, the third imaging unit 213, and the fourth imaging unit 214 to generate distortion-corrected images 321, 322, 323, and 324. Hereinafter, the two-dimensional captured images 311, 312, 313, and 314 are collectively referred to as the captured image 310. Also, the distortion-corrected images 321, 322, 323, and 324 are collectively referred to as the distortion-corrected image 320.

[0052] The viewpoint conversion process S120 is a process executed by the viewpoint conversion processing unit 230. The viewpoint conversion processing unit 230 performs three-dimensional coordinate conversion on the two-dimensional distortion-corrected images 321, 322, 323, and 324 into the projection images of the bowl-shaped three-dimensional space model MD to generate viewpoint-converted images 331, 332, 333, and 334. Hereinafter, the viewpoint-converted images 331, 332, 333, and 334 are collectively referred to as the viewpoint-converted image 330.

[0053] The image synthesis process S130 is a process executed by the image synthesis processing unit 240. The image synthesis processing unit 240 synthesizes the viewpoint-converted images 331, 332, 333, and 334 to generate a synthesized projection image 340 of the bowl-shaped three-dimensional space model MD.

[0054] The image synthesis process S130 includes a first mask process S131, a second mask process S132, and an image synthesis process S133.

[0055] The first mask process S131 is a process performed by the first mask processing unit 241.

[0056] The second mask process S132 is a process performed by the second mask processing unit 242.

[0057] The image synthesis process S133 is a process performed by the image synthesis unit 243.

[0058] Calibration is performed for each of the distortion correction process S110, the viewpoint conversion process S120, and the image synthesis process S130. Calibration is executed by calculating the internal parameters and external parameters of the imaging unit 210 using the camera coordinate system C of the imaging unit 210 and the world coordinate system W of the imaging target of the imaging unit 210. The internal parameters are the distortion coefficient, the focal length, the center position of the imaging element, etc. The external parameters are the position and orientation of the imaging unit 210, etc. Using the calibration data obtained by performing calibration on the imaging unit 210, the captured image 310 captured by the imaging unit 210 can be converted into a distortion corrected image 320, a viewpoint converted image 330, a composite projection image 340, and an aerial view drawing image 360 respectively.

[0059] Calibration data is acquired for each of the distortion correction process S110, the viewpoint conversion process S120, and the image synthesis process S130, and the acquired calibration data is stored in the calibration data storage unit 260.

[0060] Calibration can be performed for each event occurring in the working machine 100. For example, calibration can be performed every time the engine key switch of the working machine 100 is turned on and the operation of the working machine 100 is started.

[0061] The aerial view display process S200 is performed according to the processing procedure shown in FIG. 4B. The aerial view display process S200 includes a calibration data reading process S210, an aerial view drawing process S220, and a display process S230.

[0062] The calibration data reading process S210 is a process performed by the calibration data reading unit 270. The calibration data reading unit 270 executes a process of reading the calibration data stored in the calibration data storage unit 260.

[0063] The bird's-eye view drawing process S220 is a process performed by the bird's-eye view drawing unit 280. Based on the calibration data, the bird's-eye view drawing unit 280 sequentially generates a bird's-eye view drawing image 360 of the bowl-shaped three-dimensional space model MD as viewed from a viewpoint above the moving machine 100, based on the two-dimensional captured images 311, 312, 313, 314 sequentially acquired by the first imaging unit 211, the second imaging unit 212, the third imaging unit 213, and the fourth imaging unit 214, respectively.

[0064] The display process S230 is a process of displaying the bird's-eye view drawing image 360 on the display screen 251 of the display unit 250. During the operation of the working machine 100, the bird's-eye view drawing image 360 is displayed on the display screen 251 of the display unit 250 in real time according to the operator's instructions.

[0065] (Distortion correction process)

[0066] FIG. 5 is a diagram for explaining the distortion correction process S110.

[0067] FIG. 5(A) shows the captured images 311, 312, 313, 314 acquired by the first imaging unit 211, the second imaging unit 212, the third imaging unit 213, and the fourth imaging unit 214, respectively. Hereinafter, in order to identify the captured images 311, 312, 313, 314, the captured images 311, 312, 313, 314 are respectively referred to as the first captured image 311, the second captured image 312, the third captured image 313, and the fourth captured image 314.

[0068] The first captured image 311 is an image around the working machine 100 adjacent to the fourth captured image 314, and is an image corresponding to the imaging range 211A shown in FIG. 1A. The first captured image 311 overlaps with the image corresponding to the boundary image area 211B (FIG. 1A) of the fourth captured image 314.

[0069] The second captured image 312 is an image around the working machine 100 adjacent to the first captured image 311, and is an image corresponding to the imaging range 212A shown in FIG. 1A. The second captured image 312 overlaps with the image corresponding to the boundary image area 212B (FIG. 1A) of the first captured image 311.

[0070] The third captured image 313 is an image of the surroundings of the working machine 100 adjacent to the second captured image 312, and is an image corresponding to the imaging range 213A shown in FIG. 1A. In the third captured image 313, the image corresponding to the boundary image region 213B (FIG. 1A) overlaps with the second captured image 312.

[0071] The fourth captured image 314 is an image of the surroundings of the working machine 100 adjacent to the third captured image 313, and is an image corresponding to the imaging range 214A shown in FIG. 1A. In the fourth captured image 314, the image corresponding to the boundary image region 214B (FIG. 1A) overlaps with the third captured image 313.

[0072] FIG. 5(B) shows the distortion-corrected images 321, 322, 323, and 324 obtained by subjecting the first captured image 311, the second captured image 312, the third captured image 313, and the fourth captured image 314 to distortion correction processing by the distortion correction processing unit 220, respectively. In order to identify each of the distortion-corrected images 321, 322, 323, and 324, the distortion-corrected images 321, 322, 323, and 324 are respectively referred to as the first distortion-corrected image 321, the second distortion-corrected image 322, the third distortion-corrected image 323, and the fourth distortion-corrected image 324.

[0073] The calibration of the distortion correction can be performed, for example, by a method of analyzing a captured image obtained by capturing a calibration pattern image. As shown in FIG. 5(C), for example, a chessboard pattern PT1 can be used as the calibration pattern. The first imaging unit 211 captures the chessboard pattern PT1 from various angles, for example, from different imaging directions every 10 sheets, detects the intersections of the chessboard in those captured images, and analyzes each detection position, whereby the distortion coefficient of the first imaging unit 211 is obtained as calibration data. The program for calibrating the distortion correction is recorded in advance in the ROM 12 or the storage 14.

[0074] Calibration data is similarly obtained for the second imaging unit 212, the third imaging unit 213, and the fourth imaging unit 214. By using the calibration data, a distortion-corrected image 320 can be generated from the captured images 310 obtained by the same imaging unit 210.

[0075] (Viewpoint conversion processing)

[0076] FIG. 6 is a diagram for explaining the viewpoint conversion process S120.

[0077] FIG. 6(A) shows the first distortion-corrected image 321, the second distortion-corrected image 322, the third distortion-corrected image 323, and the fourth distortion-corrected image 324 before viewpoint conversion.

[0078] FIG. 6(B) shows the viewpoint-converted images 331, 332, 333, and 334 generated by three-dimensionally transforming the first distortion-corrected image 321, the second distortion-corrected image 322, the third distortion-corrected image 323, and the fourth distortion-corrected image 324 into the projection images of the bowl-shaped three-dimensional space model MD, respectively. Hereinafter, in order to identify the viewpoint-converted images 331, 332, 333, and 334, the viewpoint-converted images 331, 332, 333, and 334 are respectively referred to as the first viewpoint-converted image 331, the second viewpoint-converted image 332, the third viewpoint-converted image 333, and the fourth viewpoint-converted image 334.

[0079] On the other hand, FIG. 7A is a view of the bowl-shaped three-dimensional space model MD seen in a vertical cross-section.

[0080] FIG. 7B is a view of the bowl-shaped three-dimensional space model MD seen from above.

[0081] The bowl-shaped three-dimensional space model MD (hereinafter referred to as the three-dimensional space model MD) is formed such that the working machine 100 is disposed at the center of its bottom.

[0082] The inner region of the three-dimensional space model MD consists of a front region AR11, a right side region AR12, a rear region AR13, and a left side region AR14.

[0083] The first distortion-corrected image 321 is perspective-transformed into a first perspective-transformed image 331 attached to the inner front region AR11 of the three-dimensional space model MD.

[0084] The second distortion-corrected image 322 is perspective-transformed into a second perspective-transformed image 332 attached to the inner right region AR12 of the three-dimensional space model MD.

[0085] The third distortion-corrected image 323 is perspective-transformed into a third perspective-transformed image 333 attached to the inner rear region AR13 of the three-dimensional space model MD.

[0086] The fourth distortion-corrected image 324 is perspective-transformed into a fourth perspective-transformed image 334 attached to the inner rear region AR14 of the three-dimensional space model MD.

[0087] FIG. 8 is a diagram for explaining the coordinate conversion process for perspective transformation. The external parameters of the imaging unit 210 are acquired as calibration data. Using the external parameters, the two-dimensional distortion-corrected image 320 can be converted into a three-dimensional perspective-transformed image 330.

[0088] FIG. 8(A) shows the marker MK1 in each of the first distortion-corrected image 321, the second distortion-corrected image 322, the third distortion-corrected image 323, and the fourth distortion-corrected image 324. The marker MK1 is an object OB1 whose size and shape are known in advance among the imaging targets. The external parameters of the imaging unit 210 can be acquired by analyzing the position and shape of the marker MK1 in the distortion-corrected image 320. Using the acquired external parameters of the imaging unit 210, the two-dimensional distortion-corrected image 320 can be converted into a three-dimensional perspective-transformed image 330.

[0089] FIG. 8(B) shows the relationship between the two-dimensional first distortion-corrected image 321, the second distortion-corrected image 322, the third distortion-corrected image 323, the fourth distortion-corrected image 324 and the three-dimensional coordinate positions on the three-dimensional space model MD.

[0090] To each position Pc11, Pc12, Pc13... of the two-dimensional first distortion-corrected image 321, there correspond three-dimensional coordinate positions Pm11, Pm12, Pm13... on the three-dimensional space model MD. Each of the positions Pc11, Pc12, Pc13... is collectively referred to as position Pc. Also, the positions Pm11, Pm12, Pm13... are collectively referred to as position Pm. The same applies hereinafter.

[0091] Similarly, to each position Pc21, Pc22, Pc23... of the two-dimensional second distortion-corrected image 322, there correspond three-dimensional coordinate positions Pm21, Pm22, Pm23... on the three-dimensional space model MD.

[0092] Similarly, to each position Pc31, Pc32, Pc33... of the two-dimensional third distortion-corrected image 323, there correspond three-dimensional coordinate positions Pm31, Pm32, Pm33... on the three-dimensional space model MD.

[0093] Similarly, to each position Pc41, Pc42, Pc43... of the two-dimensional fourth distortion-corrected image 324, there correspond three-dimensional coordinate positions Pm41, Pm42, Pm43... on the three-dimensional space model MD.

[0094] FIG. 8(C) shows the first viewpoint-converted image 331, the second viewpoint-converted image 332, the third viewpoint-converted image 333, and the fourth viewpoint-converted image 334.

[0095] The first viewpoint-converted image 331 is an image obtained by projecting the two-dimensional first distortion-corrected image 321 onto the three-dimensional space model MD and viewing it from a virtual viewpoint from above.

[0096] The second viewpoint-converted image 332 is an image obtained by projecting the two-dimensional second distortion-corrected image 322 onto the three-dimensional space model MD and viewing it from a virtual viewpoint from above.

[0097] The third viewpoint-converted image 333 is an image obtained by projecting the two-dimensional third distortion-corrected image 323 onto the three-dimensional space model MD and viewing it from a virtual viewpoint from above.

[0098] The fourth viewpoint-converted image 334 is an image obtained by projecting the two-dimensional fourth distortion-corrected image 324 onto the three-dimensional space model MD and viewing it from a virtual viewpoint from above.

[0099] FIG. 9 is a diagram for explaining how the two-dimensional distortion-corrected image 320 is projected as a viewpoint-converted image 330 onto the three-dimensional space model MD.

[0100] As shown in FIG. 9, the distortion-corrected image 320 is represented in the camera coordinate system C of the imaging unit 210. The camera coordinate system C defines the three-dimensional position of the target object with the position of the lens center of the imaging unit 210 as the origin Oc. The Zc axis of the imaging viewpoint coordinate system C coincides with the optical axis of the imaging unit 210.

[0101] The viewpoint-converted image 330 is represented in the virtual viewpoint coordinate system E. The imaging viewpoint coordinate system C defines the three-dimensional position of the target object with the position of the virtual viewpoint as the origin Oe. The Ze axis of the virtual viewpoint coordinate system E coincides with the direction of the line of sight when viewing the three-dimensional space model MD from the virtual viewpoint (for example, the direction from above to below).

[0102] The position of the target object in the space of the three-dimensional space model MD is defined in the three-dimensional world coordinate system W.

[0103] FIG. 10 shows the mutual relationship among the camera coordinate system C, the world coordinate system W, and the virtual viewpoint coordinate system E shown in FIG. 9. FIG. 10 also shows the camera imaging plane CP as the imaging plane of the distortion-corrected image 320 and the virtual viewpoint image plane SP as the imaging plane of the viewpoint-converted image 330.

[0104] The three-dimensional coordinate position Pm on the three-dimensional space model MD corresponds to the two-dimensional coordinate position Pc on the camera imaging plane CP (see FIG. 8(B)). Also, the three-dimensional coordinate position Pm on the three-dimensional space model MD corresponds to the two-dimensional coordinate position Ps on the virtual viewpoint image plane SP.

[0105] (Image composition process)

[0106] FIG. 11 is a diagram for explaining the image composition process S130.

[0107] FIG. 11(A) shows the first viewpoint-converted image 331, the second viewpoint-converted image 332, the third viewpoint-converted image 333, and the fourth viewpoint-converted image 334 before synthesis.

[0108] FIG. 11(B) shows the synthesized projection image 340 obtained by synthesizing the first viewpoint-converted image 331, the second viewpoint-converted image 332, the third viewpoint-converted image 333, and the fourth viewpoint-converted image 334.

[0109] As shown in FIG. 7B, the synthesized projection image 340 has an overlapping boundary image region 332B between the first viewpoint-converted image 331 and the second viewpoint-converted image 332, an overlapping boundary image region 333B between the second viewpoint-converted image 332 and the third viewpoint-converted image 333, an overlapping boundary image region 334B between the third viewpoint-converted image 333 and the fourth viewpoint-converted image 334, and an overlapping boundary image region 331B between the fourth viewpoint-converted image 334 and the first viewpoint-converted image 331, and is synthesized in such a manner.

[0110] FIG. 12 shows the synthesized projection image 1340 of the comparative example.

[0111] The synthesized projection image 1340 of the comparative example is an image synthesized by connecting the boundary 331L of the first viewpoint-converted image 331 and the boundary 332L of the second viewpoint-converted image 332, connecting the boundary 1332L of the second viewpoint-converted image 332 and the boundary 333L of the third viewpoint-converted image 333, connecting the boundary 1333L of the third viewpoint-converted image 333 and the boundary 334L of the fourth viewpoint-converted image 334, and connecting the boundary 1334L of the fourth viewpoint-converted image 334 and the boundary 1331L of the first viewpoint-converted image 331.

[0112] Compared with the synthesized projection image 340 of the embodiment, the boundary between adjacent viewpoint-converted images in the synthesized projection image 1340 of the comparative example is prominent, and the continuity of the image is impaired at the boundary. In contrast, in the synthesized projection image 340 of the embodiment, the boundary between adjacent viewpoint-converted images is not prominent compared with the synthesized projection image 1340 of the comparative example, and the continuity of the image is maintained at the boundary.

[0113] The composite projection image 340 of the embodiment is generated through the steps of a first masking process S131, a second masking process S132, and an image composite process S133.

[0114] (First masking process, second masking process)

[0115] Hereinafter, as shown in FIG. 13(A), it will be described assuming that the two adjacent viewpoint-converted images are the second viewpoint-converted image 332 and the third viewpoint-converted image 333.

[0116] FIG. 13(B) is a diagram showing a mask 1000M used for the first masking process S131 and the second masking process S132.

[0117] The mask 1000M is configured such that a first image valid pixel region 1010AR and a second image valid pixel region 1020AR are adjacent to each other and arranged alternately in a checkered pattern.

[0118] The first image valid pixel region 1010AR is a pixel region that validates only the pixel information of the corresponding pixel region 332V of one of the two adjacent viewpoint-converted images, which is the second viewpoint-converted image 332, and invalidates the pixel information of the corresponding pixel region 333I of the other third viewpoint-converted image 333.

[0119] On the other hand, the second image valid pixel region 1020AR is a pixel region that validates only the pixel information of the corresponding pixel region 333V of the other of the two adjacent viewpoint-converted images, which is the third viewpoint-converted image 333, and invalidates the pixel information of the corresponding pixel region 332I of one of the second viewpoint-converted images 332.

[0120] The pixel region may be a region corresponding to one pixel of the viewpoint-converted image, or may be a region of a plurality of adjacent pixels. Here, "valid" refers to the process of associating pixel information such as the luminance, hue, and saturation of the pixels of the first image or the second image corresponding to the pixels of the mask 1000M with the corresponding pixels. On the other hand, "invalid" refers to the process of not associating the pixel information such as the luminance, hue, and saturation of the pixels of the first image or the second image corresponding to the pixels of the mask 1000M with the corresponding pixels.

[0121] Note that the "valid" and "invalid" in this embodiment are not limited to the meaning of adopting 100% of the pixel information of one pixel region and 0% of the pixel information of the other pixel region. For example, an implementation is also possible in which the pixel information of the pixel region that becomes "valid" is adopted at a ratio larger than that of the other pixel information. For example, an implementation is also possible in which 70% of the pixel information of the pixel region that becomes "valid" is adopted and 30% of the pixel information of the corresponding pixel region of the other is adopted, and the two are combined at that ratio.

[0122] FIG. 14 is a diagram for explaining the first mask process S131 and the second mask process S132.

[0123] Apply the mask 1000M to the first overlapping image region 332W of the second viewpoint-converted image 332 corresponding to the boundary image region 333B, and divide it into a valid pixel region 332V where the pixel information of the second viewpoint-converted image 332 becomes valid and an invalid pixel region 332I where the pixel information of the second viewpoint-converted image 332 becomes invalid and the pixel information of the third viewpoint-converted image 333 becomes valid (the first mask process S131).

[0124] Similarly, apply the mask 1000M to the overlapping image region 333W of the third viewpoint-converted image 333 corresponding to the boundary image region 333B, and divide it into a valid pixel region 333V where the pixel information of the third viewpoint-converted image 333 becomes valid and an invalid pixel region 333I where the pixel information of the third viewpoint-converted image 333 becomes invalid and the pixel information of the second viewpoint-converted image 332 becomes valid (the second mask process S132).

[0125] FIG. 15(A) is a diagram showing the result of the first masking process S131. As shown in FIG. 15(A), for one of the two adjacent viewpoint-converted images, i.e., the second viewpoint-converted image 332, only the pixel information of the pixel region 332AR corresponding to the valid pixel region 332V is made valid, and the pixel information of the remaining invalid pixel region 332I is invalidated.

[0126] FIG. 15(B) is a diagram showing the result of the second masking process S132. As shown in FIG. 15(B), for the other of the two adjacent viewpoint-converted images 330, i.e., the third viewpoint-converted image 333, only the pixel information of the pixel region 333AR corresponding to the valid pixel region 333V is made valid, and the pixel information of the remaining invalid pixel region 333I is invalidated.

[0127] (Image synthesis process)

[0128] FIG. 15(C) shows the pixel information of the boundary image region 333B between the second viewpoint-converted image 332 and the third viewpoint-converted image 333 obtained as a result of the first masking process S131 and the second masking process S132.

[0129] The second viewpoint-converted image 332 and the third viewpoint-converted image 333 are synthesized such that the valid pixel region 332V of the overlapping image region 332W of the second viewpoint-converted image 332 overlaps with the corresponding invalid pixel region 333I of the overlapping image region 332W of the third viewpoint-converted image 333, and the invalid pixel region 332I of the overlapping image region 332W of the second viewpoint-converted image 332 overlaps with the corresponding valid pixel region 333V of the overlapping image region 332W of the third viewpoint-converted image 333. That is, for the boundary image region 333B between the second viewpoint-converted image 332 and the third viewpoint-converted image 333, the valid pixel region 332V of the second viewpoint-converted image 332 and the valid pixel region 333V of the third viewpoint-converted image 333 are synthesized so as to be adjacent to each other and arranged in an alternating checkered pattern.

[0130] Note that the arrangement pattern of the effective pixel regions of one adjacent image and the effective pixel regions of the other image is not limited to a checkerboard pattern. Any arrangement is acceptable as long as the boundary between adjacent images is not conspicuous and the continuity of the images is maintained at the boundary. For example, the effective pixel regions of adjacent images may be arranged randomly.

[0131] As described above, the case of synthesizing the second viewpoint-converted image 332 and the third viewpoint-converted image 333 has been explained. The same applies to the case of synthesizing the third viewpoint-converted image 333 and the fourth viewpoint-converted image 334, and the case of synthesizing the fourth viewpoint-converted image 334 and the first viewpoint-converted image 331.

[0132] FIG. 16 illustrates the process of generating the composite projection image 340.

[0133] As shown in FIG. 16(A), masks 1100M, 1200M, 1300M, and 1400M are respectively associated with the first distortion-corrected image 321, the second distortion-corrected image 322, the third distortion-corrected image 323, and the fourth distortion-corrected image 324. The masks 1100M, 1200M, 1300M, and 1400M are configured in the same manner as the mask 1000M.

[0134] Next, as shown in FIG. 16(B), the first distortion-corrected image 321, the second distortion-corrected image 322, the third distortion-corrected image 323, and the fourth distortion-corrected image 324 are subjected to viewpoint conversion processing to generate the first viewpoint-converted image 331, the second viewpoint-converted image 332, the third viewpoint-converted image 333, and the fourth viewpoint-converted image 334.

[0135] The masks 1100M, 1200M, 1300M, and 1400M are also subjected to viewpoint conversion processing in the same manner to generate masks 2100M, 2200M, 2300M, and 2400M.

[0136] For the boundary image region 332B between the first viewpoint-converted image 331 and the second viewpoint-converted image 332, the masks 2100M and 2200M are applied to perform the first masking process S131 and the second masking process S132.

[0137] For the boundary image region 333B between the second viewpoint conversion image 332 and the third viewpoint conversion image 333, the first mask process S131 and the second mask process S132 are performed by applying the mask 2200M and the mask 2300M.

[0138] For the boundary image region 334B between the third viewpoint conversion image 333 and the fourth viewpoint conversion image 334, the first mask process S131 and the second mask process S132 are performed by applying the mask 2300M and the mask 2400M.

[0139] For the boundary image region 331B between the fourth viewpoint conversion image 334 and the first viewpoint conversion image 331, the first mask process S131 and the second mask process S132 are performed by applying the mask 2400M and the mask 2100M.

[0140] Next, as shown in FIG. 16(C), the first viewpoint conversion image 331, the second viewpoint conversion image 332, the third viewpoint conversion image 333, and the fourth viewpoint conversion image 334 are combined to generate a combined projection image 340.

[0141] That is, similar to FIG. 15(C), for the boundary image region 332B between the first viewpoint conversion image 331 and the second viewpoint conversion image 332, the effective pixel region 331V of the first viewpoint conversion image 331 and the effective pixel region 332V of the second viewpoint conversion image 332 are combined so as to be adjacent to each other and arranged in an alternating checkered pattern.

[0142] Similarly, for the boundary image region 333B between the second viewpoint conversion image 332 and the third viewpoint conversion image 333, the effective pixel region 332V of the second viewpoint conversion image 332 and the effective pixel region 333V of the third viewpoint conversion image 333 are combined so as to be adjacent to each other and arranged in an alternating checkered pattern.

[0143] Similarly, for the boundary image region 334B between the third viewpoint conversion image 333 and the fourth viewpoint conversion image 334, the effective pixel region 333V of the third viewpoint conversion image 333 and the effective pixel region 334V of the fourth viewpoint conversion image 334 are combined so as to be adjacent to each other and arranged in an alternating checkered pattern.

[0144] Similarly, for the boundary image region 331B between the fourth viewpoint-converted image 334 and the first viewpoint-converted image 331, the effective pixel regions 334V of the fourth viewpoint-converted image 334 and the effective pixel regions 331V of the first viewpoint-converted image 331 are synthesized so as to be adjacent to each other and alternately arranged in a checkered pattern.

[0145] (Calibration data reading process)

[0146] When the operator gives an instruction to display the bird's-eye view drawing image 360, for example, via the input unit 16, imaging control signals are output to the first imaging unit 211, the second imaging unit 212, the third imaging unit 213, and the fourth imaging unit 214, and the first imaging unit 211, the second imaging unit 212, the third imaging unit 213, and the fourth imaging unit 214 are controlled so that imaging is sequentially performed by each of them.

[0147] As a result, imaging signals are input from the first imaging unit 211, the second imaging unit 212, the third imaging unit 213, and the fourth imaging unit 214 via the input unit 16, and the first captured image 311, the second captured image 312, the third captured image 313, and the fourth captured image 314 are acquired. A process of reading calibration data from the calibration data storage unit 260 is executed.

[0148] (Bird's-eye view drawing process)

[0149] FIG. 17 is a diagram for explaining the bird's-eye view drawing process S220.

[0150] As shown in FIG. 17(A), for example, when generating a bird's-eye view drawing image 360 looking down on the work machine 100 on which the operator is riding from above, the Ze axis of the virtual viewpoint coordinate system E is set in the direction from above to below, and a synthetic projection image 340 is generated based on the calibration data.

[0151] As shown in FIG. 17(B), the captured image 100IM of the working machine 100 is read from the ROM 12 or the storage 14.

[0152] As shown in FIG. 17(C), the captured image 100IM of the working machine 100 is arranged at the center of the composite projection image 340 such that the image portion corresponding to the working machine 130 at the front of the upper swing body 120 is located on the front side of the composite projection image 340. Thereby, the bird's-eye view drawing image 360 is generated.

[0153] (Display processing)

[0154] FIG. 18(A) is a diagram for explaining the display process S230.

[0155] During the operation of the working machine 100, the captured images 311, 312, 313, and 314 are sequentially acquired, and on the display screen 251 of the display unit 250, a bird's-eye view drawing image 360 corresponding to the first captured image 311, the second captured image 312, the third captured image 313, and the fourth captured image 314 sequentially acquired is displayed in real time. Note that, if necessary, the bird's-eye view drawing image 360 within the display screen 251 may be enlarged, reduced, the display position may be changed, or a display combined with other images may be performed as necessary.

[0156] Also, as shown in FIG. 18(B), in addition to the bird's-eye view drawing image 360 from a viewpoint looking down from above, a side view image 370 from another viewpoint, for example, a viewpoint looking at the three-dimensional space model MD from the side, may be displayed on the display screen 251 of the display unit 250.

[0157] The image to be displayed on the display screen 251 of the display unit 250 may be automatically switched according to the state of the working machine 100 (whether the upper swing body 120 is stopped or turning, and whether the lower traveling body 110 is stopped or traveling), or may be selectable by the operator.

Explanation of reference numerals

[0158] 200 Surrounding image generation device 211 First imaging unit 212 Second imaging unit 213 Third imaging unit 214 Fourth imaging unit 220 Distortion correction processing unit 230 Viewpoint conversion processing unit 240 Image synthesis processing unit 250 Display unit 260 Calibration data storage unit 270 Calibration data reading processing unit 280 Aerial view drawing processing unit

Claims

1. A first imaging unit that captures a first image around a work machine, a second imaging unit that captures a second image around the work machine adjacent to the first image such that a boundary image area overlaps with the first image, a first masking unit that divides a first overlapping image area of the first image corresponding to the boundary image area into a first valid pixel area where pixel information of the first image is valid and a first invalid pixel area where pixel information of the first image is invalid and pixel information of the second image is valid, a second masking unit that divides a second overlapping image area of the second image corresponding to the boundary image area into a second valid pixel area where pixel information of the second image is valid and a second invalid pixel area where pixel information of the second image is invalid and pixel information of the first image is valid, an image synthesizing unit that synthesizes the first image and the second image such that the first valid pixel area of the first overlapping image area overlaps with the second invalid pixel area of the second overlapping image area and the first invalid pixel area of the first overlapping image area overlaps with the second valid pixel area of the second overlapping image area, A surrounding image generation device for a work machine including the above.

2. In the first overlapping image area, the first valid pixel area and the first invalid pixel area are arranged adjacent to each other alternately in a checkered pattern, and in the second overlapping image area, the second valid pixel area and the second invalid pixel area are arranged adjacent to each other alternately in a checkered pattern. The surrounding image generation device for a work machine according to Claim 1. The surrounding image generation device for a work machine according to Claim 1.

3. The composite image of the first image and the second image is an aerial image of the surroundings of the work machine viewed from above. The surrounding image generation device for a work machine according to Claim 1. The surrounding image generation device for a work machine according to Claim 1.

4. Further comprising a third imaging unit that captures a third image adjacent to the second image, wherein the first image and the second image are replaced with the second image and the third image, and the processing of the first masking unit, the second masking unit, and the image synthesizing unit is executed. The surrounding image generation device for a work machine according to Claim 1. The surrounding image generation device for a work machine according to Claim 1.

5. A surrounding image generation program for a work machine that causes a computer to execute the following processing to generate a surrounding image of the work machine, a first image acquisition process for acquiring a first image around the work machine, A second image acquisition process for acquiring, such that a boundary image region overlaps, a second image around the working machine adjacent to the first image; A first masking process for dividing a first overlapping image region of the first image corresponding to the boundary image region into a first valid pixel region where pixel information of the first image is valid and a first invalid pixel region where pixel information of the first image is invalid and pixel information of the second image is valid; A second masking process for dividing a second overlapping image region of the second image corresponding to the boundary image region into a second valid pixel region where pixel information of the second image is valid and a second invalid pixel region where pixel information of the second image is invalid and pixel information of the first image is valid; An image synthesis process for synthesizing the first image and the second image such that the first valid pixel region of the first overlapping image region overlaps the second invalid pixel region of the second overlapping image region and the first invalid pixel region of the first overlapping image region overlaps the second valid pixel region of the second overlapping image region; A program for generating a surrounding image of a working machine including the above.

6. The first overlapping image region has the first valid pixel region and the first invalid pixel region adjacent to each other alternately and arranged in a checkered pattern, and the second overlapping image region has the second valid pixel region and the second invalid pixel region adjacent to each other alternately and arranged in a checkered pattern. The program for generating a surrounding image of a working machine according to claim 5.

7. The composite image of the first image and the second image is a bird's-eye view image of the surroundings of the working machine seen from above. The program for generating a surrounding image of a working machine according to claim 5.

8. The program further includes a third image acquisition process for imaging and acquiring a third image adjacent to the second image, and the first masking process, the second masking process, and the image synthesis process are executed by replacing the first image and the second image with the second image and the third image. The program for generating a surrounding image of a working machine according to claim 5.

Images