Image processing device, image processing method, and image processing program

Abstract

According to the present invention, a combination unit generates a second overhead image by combining: a first overhead image that combines images from a plurality of imaging devices that are provided to a moving body; and an overlap image that is generated from an aggregate image and overlaps the moving body. On the basis of image quality information for the first overhead image and the aggregate image, a coefficient calculation unit calculates a correction coefficient for correcting the image quality of the overlap image at a boundary portion of the first overhead image and the overlap image. On the basis of the correction coefficient, a correction unit corrects the image quality of the overlap image to be used for combination by the combination unit. The present invention can thereby reduce the irregularity of an image produced by combining a first overhead image and an overlap image that is not included in the first overhead image.

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] A system is known that acquires images of the surroundings of a moving vehicle using an imaging device, repeatedly generates overhead images of the vehicle from the acquired images, and displays the generated overhead images on a display device. In this type of system, areas with large variations in brightness caused by vehicle shadows or reflections, etc., are excluded from the area used to create the historical image, thereby preventing striped areas with large variations in brightness from appearing near the vehicle in the overhead image. Another known method involves cutting out an image of the area below the vehicle that is not included in the imaging range of the imaging device from a previously captured image and superimposing it on the currently captured overhead image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-13994 [Patent Document 2] Japanese Patent Application Laid-Open No. 2015-171106 [Patent Document 3] Japanese Patent Application Laid-Open No. 2014-36326 [Patent Document 4] Japanese Patent Application Publication No. 2019-103344 Summary of the Invention [Problem to be solved by the invention]

[0004] The overhead image generated in real time and the image below the moving object superimposed on the overhead image are captured at different times, and therefore may have different image quality such as brightness. If the difference in image quality is large, for example, the image below the moving object superimposed on the overhead image may stand out, making the overhead image look unnatural.

[0005] The present invention has been made in consideration of the above points, and aims to reduce the unnaturalness of an image obtained by combining an overhead image of the area around a moving body with an overlay image that is not included in the overhead image of the area that overlaps with the moving body. [Means for solving the problem]

[0006] In one aspect of the present invention, an image processing device stores images acquired from a plurality of image capturing devices provided in a mobile object as stored images. do an image storage unit, and generating a superimposed image of an area that is not included in the imaging range of the plurality of imaging devices and that overlaps with the moving object from stored images stored in the image storage unit; do a superimposed image generating unit that generates a first overhead image showing the surroundings of the moving object by synthesizing the images acquired from the plurality of imaging devices, and generates a second overhead image by synthesizing the first overhead image and the superimposed image; do a synthesis unit that synthesizes an image at a boundary between the first overhead image and the superimposed image based on image quality information of the first overhead image and image quality information of a stored image used to generate the superimposed image. Calculating a reference compensation coefficient at each vertex of the rectangular boundary portion, and using the calculated reference compensation coefficient, calculating a compensation coefficient for each pixel of the superimposed image. a coefficient calculation unit, and a unit for adjusting the image quality of the superimposed image to be synthesized by the synthesis unit based on the adjustment coefficient calculated by the coefficient calculation unit; do and a compensation unit. [Effects of the Invention]

[0007] According to the disclosed technology, it is possible to reduce the unnaturalness of an image obtained by combining an overhead image of the area around a moving body with an overlay image that is not included in the overhead image of the area that overlaps with the moving body. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of an image processing system including an image processing device according to a first embodiment. [Figure 2] 2 is a block diagram showing an example of a functional configuration of the image processing device in FIG. 1. FIG. [Figure 3]3 is a block diagram showing an example of the configuration of the image processing device and the information processing device shown in FIG. 2. FIG. [Figure 4] 3 is an explanatory diagram showing an example of an area that can be photographed by the imaging device of FIG. 1 and a projection surface onto which an image is projected by the projection unit of FIG. 2. FIG. [Figure 5] 5 is a plan view showing a vehicle under-area and a projection surface showing the underside of the moving body of FIG. 4. FIG. [Figure 6] 2 is an explanatory diagram showing an example of an overhead image including an under-vehicle image that changes as the moving object of FIG. 1 moves, and an accumulated image. [Figure 7] 3 is an explanatory diagram showing an example of a coefficient calculation method performed by a coefficient calculation unit in FIG. 2. FIG. [Figure 8] 4 is an explanatory diagram showing an example of a method for calculating coefficients used to adjust the brightness of the under-vehicle image by the image processing device of FIG. 2. FIG. [Figure 9] 2 is an explanatory diagram showing an example of a method for generating an under-vehicle image in the image processing device of FIG. 1. FIG. [Figure 10] 3 is a flowchart showing an example of the operation of the image processing device in FIG. 2. [Figure 11] 10 is an explanatory diagram showing an example of a method for calculating coefficients used to adjust the brightness of an under-vehicle image by the image processing device of the second embodiment. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described with reference to the drawings. In the following description, image data will also be simply referred to as an image.

[0010] (First embodiment) Fig. 1 is a conceptual diagram showing an example of an image processing system including an image processing device according to a first embodiment. The image processing system 100 shown in Fig. 1 is mounted on a moving body 200 such as an automobile. Imaging devices 210A, 210B, 210C, and 210D such as cameras are installed on the front, rear, left, and right sides of the moving body 200 in the direction of travel. Hereinafter, when the imaging devices 210A, 210B, 210C, and 210D are not to be distinguished from one another, they will also be referred to as imaging devices 210.

[0011] 1 as long as it is possible to generate an overhead image of the moving body 200. Furthermore, the moving body 200 on which the image processing system 100 is mounted is not limited to an automobile, and may be, for example, a transport robot or a drone operating in a factory. In this case, the image processing system 100 may be installed outside the moving body 200 so as to be able to communicate with the moving body 200.

[0012] The image processing system 100 includes an image processing device 110, an information processing device 120, and a display device 130. In FIG. 1, for ease of understanding, the image processing system 100 is depicted superimposed on an image diagram of a moving object 200 viewed from above. However, in reality, the image processing device 110 and the information processing device 120 are mounted on a control board or the like mounted on the moving object 200, and the display device 130 is installed in a position visible to people inside the moving object 200. Furthermore, the image processing device 110 may be mounted on a control board or the like as part of the information processing device 120, or may be realized by an image processing program executed by the information processing device 120.

[0013] The image processing device 110 is connected to each imaging device 210 via a signal line or wirelessly, and acquires image data showing images of the surroundings of the moving object 200 captured by each imaging device 210. The image processing device 110 performs image processing (adjustment processing) on ​​the image data acquired from each imaging device 210, and outputs the results of the image processing to at least one of the display device 130 and the information processing device 120. For example, the image processing device 110 can output to the display device 130 an overhead image of the surroundings of the moving object 200 and an under-vehicle image below the moving object 200 that is not captured by the imaging device 210.

[0014] The display device 130 may be, for example, a display of a navigation device or the like installed in the mobile object 200. The display device 130 may display an image of the backward direction of the mobile object 200 in real time when the mobile object 200 moves backward (backing up). The display device 130 may also display a map image or the like output from a control unit of the navigation device. The display device 130 may be a display provided on the dashboard or the like of the vehicle, or may be a head-up display that projects an image onto a projection board or a windshield or the like.

[0015] The information processing device 120 includes a computer such as a processor that performs recognition processing and the like based on image data received via the image processing device 110. For example, the information processing device 120 mounted on the mobile object 200 may perform recognition processing of an image represented by image data to detect other mobile objects, traffic lights, signs, white lines on roads, people, and the like, and determine the situation around the mobile object 200 based on the detection results.

[0016] The information processing device 120 may function as a computer that controls each part of the moving body 200. The information processing device 120 may also have an automatic driving control function that controls the movement, stopping, right turns, left turns, etc. of the moving body 200. In this case, the information processing device 120 may recognize an object outside the moving body 200 based on the image generated by the image processing device 110, and track the recognized object.

[0017] Fig. 2 is a block diagram showing an example of the functional configuration of the image processing device 110 in Fig. 1. The image processing device 110 includes an image data acquisition unit 111, a projection unit 112, an under-vehicle image generation unit 113, a coefficient calculation unit 114, a correction unit 115, a synthesis unit 116, an output unit 117, a projection information storage unit 118, and an image accumulation unit 119.

[0018] The image processing device 110 may be realized by hardware or by a combination of hardware and software. For example, at least some of the functions of the projection unit 112, the under-vehicle image generation unit 113, the coefficient calculation unit 114, the correction unit 115, and the synthesis unit 116 may be realized by an image processing program executed by the image processing device 110. Furthermore, some of the functions of the image data acquisition unit 111 or some of the functions of the output unit 117 may be realized by an image processing program executed by the image processing device 110.

[0019] The image data acquisition unit 111 performs an acquisition process to acquire image data IMG (IMGa, IMGb, IMGc, IMGd) showing images of the surroundings of the moving object 200 captured by each of the imaging devices 210 (210A, 210B, 210C, 210D). The image data acquisition unit 111 performs a storage process to store the image data IMG captured by at least the imaging devices 210 installed in the traveling direction of the moving object 200 in the image storage unit 119 as stored image data.

[0020] The projection unit 112 performs a conversion process to convert the coordinates of each pixel included in the image data IMGa-IMGd acquired by the image data acquisition unit 111 into coordinates when each pixel is projected onto the projection surface 230 (FIG. 4). The projection unit 112 also performs a conversion process to convert the coordinates of each pixel included in the under-vehicle image generated by cutting out a part of the stored image stored in the image storage unit 119 into coordinates when each pixel is projected onto the projection surface 230. In other words, the projection unit 112 performs a conversion process to convert each of the two-dimensional images IMGa-IMGd and the under-vehicle image acquired by the imaging devices 210A-210D into a three-dimensional image corresponding to the projection surface 230.

[0021] The under-vehicle image generation unit 113 performs generation processing to generate an under-vehicle image by cutting out a portion of a plurality of accumulated images accumulated in the image accumulation unit 119. The under-vehicle image generation unit 113 also outputs the generated under-vehicle image and the coordinates of the under-vehicle image on the projection plane 230 to the synthesis unit 116. The under-vehicle image generation unit 113 acquires the coordinates of the under-vehicle image on the projection plane 230 from the projection unit 112. The under-vehicle image is an example of a superimposed image of an area that is not included in the shooting range of the image capture devices 210A-210D and that overlaps with the moving object 200. The under-vehicle image generation unit 113 is an example of a superimposed image generation unit that generates an under-vehicle image from the accumulated images accumulated by the image accumulation unit 119.

[0022] The coefficient calculation unit 114 performs a coefficient calculation process to calculate a correction coefficient for matching pixel information of the overhead-view image with pixel information of the under-vehicle image at the boundary between the overhead-view image and the under-vehicle image before being combined by the combination unit 116. For example, the coefficient calculation unit 114 calculates, as the correction coefficient, a ratio between pixel information of the overhead-view image and pixel information of a stored image from which the under-vehicle image is cut out, at a predetermined number of sampling points provided around the under-vehicle image.

[0023] Hereinafter, the compensation coefficients will also be simply referred to as coefficients. Furthermore, although an example in which pixel information is luminance will be described below, the pixel information may be color information, or may include both luminance and color information. That is, luminance may be represented by both a luminance value and color information. Pixel information is an example of image quality information of an image.

[0024] The correction unit 115 performs a correction process to correct the brightness of the under-vehicle image based on the correction coefficient calculated by the coefficient calculation unit 114. This reduces the brightness difference between the overhead-view image and the under-vehicle image acquired at different times, and makes it possible to generate an overhead-view image with reduced sense of incongruity even when the under-vehicle image is included in the overhead-view image. In other words, it is possible to reduce the likelihood that the overhead-view image will look unnatural when the under-vehicle image is synthesized.

[0025] The synthesis unit 116 performs synthesis processing to synthesize three-dimensional image data generated by the projection unit 112 corresponding to each of the image data IMGa-IMGd and generate an overhead image. The synthesis unit 116 also performs synthesis processing to synthesize the under-vehicle image clipped by the under-vehicle image generation unit 113 with the overhead image and generate an overhead image including the under-vehicle image. The overhead image generated by synthesizing the image data IMGa-IMGd is an example of a first overhead image. The overhead image including the under-vehicle image is an example of a second overhead image.

[0026] The output unit 117 outputs image data such as an overhead image to the display device 130. The display device 130 displays an image according to the image data received from the output unit 117.

[0027] The projection information storage unit 118 holds, for example, projection information that associates the two-dimensional coordinates of each pixel of the image data IMGa-IMGd with the three-dimensional coordinates of the projection plane 230. The projection information is referenced by the projection unit 112. The projection information storage unit 118 also holds projection information that associates the two-dimensional coordinates of each pixel of the image data IMGa and IMGb in the traveling direction of the moving object 200 with the three-dimensional coordinates of the area below the vehicle on the projection plane 230.

[0028] Image storage unit 119 stores, as stored images, the image data acquired by image data acquisition unit 111. Note that image storage unit 119 may store, as stored images, only the image data acquired by image data acquisition unit 111 from imaging device 210A or imaging device 210B installed in the traveling direction of moving body 200.

[0029] Fig. 3 is a block diagram showing an example of the configuration of the image processing device 110 and the information processing device 120 in Fig. 2. Since the image processing device 110 and the information processing device 120 have the same configuration, the configuration of the image processing device 110 will be described below. For example, the image processing device 110 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an auxiliary storage device 14, a connection device 15, and a drive device 16, which are interconnected via a bus BUS.

[0030] The CPU 11 operates by executing programs stored in the ROM 12, and controls the overall operation of the image processing device 110. The ROM 12 stores the programs executed by the CPU 11 and various data used by the programs. For example, the programs executed by the CPU 11 include an image processing program and a boot program such as a BIOS (Basic Input / Output System) or an EFI (Extensible Firmware Interface).

[0031] The RAM 13 may be a volatile memory such as a dynamic random access memory (DRAM) or a static random access memory (SRAM), or a nonvolatile memory such as a flash memory in which information can be electrically rewritten. The RAM 13 may be used as a work area when the CPU 11 develops various programs installed in the auxiliary storage device 14 into executable programs.

[0032] The auxiliary storage device 14 stores various information such as various programs and data used by the various programs transferred from outside the image processing device 110 via the drive device 16. For example, the auxiliary storage device 14 may be an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The various information stored in the auxiliary storage device 14 may be transferred from a network connected to the connection device 15.

[0033] The connection device 15 transmits and receives information such as data between the image processing device 110 and external devices or networks such as the imaging device 210, the information processing device 120, and the display device 130 connected to the image processing device 110.

[0034] For example, the connection device 15 has a signal terminal that receives image data IMG from the imaging device 210 in Fig. 2 and transfers the received image data IMG to the image data acquisition unit 111 in Fig. 2. The connection device 15 has a signal terminal that outputs image data output from the output unit 117 in Fig. 2 to the display device 130. The connection device 15 has a signal terminal that inputs and outputs information to and from the information processing device 120.

[0035] The drive device 16 has, for example, an interface to which a computer-readable recording medium 140 is connected, receives information such as various programs or various data stored on the recording medium 140, and transfers the received information to the CPU 11 or the auxiliary storage device 14. For example, the recording medium 140 connected to the drive device 16 is a CD-ROM (Compact Disc Read Only Memory), a flexible disk, a magneto-optical disk, or a semiconductor memory such as a flash memory that records information optically, electrically, or magnetically.

[0036] FIG. 4 is an explanatory diagram showing an example of an area that can be photographed by the imaging device 210 in FIG. 1 and a projection surface 230 onto which an image is projected by the projection unit 112 in FIG.

[0037] Imaging device 210A can capture an image of a subject included in area 220A on the front side of moving object 200. Imaging device 210B can capture an image of a subject included in area 220B on the rear side of moving object 200. Imaging device 210C can capture an image of a subject included in area 220C on the left side of moving object 200. Imaging device 210D can capture an image of a subject included in area 220D on the right side of moving object 200.

[0038] Areas 220A-220D partially overlap one another. Therefore, image capturing devices 210A-210D can capture images of the surrounding area excluding the area below moving object 200. Note that although each image capturing device 210 can actually capture images up to the horizon, areas 220A-220D are shown as fan-shaped areas in FIG. 4.

[0039] 2 synthesizes the image data of areas 220A-220D acquired in real time by imaging devices 210A-210D to generate an overhead image shown in three-dimensional coordinates on projection plane 230. Furthermore, synthesizing unit 116 synthesizes the overhead image shown on projection plane 230 with the under-vehicle image IMGU whose brightness has been adjusted by adjustment unit 115 to generate an overhead image including the under-vehicle image IMGU.

[0040] Fig. 5 is a plan view showing the vehicle under-area UA, which shows the area below the moving object 200 in Fig. 4, and the projection surface 230. The vehicle under-area UA is not included in the areas 220A-220D in Fig. 4, and is an area from which images are not acquired by the imaging devices 210A-210D. Therefore, strictly speaking, the vehicle under-area UA may also include a small amount of the area around the moving object 200.

[0041] 2 sets sampling points Pa0, Pb0, Pa2, and Pb2 at the positions of four vertices of the boundary between regions 220A-220D and the vehicle-under-region UA. For example, the coefficient calculation unit 114 sets sampling points Pa0 and Pb0 at both left and right ends of a region adjacent to the rear end of the vehicle-under-region UA ​​on the side opposite the traveling direction of the moving object 200. The coefficient calculation unit 114 also sets sampling points Pa2 and Pb2 at both left and right ends of a region adjacent to the front end of the vehicle-under-region UA ​​in the traveling direction of the moving object 200. Although not particularly limited, for example, each of sampling points Pa0, Pb0, Pa2, and Pb2 corresponds to one pixel in the overhead image.

[0042] The positions of the sampling points Pa0, Pb0, Pa2, and Pb2 may be set in advance and stored in the ROM 12, RAM 13, or auxiliary storage device 14 of FIG. 3. The sampling points Pa0 and Pb0 may be set at positions adjacent to the left and right of the rear end of the vehicle under area UA. The sampling points Pa2 and Pb2 may be set at positions adjacent to the left and right of the front end of the vehicle under area UA.

[0043] The coefficient calculation unit 114 calculates an average luminance, which is the average of the luminance of nine pixels, consisting of the pixel at each sampling point Pa0, Pb0 and eight pixels surrounding each sampling point Pa0, Pb0, in the overhead image acquired in real time. The coefficient calculation unit 114 also calculates an average luminance, which is the average of the luminance of nine pixels, consisting of the pixel at each sampling point Pa0, Pb0 and eight pixels surrounding each sampling point Pa0, Pb0, in past stored images stored in the image storage unit 119. The coefficient calculation unit 114 then calculates a compensation coefficient, which is the ratio of the average luminance to each sampling point Pa0, Pb0. The coefficient calculation unit 114 sets the compensation coefficient for each sampling point Pa2, Pb2 to 1.0 regardless of the luminance. The compensation coefficients for sampling points Pa0, Pb0, Pa2, and Pb2 are examples of reference compensation coefficients.

[0044] The coefficient calculation unit 114 uses multiple pixels, including the pixels at each sampling point Pa0 and Pb0, to calculate the average brightness, so that even if the brightness of the pixel at sampling point Pa0 shows an abnormal value due to noise or the like, it is possible to calculate an average brightness within the normal range.

[0045] The coefficient calculation unit 114 calculates reference compensation coefficients at the positions of the vertices of the rectangular boundary between the areas 220A-220D and the vehicle under image IMGU, and calculates compensation coefficients for each pixel of the vehicle under image IMGU using the reference compensation coefficients. This allows the coefficient calculation unit 114 to calculate compensation coefficients for each pixel of the vehicle under image IMGU based on the calculation of a minimum number of reference compensation coefficients, thereby reducing the amount of calculation.

[0046] The compensation unit 115 compensates for the luminance of each pixel of the under-vehicle image IMGU based on the compensation coefficients for the sampling points Pa0, Pb0 and the compensation coefficient (=1.0) for the sampling points Pa2, Pb2 calculated by the coefficient calculation unit 114. Compensation of the luminance of each pixel of the under-vehicle image IMGU will be described with reference to FIG.

[0047] Fig. 6 is an explanatory diagram showing an example of an overhead image including a vehicle under image IMGU and an accumulated image that change as the moving object 200 of Fig. 1 moves. For example, the overhead image including the vehicle under image IMGU is displayed in real time on the display device 130. The moving object 200 moves from a stopped state shown in Fig. 6(a) in the direction of the white arrow shown in the overhead image, and transitions sequentially to the states shown in Fig. 6(b) and Fig. 6(c).

[0048] The stored images are images that the image data acquisition unit 111 acquires from the imaging device 210A or 210B located in the traveling direction of the moving body 200 and stores in the image storage unit 119. Note that, for ease of understanding, Fig. 6 shows the display state of the stored images, but the stored images are not displayed on the display device 130 or the like except during development or maintenance of the image processing device 110.

[0049] 6(a), all of the stored images are images of the area ahead of the vehicle under area UA of the moving object 200, so no under-vehicle image IMGU is generated. Therefore, the synthesis unit 116 generates an overhead image that does not include the under-vehicle image IMGU, and displays it on the display device 130 via the output unit 117. The coefficient calculation unit 114 does not calculate coefficients because there are no images of sampling points Pa0 and Pb0 in the stored images.

[0050] In FIG. 6(b), the stored image acquired in FIG. 6(a) includes a portion of the vehicle under area UA in FIG. 6(b). Therefore, the synthesis unit 116 generates an overhead image synthesized with the vehicle under image IMGU of a portion of the vehicle under area UA, and displays the generated overhead image on the display device 130 via the output unit 117. The coefficient calculation unit 114 does not calculate a compensation coefficient because the stored image does not include images at sampling points Pa0 and Pb0. Therefore, the compensation unit 115 does not compensate for the brightness of the vehicle under image IMGU included in the overhead image.

[0051] In Fig. 6(c), the stored images acquired in Fig. 6(a) and (b) include the entire vehicle under area UA in Fig. 6(c). Therefore, the synthesis unit 116 synthesizes the vehicle under image IMGU of the vehicle under area UA with the overhead image, and displays the synthesized image on the display device 130 via the output unit 117. The coefficient calculation unit 114 calculates a correction coefficient to be used to correct the brightness of the vehicle under image IMGU because the stored images include images of sampling points Pa0 and Pb0.

[0052] The adjustment unit 115 adjusts the brightness of the under-vehicle image IMGU included in the overhead-view image using the adjustment coefficient. That is, the brightness of the under-vehicle image IMGU included in the overhead-view image in FIG. 6(c) is adjusted to match the brightness of the overhead-view image acquired in real time. This allows the image processing device 110 to generate an under-vehicle image IMGU that does not look unnatural with the overhead-view image generated in real time. That is, it is possible to reduce the likelihood of the image appearing unnatural when the overhead-view image and the under-vehicle image IMGU are combined.

[0053] Fig. 7 is an explanatory diagram showing an example of a method for calculating the adjustment coefficients by the coefficient calculation unit 114 of Fig. 2. In Fig. 7, the overhead image and accumulated image including the under-vehicle image IMGU are similar to the overhead image and accumulated image of Fig. 6(c). The white arrow shown in Fig. 7 indicates the traveling direction of the moving object 200.

[0054] The coefficient calculation unit 114 calculates average brightnesses Ar0, Br0, which are the average brightnesses of the pixels at sampling points Pa0, Pb0 located on the opposite side of the traveling direction in the overhead image acquired in real time and the surrounding eight pixels. Furthermore, when the stored image includes pixels at sampling points Pa0, Pb0, the coefficient calculation unit 114 calculates average brightnesses Au0, Bu0, which are the average brightnesses of the pixels at sampling points Pa0, Pb0 in the stored image and the surrounding eight pixels.

[0055] The coefficient calculation unit 114 then sets the ratio Ar0 / Au0 of the average brightnesses Ar0 and Au0 as the compensation coefficient a0, and the ratio Br0 / Bu0 of the average brightnesses Br0 and Bu0 as the compensation coefficient b0. Note that the number of pixels used to calculate the average brightnesses Ar0, Br0, Au0, and Bu0 is not limited to nine pixels, as long as it is two or more pixels. Note that, as described above, the coefficient calculation unit 114 sets the compensation coefficients a1 and b1 of the sampling points Pa2 and Pb2 located in the traveling direction to "1.0."

[0056] For example, when the compensation coefficients a0 and b0 are greater than 1.0, the luminance of pixels in the under-vehicle image IMGU in the area close to the sampling points Pa0 and Pb0 is considered to be lower than the luminance of the overhead-view image. When the compensation coefficients a0 and b0 are less than 1.0, the luminance of pixels in the under-vehicle image IMGU in the area close to the sampling points Pa0 and Pb0 is considered to be higher than the luminance of the overhead-view image. Therefore, by having the compensation unit 115 multiply the luminance of pixels in the under-vehicle image IMGU in the area close to the sampling points Pa0 and Pb0 by the compensation coefficient, the luminance of the under-vehicle image IMGU in the area close to the sampling points Pa0 and Pb0 can be adjusted to match the luminance of the overhead-view image.

[0057] Fig. 8 is an explanatory diagram showing an example of a method for calculating compensation coefficients used to adjust the brightness of the vehicle under image IMGU by the image processing device 110 of Fig. 2. The method for calculating compensation coefficients shown in Fig. 8 is started when any of the stored images stored in the image storage unit 119 contains a predetermined number of pixels (for example, 9 pixels) including sampling points Pa0 and Pb0 in the vehicle under area UA at the current position of the moving object 200.

[0058] 7, coefficient calculation unit 114 first calculates the ratio Ar0 / Au0 between the average brightness Ar0 of the overhead image at sampling point Pa0 and the average brightness Au0 of the accumulated image, and sets the calculated ratio Ar0 / Au0 as the compensation coefficient a0 at sampling point Pa0. Coefficient calculation unit 114 also calculates the ratio Br0 / Bu0 between the average brightness Br0 of the overhead image at sampling point Pb0 and the average brightness Bu0 of the accumulated image, and sets the calculated ratio Br0 / Bu0 as the compensation coefficient b0 at sampling point Pb0.

[0059] Next, the coefficient calculation unit 114 calculates the compensation coefficient of each pixel in the vehicle under area UA located between the sampling points Pa0 and Pa2 by linear interpolation using the compensation coefficient a0 and the compensation coefficient = 1.0 at the sampling point Pa2. Similarly, the coefficient calculation unit 114 calculates the compensation coefficient of each pixel in the vehicle under area UA located between the sampling points Pb0 and Pb2 by linear interpolation using the compensation coefficient b0 and the compensation coefficient = 1.0 at the sampling point Pb2.

[0060] Furthermore, the coefficient calculation unit 114 calculates the compensation coefficients of each pixel aligned in the width direction of the moving body 200 in the vehicle under area UA by linear interpolation using, among the compensation coefficients calculated by linear interpolation, compensation coefficients a and b that are located at the same position in the vehicle under area UA in the traveling direction of the moving body 200. Note that the coefficient calculation unit 114 may calculate the compensation coefficients by linear interpolation for each of a plurality of pixels, rather than for each pixel.

[0061] As described above, the coefficient calculation unit 114 can calculate the compensation coefficients used to adjust the brightness of all pixels in the vehicle-under area UA. The coefficient calculation unit 114 calculates, by linear interpolation, the compensation coefficients for each pixel in the vehicle-under area UA located between the sampling points Pa0 and Pa2 and the compensation coefficients for each pixel in the vehicle-under area UA located between the sampling points Pb0 and Pb2. At this time, at the front end of the vehicle-under image IMGU, which is in the traveling direction of the moving object 200, the difference in pixel value (e.g., brightness value) between the overhead-view image and the vehicle-under image IMGU is small, so the compensation coefficients for the sampling points Pa2 and Pb2 can be set to "1.0" regardless of the image.

[0062] Furthermore, the coefficient calculation unit 114 calculates the compensation coefficients for the pixels along the width direction of the vehicle under-image IMGU by linearly interpolating the compensation coefficients calculated for the pixels on both sides in the width direction. This makes it possible to generate the compensation coefficients more easily than when the compensation coefficients are obtained by calculating the ratio between the luminance of the overhead-view image and the luminance of the stored image for each pixel, thereby reducing the calculation load on the coefficient calculation unit 114.

[0063] For the sake of simplicity, assume that in the under-vehicle area UA, 10 pixels are aligned in the direction of travel of the moving object 200, and 5 pixels are aligned in the direction perpendicular to the direction of travel of the moving object (i.e., the width direction). For example, if the adjustment coefficients a0 and b0 are 0.5, the adjustment coefficients of the 10 pixels aligned from the rear to the front of the moving object 200 in the under-vehicle area UA increase by approximately 0.0556 from the rear to the front, becoming 0.5, 0.556, 0.611, ..., 0.889, 0.944, and 1.0.

[0064] Also, for example, if the compensation coefficient b0 is 0.4, the compensation coefficients of the 10 pixels aligned from sampling point Pb0 to sampling point Pb2 of the moving body 200 in the under-vehicle area UA increase by approximately 0.0667 from rear to front, becoming 0.4, 0.467, 0.533, ..., 0.867, 0.933, and 1.0.

[0065] For example, when the compensation coefficient a0 is 0.5 and the compensation coefficient b0 is 0.4, the coefficient calculation unit 114 sets the compensation coefficients of the three pixels in the under-vehicle area UA, which are arranged between the pixel on the left side of the direction of travel with a compensation coefficient of 0.611 and the pixel on the right side of the direction of travel with a compensation coefficient of 0.533, to 0.553, 0.572, and 0.592.

[0066] Fig. 9 is an explanatory diagram showing an example of a method for generating an under-vehicle image IMGU in the image processing device 110 of Fig. 1. For example, the image data acquisition unit 111 of Fig. 2 sequentially stores, in the image storage unit 119, image data IMG0, IMG1, IMG2, IMG3, IMG4, IMG5, and IMG6 acquired at predetermined time intervals by the imaging device 210A installed ahead in the traveling direction of the moving object 200. Hereinafter, the image data IMG0-IMG6 stored in the image storage unit 119 will also be referred to as stored images IMG0-IMG6.

[0067] Each of the stored images IMG0-IMG6 actually includes a subject image that spreads out in a fan shape as shown in FIG. 4. However, for ease of explanation, FIG. 9 shows only images that can overlap with the under-vehicle area UA that moves as the moving object 200 moves. Each of the image data IMG0-IMG6 includes multiple sub-image data IMG indicated by sub-numbers such as "-1" and "-2." Furthermore, the under-vehicle image IMGU displayed in the under-vehicle area UA is generated by combining up to five sub-image data IMG. The number of sub-image data IMG used to generate the under-vehicle image IMGU varies depending on the frame rate of the imaging device 210 and the speed of the moving object 200.

[0068] At time T0, the moving object 200 starts moving. When the moving object 200 starts moving, the image storage unit 119 does not hold any stored images corresponding to the vehicle under area UA. Therefore, the vehicle under image generation unit 113 does not generate an vehicle under image IMGU.

[0069] Next, at time T1, the under-vehicle image generation unit 113 determines that the sub-image IMG0-1 of the accumulated image IMG0 at time T0 is included in the under-vehicle area UA due to movement of the moving object 200. Therefore, the under-vehicle image generation unit 113 cuts out the sub-image IMG0-1 from the sub-image IMG0.

[0070] The under-vehicle image generation unit 113 outputs the two-dimensional coordinates of each pixel of the sub-image IMG0 to the projection unit 112. The projection unit 112 generates three-dimensional coordinates on the projection plane 230 of each pixel of the sub-image IMG0-1 based on the projection information held in the projection information storage unit 118, and outputs the generated three-dimensional coordinates to the under-vehicle image generation unit 113. The under-vehicle image generation unit 113 outputs three-dimensional sub-image data IMG0-1 including the three-dimensional coordinates received from the projection unit 112 to the synthesis unit 116. The projection unit 112 converts the two-dimensional image data IMGa-IMGd generated in real time by the imaging devices 210A-210D into three-dimensional image data IMGa-IMGd and outputs the three-dimensional image data IMGa-IMGd to the synthesis unit 116.

[0071] The synthesis unit 116 synthesizes the three-dimensional sub-images IMG0-1 received from the under-vehicle image generation unit 113 and the three-dimensional image data IMGa-IMGd at time T1 received from the projection unit 112 to generate an overhead image including the under-vehicle image IMGU. The synthesis unit 116 outputs the generated overhead image to the display device 130 via the output unit 117. This allows the overhead image including the under-vehicle image IMGU to be displayed on the display device 130.

[0072] Next, at time T2, the under-vehicle image generation unit 113 determines that the sub-image IMG0-1 at time T0 and the sub-image IMG1-1 at time T1 are included in the under-vehicle area UA due to movement of the moving object 200. Therefore, the under-vehicle image generation unit 113 outputs the two-dimensional coordinates of each pixel of the sub-images IMG0-1 and IMG1-1 to the projection unit 112.

[0073] After that, similarly to time T1, the synthesis unit 116 receives the three-dimensional sub-image data IMG0-1 and IMG1-1 from the under-vehicle image generation unit 113, and receives the three-dimensional image data IMGa-IMGd from the projection unit 112. Then, the synthesis unit 116 synthesizes the three-dimensional sub-images IMG0-1 and IMG1-1 with the three-dimensional image data IMGa-IMGd at time T2 to generate an overhead image including the under-vehicle image IMGU, and causes the display device 130 to display the generated image.

[0074] At times T3 and T4, the under-vehicle image generation unit 113, the projection unit 112, and the synthesis unit 116 operate in the same manner as at times T1 and T2. Then, at time T3, an overhead image including an under-vehicle image IMGU, which is obtained by synthesizing the three-dimensional sub-images IMG0-1, IMG1-1, and IMG2-1 with the three-dimensional image data IMGa-IMGd at time T3, is displayed on the display device 130.

[0075] At time T4, the display device 130 displays an overhead image including an under-vehicle image IMGU, which is a combination of the three-dimensional sub-images IMG0-1, IMG1-1, IMG2-1, and IMG3-1 and the three-dimensional image data IMGa-IMGd at time T4.

[0076] Note that until time T4, since there are no images at sampling points Pa0 and Pb0 in the stored image, the coefficient calculation unit 114 does not calculate the compensation coefficient. Therefore, the compensation unit 115 does not compensate for the luminance of the under-vehicle image IMGU included in the overhead image. By suppressing the calculation of the compensation coefficient by the coefficient calculation unit 114 and the luminance compensation by the compensation unit 115 during the period when there are no images at sampling points Pa0 and Pb0 in the stored image, the processing load on the image processing device 110 can be reduced. This makes it possible to reduce, for example, the power consumption of the image processing device 110.

[0077] At time T5, the stored image includes images at sampling points Pa0 and Pb0. Therefore, the coefficient calculation unit 114 calculates the compensation coefficient in the same manner as described in Fig. 8. When the compensation coefficient has been calculated, the under-vehicle image generation unit 113 outputs a three-dimensional under-vehicle image IMGU including the three-dimensional coordinates received from the projection unit 112 to the compensation unit 115. The under-vehicle image IMGU includes sub-image data IMG0-1, IMG1-1, IMG2-1, IMG3-1, and IMG4-1.

[0078] The adjustment unit 115 adjusts the brightness of the three-dimensional under-vehicle image IMGU received from the under-vehicle image generation unit 113 in accordance with the adjustment coefficient, and outputs the brightness-adjusted under-vehicle image IMGU to the synthesis unit 116. Thereafter, as in the case of times T1 to T4, the synthesis unit 116 combines the brightness-adjusted three-dimensional under-vehicle image IMGU received from the adjustment unit 115 with the three-dimensional image data IMGa-IMGd at time T5 received from the projection unit 112.

[0079] The synthesis unit 116 then generates an overhead image including the under-vehicle image IMGU and outputs the generated overhead image to the display device 130 via the output unit 117. This makes it possible to display on the display device 130 an overhead image with reduced sense of incongruity, including the under-vehicle image IMGU whose luminance has been adjusted to match the image data IMGa-IMGd generated in real time. In other words, even if the overhead image and the under-vehicle image IMGU were generated at different times and have different luminances, it is possible to display on the display device 130 an overhead image with reduced sense of incongruity due to the difference in luminance. Note that, after time T5, as long as the moving object 200 continues to move in the same direction, the under-vehicle image IMGU is displayed in all areas of the under-vehicle area UA.

[0080] Next, at time T6, the under-vehicle image generation unit 113 determines that the sub-image IMG0-1 is out of the under-vehicle area UA, and that the sub-images IMG1-1, IMG2-1, IMG3-1, IMG4-1, and IMG5-1 are included in the under-vehicle area UA. After this, the under-vehicle image generation unit 113, the projection unit 112, and the synthesis unit 116 operate in the same manner as at time T5, and a bird's-eye view image with reduced discomfort, including the under-vehicle image IMGU with its brightness adjusted, is displayed on the display device 130.

[0081] Fig. 10 is a flow diagram showing an example of the operation of the image processing device 110 in Fig. 2. That is, Fig. 10 shows an example of an image processing method by the image processing device 110, and shows an example of an image processing program executed by the image processing device 110.

[0082] First, in step S10, the image processing device 110 initializes a counter value n to "0." Next, in step S12, the image data acquisition unit 111 of the image processing device 110 acquires images at time Tn from the imaging devices 210A-210D. Next, in step S14, the image data acquisition unit 111 accumulates the images acquired by the imaging device 210 (e.g., 210A) installed in the traveling direction of the moving body 200 in the image accumulation unit 119 as accumulated images.

[0083] Next, in step S16, the image processing device 110 determines whether any of the stored images stored in the image storage unit 119 meets the conditions for image quality adjustment. For example, in FIG. 9, the conditions for image quality adjustment are not met at times T0-T4, and the conditions for image quality adjustment are met at times T5 and T6. If the conditions for image quality adjustment are met, the process proceeds to step S18, and if the conditions for image quality adjustment are not met, the process proceeds to step S20. For example, the process of step S16 is executed by the under-vehicle image generation unit 113.

[0084] In step S18, the coefficient calculation unit 114 of the image processing device 110 calculates a compensation coefficient using the method shown in Fig. 8. The compensation unit 115 of the image processing device 110 uses the calculated compensation coefficient to adjust the image quality of the three-dimensional under-vehicle image IMGU that is generated from the stored image and aligned with the projection plane 230. For example, the image processing device 110 adjusts the brightness of the three-dimensional under-vehicle image IMGU based on the calculated compensation coefficient. After step S18, the image processing device 110 proceeds to step S20.

[0085] In step S20, the image processing device 110 displays the three-dimensional under-vehicle image IMGU cut out from the stored image on the display device 130, and proceeds to step S22. If the image processing device 110 determines in step S16 that the conditions for image quality adjustment are met, it displays the under-vehicle image IMGU with adjusted image quality on the display device 130. If the image processing device 110 does not determine in step S16 that the conditions for image quality adjustment are met, it displays the under-vehicle image IMGU with unadjusted image quality on the display device 130. Note that the image processing device 110 omits the processing of step S20 if there is no under-vehicle image IMGU, as shown at time T0 in FIG. 9.

[0086] Next, in step S22, the image processing device 110 displays an overhead image generated from the images acquired from the image capturing devices 210A-210D at time Tn on the display device 130. For example, the processes of steps S20 and S22 are executed by the output unit 117.

[0087] Next, in step S24, if the image processing device 110 continues the display processing of the overhead image, the process proceeds to step S26, and if the image processing device 110 stops the display processing of the overhead image, the process shown in Fig. 10 ends. In step S26, the image processing device 110 updates the counter value n by "1" and returns the process to step S12.

[0088] The image processing device 110 may execute the processes of steps S20 and S22 as a single process. In this case, if the conditions for adjusting the image quality are met, the image processing device 110 generates an overhead image by combining the under-vehicle image IMGU whose image quality has been adjusted with the image data IMG acquired in real time. On the other hand, if the conditions for adjusting the image quality are not met, the image processing device 110 generates an overhead image by combining the under-vehicle image IMGU whose image quality has not been adjusted with the image data IMG acquired in real time. If there is no under-vehicle image IMGU, the image processing device 110 generates an overhead image from only the image data IMG acquired in real time.

[0089] As described above, in this embodiment, the adjustment unit 115 adjusts the brightness of the under-vehicle image based on the adjustment coefficient calculated by the coefficient calculation unit 114. This allows the image processing device 110 to generate an under-vehicle image IMGU that does not appear unnatural when compared to an overhead-view image generated in real time. That is, it is possible to reduce the likelihood of the image appearing unnatural when the overhead-view image and the under-vehicle image IMGU are combined. In other words, even if the overhead-view image and the under-vehicle image IMGU are generated at different times and have different brightnesses, it is possible to display an overhead image on the display device 130 with reduced unnaturalness due to the brightness difference.

[0090] The coefficient calculation unit 114 calculates reference compensation coefficients at the positions of each vertex of the rectangular boundary between the overhead-view image and the vehicle under-image before they are combined by the combination unit 116, and calculates compensation coefficients for each pixel of the vehicle under-image IMGU using the reference compensation coefficients. This allows the coefficient calculation unit 114 to calculate compensation coefficients for each pixel of the vehicle under-image IMGU based on the calculation of a minimum number of reference compensation coefficients, thereby reducing the amount of calculation.

[0091] The coefficient calculation unit 114 calculates, by linear interpolation, the compensation coefficients of each pixel in the vehicle-under area UA located between the sampling points Pa0 and Pa2 and the compensation coefficients of each pixel in the vehicle-under area UA located between the sampling points Pb0 and Pb2. At this time, at the front end of the vehicle-under image IMGU, which is in the traveling direction of the moving object 200, the difference in pixel value (e.g., luminance value) of the pixels between the overhead-view image and the vehicle-under image IMGU is small, so the compensation coefficients of the sampling points Pa2 and Pb2 can be set to "1.0" regardless of the image.

[0092] Furthermore, the coefficient calculation unit 114 calculates the compensation coefficients for the pixels along the width direction of the vehicle under-image IMGU by linearly interpolating the compensation coefficients calculated for the pixels on both sides in the width direction. This makes it possible to generate the compensation coefficients more easily than when the compensation coefficients are obtained by calculating the ratio between the luminance of the overhead-view image and the luminance of the stored image for each pixel, thereby reducing the calculation load on the coefficient calculation unit 114.

[0093] During a period when there are no images at sampling points Pa0 and Pb0 in the stored image, the calculation of the compensation coefficient by coefficient calculation unit 114 and the compensation of luminance by compensation unit 115 are suppressed, thereby reducing the processing load on image processing device 110. This makes it possible to reduce, for example, the power consumption of image processing device 110.

[0094] The coefficient calculation unit 114 uses multiple pixels, including the pixels at each sampling point Pa0, Pb0, to calculate the average brightness, so that even if the brightness of the pixels at each sampling point Pa0, Pb0 indicates an abnormal value due to noise or the like, it is possible to calculate an average brightness within the normal range.

[0095] (Second embodiment) Fig. 11 is an explanatory diagram showing an example of a method for calculating compensation coefficients used to adjust the brightness of an under-vehicle image by the image processing device of the second embodiment. Elements similar to those in the above-described embodiment are described using the same reference numerals. The image processing device 110 of this embodiment and the image processing system 100 including the image processing device 110 have the same configurations as those in Figs. 1 to 3, and are mounted on, for example, a moving object 200. Detailed description of processes similar to the compensation coefficient calculation methods described in Figs. 7 and 8 will be omitted.

[0096] The method of calculating the correction coefficient shown in Figure 11, similar to Figure 9, is started when one of the stored images stored in the image storage unit 119 contains a predetermined number of pixels (e.g., 9 pixels) including sampling points Pa0 and Pb0 in the vehicle under area UA at the current position of the moving body 200.

[0097] In this embodiment, sampling point Pa1 is set between sampling points Pa0 and Pa2, and sampling point Pb1 is set between sampling points Pb0 and Pb2. The method of calculating compensation coefficient a1 at sampling point Pa1 and the method of calculating compensation coefficient b1 at sampling point Pb1 are the same as the method of calculating compensation coefficients a0 and b0 at sampling points Pa0 and Pb0 shown in Figure 8. The compensation coefficients at sampling points Pa2 and Pb2 are set to "1.0", the same as in Figure 8.

[0098] The coefficient calculation unit 114 determines the ratio Ar1 / Au1 of the average brightness Ar1 of the overhead image at sampling point Pa1 to the average brightness Au1 of the accumulated image as the compensation coefficient a1. The coefficient calculation unit 114 also determines the ratio Br1 / Bu1 of the average brightness Br1 of the overhead image at sampling point Pb1 to the average brightness Bu1 of the accumulated image as the compensation coefficient b1.

[0099] The coefficient calculation unit 114 then calculates the compensation coefficient for each pixel in the vehicle under area UA located between the sampling points Pa0 and Pa1 by linear interpolation using the compensation coefficient a0 for the sampling point Pa0 and the compensation coefficient a1 for the sampling point Pa1. The coefficient calculation unit 114 also calculates the compensation coefficient for each pixel in the vehicle under area UA located between the sampling points Pa1 and Pa2 by linear interpolation using the compensation coefficient a1 for the sampling point Pa1 and the compensation coefficient a2 for the sampling point Pa2.

[0100] Similarly, the coefficient calculation unit 114 calculates the compensation coefficient for each pixel in the vehicle under area UA located between the sampling points Pb0 and Pb1 by linear interpolation using the compensation coefficient b0 for the sampling point Pb0 and the compensation coefficient b1 for the sampling point Pb1. Also, the coefficient calculation unit 114 calculates the compensation coefficient for each pixel in the vehicle under area UA located between the sampling points Pb1 and Pb2 by linear interpolation using the compensation coefficient b1 for the sampling point Pb1 and the compensation coefficient b2 for the sampling point Pb2.

[0101] That is, the coefficient calculation unit 114 uses the correction coefficients a0 and a1 calculated at two adjacent sampling points (e.g., Pa0 and Pa1) along the traveling direction of the moving body 200 to calculate the correction coefficient of the pixel between the two adjacent sampling points.

[0102] 8, the coefficient calculation unit 114 calculates, by linear interpolation, the compensation coefficients of each pixel aligned in the width direction of the moving object 200 in the vehicle under area UA based on the compensation coefficients a and b that are located at the same position in the traveling direction of the moving object 200 in the vehicle under area UA, out of the compensation coefficients calculated by linear interpolation. Note that the coefficient calculation unit 114 may calculate the compensation coefficients by linear interpolation for each of a plurality of pixels, rather than for each pixel.

[0103] The coefficient calculation unit 114 then calculates a compensation coefficient to be used for adjusting the brightness of all pixels in the vehicle under area UA. The compensation unit 115 compensates the brightness of the three-dimensional vehicle under image IMGU received from the vehicle under image generation unit 113 in accordance with the compensation coefficient, and outputs the brightness-adjusted vehicle under image IMGU to the synthesis unit 116. The synthesis unit 116 combines the brightness-adjusted vehicle under image IMGU received from the compensation unit 115 with the overhead image to generate an overhead image including the vehicle under image IMGU. The display device 130 displays the overhead image including the vehicle under image IMGU received from the compensation unit 115 via the output unit 117.

[0104] 4, 6, and 9 are also applicable to this embodiment. Two or more sampling points may be set between sampling points Pa0 and Pa2, and two or more sampling points may be set between sampling points Pb0 and Pb2.

[0105] As described above, this embodiment also achieves the same effects as the above-described embodiment. Furthermore, in this embodiment, the coefficient calculation unit 114 calculates compensation coefficients between four sets of sampling points, namely, Pa0, Pa1, Pa1, Pa2, Pb0, Pb1, and Pb1, Pb2, based on the four sets of sampling points. This improves the accuracy of the calculation of compensation coefficients compared to when compensation coefficients are calculated based on the two sets of sampling points. As a result, the image processing device 110 can generate an under-vehicle image IMGU that is more natural than the overhead-view image generated in real time. In other words, it is possible to further reduce the likelihood of the image appearing unnatural when the overhead-view image and the under-vehicle image IMGU are combined.

[0106] Although the present invention has been described above based on the embodiments, the present invention is not limited to the requirements shown in the above embodiments. These requirements can be changed without departing from the spirit of the present invention, and can be appropriately determined depending on the application form. [Explanation of symbols]

[0107] 11 CPU 12 ROM 13 RAM 14 Auxiliary storage 15 Connecting Devices 16 Drive device 100 Image Processing System 110 Image processing device 111 Image data acquisition unit 112 Projection section 113 Under-vehicle image generation unit 114 Coefficient calculation section 115 Compensation section 116 Synthesis Section 117 Output section 118 Projection information storage unit 119 Image storage unit 120 Information processing equipment 130 Display device 140 Recording Media 200 Mobile 210A, 210B, 210C, 210D Imaging device 220A, 220B, 220C, 220D area 230 Projection plane BUS IMG (IMGa, IMGb, IMGc, IMGd) image data IMGU Under-vehicle image Pa0, Pa1, Pa2 sampling points Pb0, Pb1, Pb2 sampling points UA Under-vehicle area

Claims

1. an image storage unit that stores images acquired from a plurality of imaging devices provided in the mobile object as stored images; a superimposed image generating unit that generates a superimposed image of an area that is not included in the imaging ranges of the plurality of imaging devices and that overlaps with the moving object from stored images stored in the image storage unit; a synthesis unit that synthesizes the images acquired from the plurality of imaging devices to generate a first overhead image showing the surroundings of the moving object, and synthesizes the first overhead image with the superimposed image to generate a second overhead image; a coefficient calculation unit that calculates a reference compensation coefficient at each vertex of the rectangular boundary portion between the first overhead image and the superimposed image based on image quality information of the first overhead image and image quality information of the stored image used to generate the superimposed image, and calculates a compensation coefficient for each pixel of the superimposed image using the calculated reference compensation coefficient; a compensation unit that compensates for the image quality of the superimposed image synthesized by the synthesis unit based on the compensation coefficient calculated by the coefficient calculation unit; An image processing device having:

2. an image storage unit that stores images acquired from a plurality of imaging devices provided in the mobile object as stored images; a superimposed image generating unit that generates a superimposed image of an area that is not included in the imaging ranges of the plurality of imaging devices and that overlaps with the moving object from stored images stored in the image storage unit; a synthesis unit that synthesizes the images acquired from the plurality of imaging devices to generate a first overhead image showing the surroundings of the moving object, and synthesizes the first overhead image with the superimposed image to generate a second overhead image; a coefficient calculation unit that calculates, at a boundary between the first overhead image and the superimposed image, reference compensation coefficients at each of a vertex position along a traveling direction of the moving object and at least one position set between the vertices along the traveling direction based on image quality information of the first overhead image and image quality information of a stored image used to generate the superimposed image, and calculates compensation coefficients for pixels between the two adjacent positions along the traveling direction using the reference compensation coefficients calculated at the two adjacent positions; a compensation unit that compensates for the image quality of the superimposed image synthesized by the synthesis unit based on the compensation coefficient calculated by the coefficient calculation unit; An image processing device having:

3. The coefficient calculation unit calculates a compensation coefficient for a pixel between two positions in the superimposed image by linear interpolation using the two reference compensation coefficients calculated at the two positions along the traveling direction of the moving object.

3. The image processing device according to claim 1.

4. The coefficient calculation unit further calculates compensation coefficients for pixels along the orthogonal direction in the superimposed image by linear interpolation using the compensation coefficients calculated at positions along the orthogonal direction to the traveling direction of the moving object. The image processing device according to claim 3 .

5. The coefficient calculation unit sets the reference compensation coefficient at a vertex located ahead in the moving direction of the moving object to "1.0." 5. The image processing device according to claim 1.

6. the coefficient calculation unit does not calculate the adjustment coefficient when an image of any of the vertices of the rectangular boundary portion is not included in the superimposed image generated by the superimposed image generation unit; When the coefficient calculation unit does not calculate the compensation coefficient, the compensation unit does not compensate the image quality of the superimposed image to be combined by the combination unit.

6. The image processing device according to claim 1.

7. The coefficient calculation unit calculates an average value of image quality information of a plurality of pixels in the boundary portion, and calculates a compensation coefficient based on the calculated average value.

7. The image processing device according to claim 1.

8. The synthesizing unit synthesizes the first overhead image obtained by synthesizing images acquired from the plurality of imaging devices in real time with the superimposed image generated from past stored images stored in the image storage unit to generate the second overhead image.

8. The image processing device according to claim 1.

9. a storage process for storing images acquired from a plurality of imaging devices provided in the moving body as stored images; a superimposed image generation process for generating a superimposed image of an area that is not included in the imaging range of the plurality of imaging devices and that overlaps with the moving object from the accumulated images; a synthesis process of synthesizing the images acquired from the plurality of imaging devices to generate a first overhead image showing the surroundings of the moving object, and synthesizing the first overhead image with the superimposed image to generate a second overhead image; a coefficient calculation process for calculating, at a boundary between the first overhead image and the superimposed image, a reference compensation coefficient at each of a vertex position along the traveling direction of the moving object and at least one position set between the vertices along the traveling direction, based on image quality information of the first overhead image and image quality information of a stored image used to generate the superimposed image, and calculating a compensation coefficient for a pixel between the two adjacent positions using the reference compensation coefficients calculated at two adjacent positions along the traveling direction; a correction process for correcting the image quality of the superimposed image to be synthesized based on the calculated correction coefficient; An image processing method that performs the above.

10. a storage process for storing images acquired from a plurality of imaging devices provided in the moving body as stored images; a superimposed image generation process for generating a superimposed image of an area that is not included in the imaging range of the plurality of imaging devices and that overlaps with the moving object from the accumulated images; a synthesis process of synthesizing the images acquired from the plurality of imaging devices to generate a first overhead image showing the surroundings of the moving object, and synthesizing the first overhead image with the superimposed image to generate a second overhead image; a coefficient calculation process for calculating, at a boundary between the first overhead image and the superimposed image, a reference compensation coefficient at each of a vertex position along the traveling direction of the moving object and at least one position set between the vertices along the traveling direction, based on image quality information of the first overhead image and image quality information of a stored image used to generate the superimposed image, and calculating a compensation coefficient for a pixel between the two adjacent positions using the reference compensation coefficients calculated at two adjacent positions along the traveling direction; a correction process for correcting the image quality of the superimposed image to be synthesized based on the calculated correction coefficient; An image processing program that causes a computer to perform the above.

Images