Synthetic image generation system and posterior image display system

Abstract

To provide "a synthetic image generation system and a posterior image display system" which correct images photographed with a plurality of cameras whose photographing regions partially overlap, with suppressed image failure due to the correction., and synthesize the images.SOLUTION: The synthetic image generation system applies prescribed image conversion which brings the ratio of the pixel values of a first image and a second image closer to 1, to the images. The system extracts from the image-converted first image and the image-converted second image an overlapping portion of photographic regions, as comparison images. The system calculates as a correction gain the ratio of the average of the gradation values of the comparison image extracted from the image-converted first image to the average of the gradation values of the comparison image extracted from the image-converted second image, and corrects the gradation value of the image-converted second image with the correction gain, and applies inverse conversion of the prescribed image conversion to the corrected second image to synthesize with the first image.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a technique for synthesizing images captured by a plurality of cameras whose imaging regions partially overlap.

Background Art

[0002] As a technique for synthesizing images captured by a plurality of cameras whose imaging regions partially overlap, a plurality of cameras for imaging the periphery of a vehicle are arranged so that the imaging regions partially overlap, and the overlapping portion of the imaging regions of each image captured by each camera is used as a comparison image. The average value of the pixel values of the comparison image of each image is obtained, and the ratio of the average value of the comparison image of one image (hereinafter, "first image") to the average value of the comparison image of the other image (hereinafter, "second image") is calculated as a correction gain. A technique is known in which a second image subjected to gain correction by the calculated correction gain is synthesized with the first image and displayed, thereby suppressing the occurrence of discontinuity of each image in the synthesized image due to differences in color tone and brightness of each image in the synthesized image (for example, Patent Documents 1 and 2).

[0003] In addition, as a technique related to the present invention, there is also known a technique in which an image captured by a camera arranged at the rear of an automobile is synthesized by performing viewpoint conversion on images captured rearward by left and right cameras arranged on the left and right sides of the automobile and displayed on a monitor provided in the vehicle (for example, Patent Document 3).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to the technique of synthesizing a second image, in which gain correction is performed using, as a correction gain, the ratio of the average value of the comparison image of the first image described above to the average value of the comparison image of the second image, the correction gain may become extremely large when the comparison image is an image with small pixel values.

[0006] And in such a case, when the second image is corrected with the calculated correction gain, a portion where the pixel value outside the comparison image of the second image is large may become a portion where the pixel value is saturated and the color tone is abnormal after correction, and the corrected second image may break down. Therefore, an object of the present invention is to suppress the occurrence of discontinuity of each image in a synthesized image while suppressing the breakdown of an image due to correction in a synthesized image generation system that corrects and synthesizes images captured by a plurality of cameras whose imaging regions partially overlap.

Means for Solving the Problem

[0007] To achieve the above object, the present invention provides a composite image generation system that generates a composite image by combining a first image captured by a first camera and a second image captured by a second camera whose imaging area partially overlaps with that of the first camera, and connecting at least a part of the first image and at least a part of the second image. The system includes: a first image conversion means for performing a predetermined image conversion on the first image to generate a converted first image; a second image conversion means for performing the predetermined image conversion on the second image to generate a converted second image; a comparison image extraction means for extracting, as a first comparison image, a part of the converted first image in which the range where the imaging areas partially overlap is reflected, or a part of the part, and, as a second comparison image, a part of the converted second image in which the range that is the same as the range where the first comparison image is reflected is reflected; a correction gain calculation means for calculating, as a correction gain, the ratio of the representative value of the pixel values of the first comparison image extracted by the comparison image extraction means to the representative value of the pixel values of the second comparison image; a correction means for correcting the converted second image with the calculated correction gain to generate a corrected second image; a second image inverse conversion means for performing an inverse conversion of the predetermined image conversion on the corrected second image to generate an inversely converted second image; and a composite means for generating the composite image by combining the inversely converted second image with the first image or an image obtained by performing an inverse conversion of the predetermined image conversion on the converted first image. However, the predetermined image conversion is an image conversion for converting pixel values such that the ratio of the pixel value of the converted first image corresponding to the first pixel value of the first image to the pixel value of the converted second image corresponding to the second pixel value of the second image approaches 1 more than the ratio of the first pixel value to the second pixel value.

[0008] Here, in such a composite image generation system, the predetermined image conversion may be an image conversion for converting the range of pixel values of an image such that at least the value that is the lower limit of the range of pixel values of the image becomes a larger value than before conversion. The predetermined image conversion is an image conversion for converting the range of pixel values of an image such that at least the value that is the lower limit of the range of pixel values of the image becomes a larger value than before conversion. Here, in such a synthetic image generation system, the predetermined image conversion may be an image conversion in which a value obtained by adding m (where m > 0) to a value obtained by setting the pixel value of the image to 1 / n (where n > 1) is used as the pixel value of the converted image. Alternatively, in such a synthetic image generation system, the predetermined image conversion may be an image conversion in which a value obtained by adding m (where m > 0) to the pixel value of the image is used as the pixel value of the converted image. Alternatively, in such a synthetic image generation system, the predetermined image conversion is such that when m (where m > 0) is added to the pixel value of the first image and the pixel value of the second image, and a value exceeding the maximum value that the pixel value can take does not occur, the value obtained by adding the m is used as the pixel value of the converted image, and when a value exceeding the maximum value that the pixel value can take occurs, an image conversion is performed in which a value obtained by adding m to a value obtained by setting the pixel value of the image to 1 / n (where n > 1) is used as the pixel value of the converted image.

[0009] Further, in the above synthetic image generation system, the pixel value may be the gradation value of each color component of the RGB of the image. In this case, in the correction gain calculation means, as the correction gain, for each color component of the RGB, the ratio of the representative value of the gradation value of the pixel of the first comparison image to the representative value of the gradation value of the pixel of the second comparison image is calculated as the gain correction characteristic, and the correction means may correct the gradation value of each pixel of the second image after conversion with the set correction gain for each color component of the RGB.

[0010] Alternatively, in the above synthetic image generation system, the pixel value may be the luminance value of the image. In this case, in the correction gain calculation means, as the correction gain, the ratio of the representative value of the luminance value of the pixel of the first comparison image to the representative value of the luminance value of the pixel of the second comparison image is calculated as the gain correction characteristic, and in the correction means, the luminance value of each pixel of the second image after conversion is corrected with the set correction gain.

[0011] In addition, in such a composite image generation system in which the pixel values are the gradation values of the respective color components of RGB of the image, the gradation number of each color component of RGB of the first image and the second image is set to 256, and the predetermined image conversion may be an image conversion in which, for each color component of RGB, the gradation value of the converted image is a value obtained by adding 128 to a value obtained by dividing the gradation value of the image by 2.

[0012] In addition, in such a composite image generation system in which the pixel values are the gradation values of the respective color components of RGB of the image, the gradation number of each color component of RGB of the first image and the second image is set to 256, and the predetermined image conversion may be an image conversion in which, for each color component of RGB, the gradation value obtained by adding 128 to the gradation value of the image before conversion is used as the gradation value on the lower side of the converted image with an expanded gradation number.

[0013] In addition, in the above composite image generation system, the representative value may be an average value. The present invention also provides a rear image display system mounted on an automobile, which includes the above composite image generation system. The rear image display system includes a monitor for displaying the composite image generated by the composite image generation system, and as the first camera, a camera for photographing the rear of the automobile from the side of the automobile, and as the second camera, a camera for photographing the rear of the automobile from the rear of the automobile.

[0014] According to the above composite image generation system and the rear image display system, when extracting the first comparison image and the second comparison image from the first image and the second image, calculating the ratio of the representative values of the pixel values of both comparison images as the correction gain, and synthesizing the corrected second image as the second image to be synthesized into the composite image, compared with the case of synthesizing the second image after inverse conversion, which is the second image to be synthesized into the composite image, into the composite image, the increase amount of the pixel value with respect to the original second image can be suppressed to be small, and the occurrence of saturation of the pixel value in the second image to be synthesized and the appearance of a portion where the color tone of the synthesized image becomes abnormal can be suppressed by relatively simple calculation processing.

Advantages of the Invention

[0015] As described above, according to the present invention, in a composite image generation system that corrects and synthesizes images captured by a plurality of cameras whose shooting areas partially overlap, it is possible to suppress the occurrence of discontinuity in each image in the synthesized image and suppress the breakdown of the image due to correction.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Modes for Carrying Out the Invention

[0017] Hereinafter, embodiments of the present invention will be described. FIG. 1 shows the configuration of the image display system according to the present embodiment. The image display system is a system mounted on an automobile, and as shown in the figure, it includes a left side camera 1, a rear camera 2, a right side camera 3, an image processing device 4, a left side monitor 5, and a right side monitor 6. In the configuration of such an image display system, as shown in FIGS. 2a1 and 2a2, the rear camera 2 is arranged at the rear of the vehicle and captures images in the rear direction of the vehicle from that position. Also, the left side camera 1 is arranged on the left side of the vehicle and captures images in the rear direction of the vehicle from that position, and the right side camera 3 is arranged on the right side of the vehicle and captures images in the rear direction of the vehicle from that position. Here, the rear camera 2, the left side camera 1, and the right side camera 3 output images with pixel values being the values of the respective color components of R, G, and B.

[0018] Also, the shooting areas of the rear camera 2 and the left side camera 1 partially overlap at the rear of the vehicle, and the shooting areas of the rear camera 2 and the right side camera 3 partially overlap at the rear of the vehicle. Also, the left side monitor 5 and the right side monitor 6 are displays for displaying images. For example, as shown in FIG. 2b, the left side monitor 5 is arranged to be located on the left side of the center in the left - right direction of the driver's seat, and the right side monitor 6 is arranged to be located on the right side of the center in the left - right direction of the driver's seat.

[0019] Returning to FIG. 1, the image processing device 4 includes a left image processing unit 41 and a right image processing unit 42. The left image processing unit 41 synthesizes the image captured by the left side camera 1 and the image captured by the rear camera 2 and displays it on the left side monitor 5. The right image processing unit 42 synthesizes the image captured by the right side camera 3 and the image captured by the rear camera 2 and displays it on the right side monitor 6.

[0020] Hereinafter, the left image processing unit 41 and the right image processing unit 42 will be described. The left image processing unit 41 and the right image processing unit 42 have the same configuration and perform the same operations. Therefore, the left image processing unit 41 will be described as a representative. Note that the description of the right image processing unit 42 will be the same as that of the left image processing unit 41 below with the left and right interchanged. FIG. 3 shows the configuration of the left image processing unit 41. As shown in the figure, the left image processing unit 41 includes a first image conversion unit 411, a second image conversion unit 412, a first comparison image extraction unit 413, a second comparison image extraction unit 414, a correction gain calculation unit 415, a correction unit 416, a second image inverse conversion unit 417, and a synthesis unit 418. Then, an image captured by the left side camera 1 is input to the left image processing unit 41 as the first image, and an image captured by the back camera 2 is input as the second image. However, it can also be configured such that an image captured by the back camera 2 is input as the first image, and an image captured by the left side camera 1 is input as the second image.

[0021] The first image conversion unit 411 performs a predetermined image conversion on the first image to generate a converted first image, and the second image conversion unit 412 performs a predetermined image conversion on the second image to generate a converted second image. The image conversion performed by the first image conversion unit 411 and the image conversion performed by the second image conversion unit 412 are the same image conversion, and the ratio of the pixel value of the converted first image to the pixel value of the converted second image is closer to 1 than the ratio of the pixel value of the first image to the pixel value of the second image. However, when the ratio of the pixel value of the first image to the pixel value of the second image is 1, the ratio of the pixel value of the converted first image to the pixel value of the converted second image remains 1.

[0022] For example, when the first image and the second image are 8-bit RGB images, represented in hexadecimal notation where "00" represents the first gradation and "FF" represents the 256th gradation, and are 256-gradation images from the first gradation to the 256th gradation, as the image conversion, as shown in FIGS. 4a1 to 4a2, the gradation value of R of each pixel of the image before conversion is set to 1 / 2, and then 128 is added, and the resulting value is set as the gradation value of R of each pixel of the image after conversion. The gradation value of G of each pixel of the image before conversion is set to 1 / 2, and then 128 is added, and the resulting gradation value is set as the gradation value of G of each pixel of the image after conversion. The gradation value of B of each pixel of the image before conversion is set to 1 / 2, and then 128 is added, and the resulting value is set as the gradation value of B of each pixel of the image after conversion. An image conversion can be used.

[0023] Alternatively, as shown in FIGS. 4b1 to 4b2, the converted image can be an image from the first gradation to the 1024th gradation, with each of RGB being 10 bits and in hexadecimal notation where "00" represents the first gradation and "3FF" represents the 1024th gradation. The gradation value obtained by adding 128 to the R gradation value of each pixel in the pre-conversion image is used as the R gradation value of each pixel in the post-conversion image. The gradation value obtained by adding 128 to the G value of each pixel in the pre-conversion image is used as the G value of each pixel in the post-conversion image. The gradation value obtained by adding 128 to the B gradation value of each pixel in the pre-conversion image is used as the B gradation value of each pixel in the post-conversion image. Image conversion or the like can also be used.

[0024] Returning to FIG. 3, in the first comparison image extraction unit 413, an area in the converted first image corresponding to a part of the overlapping portion of the shooting area of the left side camera 1 and the shooting area of the back camera 2 is preset as the first comparison area. The first comparison image extraction unit 413 extracts the image within the first comparison area of the converted first image as the first comparison image.

[0025] Also, in the second comparison image extraction unit 414, an area in the converted second image corresponding to the first comparison area of the converted first image, that is, an area in the converted second image where the same subject as in the first comparison area of the converted first image is captured, is preset as the second comparison area. The second comparison image extraction unit 414 extracts the image within the second comparison area of the converted second image as the second comparison image.

[0026] The correction gain calculation unit 415 calculates, for each of the R, G, and B color components, the ratio of the representative value of the gradation values of the first comparison image to the representative value of the gradation values of the second comparison image as the correction gain. Here, in this embodiment, the average value is used as this representative value. However, depending on the use and purpose of the image display system, other representative values such as the median value and the mode value can also be used as the representative value. Next, the correction unit 416 corrects the gain of each color component of the converted second image with the correction gain set by the correction gain setting unit for each of the R, G, and B color components, and outputs it to the second image inverse conversion unit 417. The second image inverse conversion unit 417 performs an inverse image conversion, which is the inverse of the image conversion applied by the second image conversion unit 412 to the second image, on the converted second image corrected by the correction unit 416, and outputs it to the composition unit 418 as the inverse-converted second image. That is, when the second image conversion unit 412 performs an image conversion that converts the gradation values of R, G, and B of each pixel as shown in FIGS. 4a1-4a2, as shown in FIGS. 4a2-4a3, a value obtained by doubling the value obtained by subtracting 128 from the gradation value of R of each pixel of the image before conversion is used as the gradation value of R of each pixel of the image after conversion, a value obtained by doubling the value obtained by subtracting 128 from the gradation value of G of each pixel of the image before conversion is used as the gradation value of G of each pixel of the image after conversion, and a value obtained by doubling the value obtained by subtracting 128 from the gradation value of B of each pixel of the image before conversion is used as the gradation value of B of each pixel of the image after conversion. The second image inverse conversion unit 417 performs an image conversion.

[0027] Also, when the second image conversion unit 412 performs an image conversion that converts the gradation values of R, G, and B of each pixel as shown in FIGS. 4b1-4b2, as shown in FIGS. 4b2-4b3, the gradation value from 1 to 256 represented by the lower 8 bits of the value obtained by subtracting 128 from the gradation value of R of each pixel of the image before conversion is used as the gradation value of R of each pixel of the image after conversion, the gradation value from 1 to 256 represented by the lower 8 bits of the value obtained by subtracting 128 from the value of G of each pixel of the image before conversion is used as the gradation value of G of each pixel of the image after conversion, and the gradation value from 1 to 256 represented by the lower 8 bits of the value obtained by subtracting 128 from the value of B of each pixel of the image before conversion is used as the gradation value of B of each pixel of the image after conversion. The second image inverse conversion unit 417 performs an image conversion.

[0028] Now, returning to FIG. 3, the composition unit 418 performs a viewpoint conversion on the inverse-converted second image inverse-converted by the second image inverse conversion unit 417 into an image representing the view when observing the rear from the viewpoint of the left side camera 1, and composes the embedded image, which is the part used for composition with the first image in the viewpoint-converted inverse-converted second image, into the first image so as to form a continuous single image representing the view of the rear of the automobile, and outputs and displays it on the left side monitor 5.

[0029] Here, a processing example of such a left image processing unit 41 is shown below. Now, assume that the first image is captured by the left side camera 1 as shown in FIG. 5a1, and the second image is captured by the back camera 2 as shown in FIG. 5b1. In this case, the first image conversion unit 411 generates a converted first image as shown in FIG. 5a2, for example, from the first image in FIG. 5a1, and a converted second image as shown in FIG. 5b2, for example, from the second image in FIG. 5b1. Then, the first comparison image extraction unit 413 extracts, as a first comparison image as shown in FIG. 5a3, an image portion of a first comparison region C1 that is set as a region corresponding to a part of the overlapping region between the shooting region of the left side camera 1 and the shooting region of the back camera 2 from the converted first image in FIG. 5a2. The second comparison image extraction unit 414 extracts, as a second comparison image as shown in FIG. 5b3, an image portion of a second comparison region C2 that is set as a region corresponding to the first comparison region C1 from the converted second image in FIG. 5b2.

[0030] Then, the correction gain calculation unit 415 calculates, for each of the R, G, and B color components, from the first comparison image in FIG. 5a3 and the second comparison image in FIG. 5b3, the ratio of the average value of the gradation values of the first comparison image to the average value of the gradation values of the second comparison image as the correction gain. Then, the gain of the converted second image in FIG. 5b2 is adjusted according to the set correction gain, and a corrected converted second image as shown in FIG. 6a1 is generated. Further, the second image inverse conversion unit 417 performs image conversion on the corrected converted second image shown in FIG. 6a1, and generates an inversely converted second image as shown in FIG. 6a2.

[0031] Here, the matching of the average value of the gradation values of the first comparison image and the average value of the gradation values of the second comparison image by correction with the correction gain is inherited between the generated second image after inverse transformation and the first image. The average values of the respective R, G, and B color components of the region corresponding to the second comparison region C2 of the second image after inverse transformation match the average values of the respective R, G, and B color components of the region corresponding to the first comparison region C1 of the second image. Therefore, the occurrence of discontinuity between the second image after inverse transformation and the first image in the synthesized image, such as the manifestation of the boundary between the second image after inverse transformation and the first image in the synthesized image, is suppressed.

[0032] Then, the second image after inverse transformation shown in FIG. 6a2 is subjected to a viewpoint transformation by the synthesizing unit 418 and synthesized with the first image in FIG. 6b so as to form a continuous single image representing the rear view of the automobile as shown in FIG. 6c, and is displayed on the left side monitor 5. Here, this synthesis is performed, for example, by extracting, as the incorporation image R, a portion where the state of the shielding region, which is a region behind the own vehicle shielded from the left side camera 1 by the own vehicle in the second image after inverse transformation, appears, and overwriting and replacing, with the incorporation image R, the region in the first image where the state of the shielding region would appear if the own vehicle were transparent.

[0033] According to such a left image processing unit 41, it is possible to suppress the calculation of an extremely large value as the correction gain as compared with the case of directly calculating the ratio of the average value of the gradation values of the first comparison region with the first image and the second comparison region with the second image as the correction gain. For example, when the average values of the gradation values of the respective color components R, G, and B of the first comparison region of the first image and the second comparison region of the second image are as shown in FIG. 7a, when calculating the ratio of the average value of the gradation values of the first comparison region with the first image and the second comparison region with the second image as the correction gain, a correction gain of approximately 39.08, which is the ratio of the average value of the gradation value of B in the first comparison region of the first image, 18.76, to the average value of the gradation value of B in the second comparison region of the second image, 0.48, is calculated. Therefore, the gradation value of B of the second image synthesized into the synthesized image increases by approximately 39.08×a when the original gradation value is a.

[0034] On the one hand, when image conversion for converting the R, G, and B values of each pixel as shown in FIGS. 4a1-4a2 is performed on the first image and the second image, the average value of the gradation values in the first comparison region of the first image after conversion is 137.38, and the average value of the gradation values in the second comparison region of the second image after conversion is 128.24. An approximate ratio of 1.07 between the two is calculated as the correction gain. Then, the B gradation value of the second image after inverse conversion to be synthesized into the composite image is 2×[{G×(a / 2 + 128)} - 128] when the original gradation value is a and the calculated correction gain is G. With respect to the original gradation value a, 2×[{G×(a / 2 + 128)} - 128] - a = (G - 1)a + {(G - 1)×256} increases. Therefore, in the case of FIG. 7b, the increase in the B gradation value of the second image after inverse conversion with respect to the original gradation value can be suppressed to be as small as (0.07×a) + 17.92.

[0035] Also, when image conversion for converting the R, G, and B values of each pixel as shown in FIGS. 4b1-4b2 is performed on the first image and the second image, the average value of the gradation values in the first comparison region of the first image after conversion is 146.76, and the average value of the gradation values in the second comparison region of the second image after conversion is 128.48. An approximate ratio of 1.14 between the two is calculated as the correction gain. Then, the B gradation value of the second image after inverse conversion to be synthesized into the composite image is {G×(a + 128)} - 128} when the original gradation value is a and the calculated correction gain is G. With respect to the original gradation value a, {G×(a + 128)} - 128} - a = (G - 1)a + {(G - 1)×128} increases. Therefore, in the case of FIG. 7c, the increase in the B gradation value of the second image after inverse conversion with respect to the original gradation value can be suppressed to be as small as (0.14×a) + 5.12.

[0036] Therefore, according to the present embodiment, when the ratio of the average values of the gradation values in the first comparison region with the first image and the second comparison region with the second image is calculated as the correction gain, as shown in FIG. 8a, when the correction gain is too large and the gradation value of the second image after correction saturates due to the correction, resulting in an abnormal color portion NG in the synthesized image, this can be suppressed in an approximate shooting scene. As shown in FIG. 8b, a seamless composite image can be generated and displayed.

[0037] In addition, the addition or subtraction of 128, or the multiplication by 1 / 2 or 2 of the gradation values in the image conversion shown in FIGS. 4b1 - 4b2 - b3 and FIGS. 4c1 - 4c2 - c3 can also be realized by simple operations such as bit shifting of the data representing the gradation values. Therefore, when using these image conversions, generation and display of a seamless composite image can be realized by simple processing.

[0038] The embodiments of the present invention have been described above. Here, in the above embodiments, when the first image and the second image are 256 - gradation images from the first gradation to the 256th gradation, even if 128 is added to the maximum value of the gradation values of the pixels of the first image and the maximum value of the gradation values of the pixels of the second image, an image conversion may be used in which the maximum value of the values that do not exceed 256, or the maximum value of the powers of 2 less than or equal to the maximum value, is used as the gradation value of the converted image with the gradation value added to the pre - conversion image as the gradation value of the post - conversion image.

[0039] Alternatively, as this image conversion, when 128 is added to the maximum value of the gradation values of the pixels of the first image and the maximum value of the gradation values of the pixels of the second image and no value exceeding 256 is generated, an image conversion may be used in which the gradation value obtained by adding 128 to the gradation value of the pre - conversion image is used as the gradation value of the post - conversion image. When a value exceeding 256 is generated, an image conversion may be used in which the gradation value of the pre - conversion image is halved and then 128 is added, and the resulting value is used as the gradation value of the post - conversion image.

[0040] In addition, in the above embodiments, the first image and the second image have been described as images having pixel values of the values of each color component of R, G, and B. However, as the first image and the second image, an image having a luminance value and two color difference values as pixel values, such as Y - Cb - Cr, may be used. Also, in this case, the above - described correction process of the second image may be performed only on the luminance value Y component. That is, for example, an image consisting only of the luminance value Y component is generated from the first image and the second image, and the generated image is used as the first image and the second image, and the image conversion of the first image conversion unit 411 and the second image conversion unit 412 described above, the calculation of the correction gain by the first comparison image extraction unit 413, the second comparison image extraction unit 414, and the correction gain calculation unit 415, the correction by the correction unit 416, and the image conversion of the second image inverse conversion unit 417 are performed. Also, an image generated by replacing the luminance value Y of the second image with the inverse-converted image output from the second image inverse conversion unit 417 may be used as the image to be combined with the first image in the combining unit 418. However, the image conversion and the like shown in FIGS. 4b1-4b2-b3 and FIGS. 4c1-4c2-c3 may be corrected and applied according to the number of bits of the luminance value Y.

[0041] Also, in the above, in the combining unit 418, the inverse-converted second image and the first image are combined. However, this may be achieved by providing a first image inverse conversion unit that performs an image conversion that is the inverse conversion of the image conversion of the first image conversion unit 411 on the converted first image to generate an inverse-converted first image, and combining the inverse-converted second image and the inverse-converted first image in the combining unit 418.

[0042] Also, in the above, the case where an image representing the rear view is generated by combining the images captured by the left side camera 1 and the right side camera 3 provided on the side of the automobile and the image captured by the back camera 2 provided at the rear of the automobile has been shown. However, this embodiment is applicable to any combination as long as it combines images captured by a plurality of cameras with partially overlapping shooting areas that capture the surroundings of the automobile, and performs viewpoint conversion on the images to represent the view of observing the surroundings of the automobile from above, such as generating an overhead (bird's-eye view) image, including cases where the cameras are not provided on the automobile.

Explanation of Reference Numerals

[0043] 1… Left side camera, 2… Rear camera, 3… Right side camera, 4… Image processing device, 5… Left side monitor, 6… Right side monitor, 41… Left image processing unit, 42… Right image processing unit, 411… First image conversion unit, 412… Second image conversion unit, 413… First comparison image extraction unit, 414… Second comparison image extraction unit, 415… Correction gain calculation unit, 416… Correction unit, 417… Second image inverse conversion unit, 418… Composition unit.

Claims

1. A composite image generation system that generates a composite image by combining a first image captured by a first camera and a second image captured by a second camera whose imaging area partially overlaps with that of the first camera, and joining at least a part of the first image and at least a part of the second image, comprising: first image conversion means for performing a predetermined image conversion on the first image to generate a converted first image; second image conversion means for performing the predetermined image conversion on the second image to generate a converted second image; comparative image extraction means for extracting, as a first comparative image, a part of the converted first image or a part of the part where the range where the imaging areas partially overlap is reflected, and, as a second comparative image, a part of the converted second image where the range that is the same as the range where the first comparative image is reflected is reflected; correction gain calculation means for calculating, as a correction gain, the ratio of the representative value of the pixel values of the first comparative image extracted by the comparative image extraction means to the representative value of the pixel values of the second comparative image; correction means for correcting the converted second image with the calculated correction gain to generate a corrected second image; second image inverse conversion means for performing an inverse conversion of the predetermined image conversion on the corrected second image to generate an inversely converted second image; composition means for generating the composite image by composing the inversely converted second image with the first image or an image obtained by performing an inverse conversion of the predetermined image conversion on the converted first image; and the predetermined image conversion is an image conversion for converting pixel values such that the ratio of the pixel value of the converted first image corresponding to the first pixel value of the first image to the pixel value of the converted second image corresponding to the second pixel value of the second image is closer to 1 than the ratio of the first pixel value to the second pixel value. The composite image generation system is characterized by this.

2. The composite image generation system according to claim 1, wherein the predetermined image conversion is an image conversion for converting the range of pixel values of the image such that at least the value that is the lower limit of the range of pixel values of the image becomes a larger value than before conversion. The composite image generation system is characterized by this.

3. The composite image generation system according to claim 1, wherein the predetermined image conversion is an image conversion in which the pixel value of the image is set to a value obtained by adding m (where m > 0) to a value obtained by dividing the pixel value by 1 / n (where n > 1), and this value is used as the pixel value of the converted image. The composite image generation system is characterized by this.

4. The composite image generation system according to claim 1, wherein The synthetic image generation system is characterized in that the predetermined image conversion is an image conversion in which a value obtained by adding m (where m > 0) to the pixel value of the image is used as the pixel value of the converted image.

5. The synthetic image generation system according to claim 1, wherein, in the predetermined image conversion, when adding m (where m > 0) to the pixel value of the first image and the pixel value of the second image does not result in a value exceeding the maximum value that the pixel value can take, the value obtained by adding m is used as the pixel value of the converted image; and when a value exceeding the maximum value that the pixel value can take occurs, the value obtained by adding m to a value obtained by setting the pixel value of the image to 1 / n (where n > 1) is used as the pixel value of the converted image. The synthetic image generation system is characterized by this image conversion.

6. The synthetic image generation system according to claim 1, 2, 3, 4, or 5, wherein the pixel value is the gradation value of each color component of RGB of the image, and the correction gain calculation means calculates, as the correction gain, for each color component of RGB, the ratio of the representative value of the gradation value of the pixel of the first comparison image to the representative value of the gradation value of the pixel of the second comparison image, as the gain correction characteristic, and the correction means corrects the gradation value of each pixel of the second image after conversion with a set correction gain for each color component of RGB. The synthetic image generation system is characterized by this.

7. The synthetic image generation system according to claim 1, 2, 3, 4, or 5, wherein the pixel value is the luminance value of the image, and the correction gain calculation means calculates, as the correction gain, the ratio of the representative value of the luminance value of the pixel of the first comparison image to the representative value of the luminance value of the pixel of the second comparison image, as the gain correction characteristic, and the correction means corrects the luminance value of each pixel of the second image after conversion with a set correction gain. The synthetic image generation system is characterized by this.

8. The synthetic image generation system according to claim 1, wherein the pixel value is the gradation value of each color component of RGB of the image, and the correction gain calculation means calculates, as the correction gain, for each color component of RGB, the ratio of the representative value of the gradation value of the pixel of the first comparison image to the representative value of the gradation value of the pixel of the second comparison image, as the gain correction characteristic, the correction means corrects the gradation value of each pixel of the second image after conversion with a set correction gain for each color component of RGB, and the number of gradations of each color component of RGB of the first image and the second image is 256. The synthetic image generation system is characterized in that the predetermined image conversion is an image conversion in which, for each color component of the RGB, the tone value of the image after conversion is a value obtained by adding 128 to a value obtained by dividing the tone value of the image by 1 / 2.

9. The synthetic image generation system according to claim 1, wherein the pixel value is the tone value of each color component of the RGB of the image, and the correction gain calculation means calculates, as the correction gain, for each color component of the RGB, the ratio of the representative value of the tone value of the pixel of the first comparison image to the representative value of the tone value of the pixel of the second comparison image as the gain correction characteristic, and the correction means corrects the tone value of each pixel of the second image after conversion with the set correction gain for each color component of the RGB. The number of tones of each color component of the RGB of the first image and the second image is 256. The synthetic image generation system is characterized in that the predetermined image conversion is an image conversion in which, for each color component of the RGB, the tone value obtained by adding 128 to the tone value of the image before conversion is used as the tone value on the lower side of the image after conversion with an expanded number of tones.

10. The synthetic image generation system according to claim 1, 2, 3, 4, 5, 8 or 9, wherein the representative value is an average value.

11. A rear image display system mounted on an automobile, comprising the synthetic image generation system according to claim 1, 2, 3, 4, 5, 8 or 9, comprising a monitor for displaying the synthetic image generated by the synthetic image generation system, wherein the first camera is a camera for photographing the rear of the automobile from the side of the automobile, and the second camera is a camera for photographing the rear of the automobile from the rear of the automobile.

Images