Composite image generation system and rear image display system

Abstract

To provide a "composite image generation system and rear image display system" for performing natural composition of a YCbCr image using a small amount of processing.SOLUTION: From first / second images of a YCbCr system, portions of overlapped imaging areas are extracted as first / second comparison images. A Y gain calculation section 4114 calculates a ratio of representative values of Y signals of the first comparison image and the second comparison image, as a correction gain d. An RGB gain calculation section 4118 calculates a ratio of representative values of R / G / B signals of the first comparison image and the second comparison image converted in an RGB system, as a correction gain a / b / c. A CbCr correction section 4119 corrects Cb, Cr signals of the second image so that gain correction of R, G, B components using correction gains a, b, c is reflected. A Y correction section 4120 corrects a Y signal of the second image using the correction gain d. A composition section 4121 composes the first image and the corrected second image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a technique for synthesizing images captured by a plurality of cameras whose capture areas partially overlap. [Background technology]

[0002] A known technique for combining images captured by multiple cameras with partially overlapping capture areas is to arrange multiple cameras capturing images of the area around a vehicle so that their capture areas partially overlap, use the overlapping portions of the capture areas of each image captured by each camera as a comparison image, calculate the average pixel value of each comparison image, calculate a correction gain as the ratio of the average value of the comparison image of one image (hereinafter referred to as the "first image") to the average value of the comparison image of the other image (hereinafter referred to as the "second image"), and perform gain correction on the second image using the calculated correction gain to combine the second image, which has its white balance adjusted to the first image, with the first image and display it, thereby suppressing the appearance of discontinuity at the boundaries between each image in the combined image and differences in image quality between each image in the combined image (for example, Patent Documents 1 and 2).

[0003] Furthermore, when the first and second images described above are images that use the YUV color representation method, a technique is known in which the Y, U, and V of the first and second images are corrected and combined so that the average values ​​of Y, U, and V of the comparison image of the first image and the comparison image of the second image are aligned (for example, Patent Document 3).

[0004] Another known technology related to the present invention is one in which an image of the rear view taken by a camera placed at the rear of a vehicle is combined with images of the rear view taken by cameras placed on the left and right sides of the vehicle through viewpoint conversion, and then displayed on a monitor installed inside the vehicle (for example, Patent Document 4). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 11-102430 [Patent Document 2] JP 2002-324235 A [Patent Document 3] JP 2018-206323 A [Patent Document 4] Japanese Patent Application Laid-Open No. 2009-206747 Summary of the Invention [Problem to be solved by the invention]

[0006] If the first and second images described above are images that use a luminance signal and two color difference signals to represent color, such as the YCbCr, YPbPr, or YUV format, correcting the gain of the luminance signal and two color difference signals of the second image so that the average values ​​of the luminance signal and two color difference signals of the comparison image of the first image and the comparison image of the second image are aligned may result in the hue of the second image changing as if it were rotating within color space, and a relatively large color tint may be added to the entire image.

[0007] Therefore, one possible approach is to convert the comparison image of the first image and the comparison image of the second image into RGB format images, calculate correction gains for each of the R, G, and B of the second image so that the average values ​​of R, G, and B of the comparison image of the first image and the comparison image of the second image are the same, correct the luminance signal and two color difference signals of the second image so that the gain correction using the calculated correction gains is reflected in each of the R, G, and B components, and then combine the corrected second image with the first image.

[0008] More specifically, for example, in accordance with the ITU-R BT.601 standard, conversion from the YCbCr system to the RGB system is shown in Equation 1, and conversion from the RGB system to the YCbCr system is shown in Equation 2, so

[0009]

number

[0010]

number

[0011] If the luminance signal and two color difference signals of a YCbCr image are Y0Cb0Cr0, then from equation 1, conversion of Y0Cb0Cr0 to RGB color signals R0G0B0 is expressed by equation 3.

[0012]

number

[0013] Then, assuming that the correction gain of R is a, the correction gain of G is b, and the correction gain of B is c, the gain correction of the color signal R0G0B0 is expressed by Equation 4, where R'G'B' is the color signal after gain correction.

[0014]

number

[0015] Furthermore, from Equation 2, conversion of R'G'B' after gain correction into the YCbCr format is expressed by Equation 5, where the converted luminance signal and two color difference signals are Y'Cb'Cr'.

[0016]

number

[0017] Therefore, gain corrections of R correction gain a, G correction gain b, and B correction gain c are reflected in each of the R, G, and B components, and the conversion of Y0Cb0Cr0 is the conversion of Equation 6 in which Equations 3, 4, and 5 are applied sequentially to Y0Cb0Cr0.

[0018]

number

[0019] However, the conversion shown in Equation 6 is complex and requires a large processing load. Furthermore, because the conversion shown in Equation 6 must be performed for each pixel having a luminance signal Y, it is redundant when handling images in a format such as 422 or 420 in which the number of Cb0Cr0 pixels having color difference signals is less than the number of pixels having a luminance signal Y by thinning out Cb0Cr0 pixels having color difference signals.

[0020] Therefore, the present invention aims to realize the synthesis of multiple images with partially overlapping shooting areas using a method that uses a luminance signal and two color difference signals to represent color, with a relatively small amount of processing while suppressing the occurrence of unnaturalness. [Means for solving the problem]

[0021] To achieve the above object, the present invention provides a composite image generation system that combines a first image captured by a first camera with a second image captured by a second camera whose capture area partially overlaps with that of the first camera to generate a composite image that is an image obtained by stitching together at least a portion of the first image and at least a portion of the second image, where the first image and the second image are images that are expressed in a predetermined format using a luminance signal, a first color difference signal, and a second color difference signal. The composite image generation system further includes a comparison image extraction means for extracting a portion of the first image in which a range where the shooting areas partially overlap or a portion of that portion as a first comparison image, and a portion of the second image in which the range of the first comparison image is the same as that of the first comparison image as a second comparison image; a brightness correction gain calculation means for calculating a brightness correction gain that is a ratio of a representative value of a brightness signal of the first comparison image extracted by the comparison image extraction means to a representative value of a brightness signal of the second comparison image; an RGB conversion means for converting the first comparison image and the second comparison image into images in an RGB system that represent colors with R, G, and B signals; and an R correction gain that is a ratio of a representative value of an R signal of the first comparison image converted into the RGB system to a representative value of the R signal of the second comparison image converted into the RGB system. The image processing apparatus includes an RGB correction gain calculation unit that calculates, as a G correction gain, a ratio of a representative value of a G signal of a comparison image to a representative value of the G signal of the second comparison image converted to an RGB format, and a B correction gain that calculates, as a B correction gain, a ratio of a representative value of a B signal of the first comparison image converted to an RGB format to a representative value of the B signal of the second comparison image converted to an RGB format; a correction unit that corrects a gain of a luminance signal of the second image with the luminance correction gain, and corrects a first color difference signal and a second color difference signal of the second image so that they represent the first color difference signal and the second color difference signal of the second image in which the gains of the R component, G component, and B component have been corrected with the R correction gain, the B correction gain, and the G correction gain, thereby generating a corrected second image; and a combination unit that combines the corrected second image with the first image to generate the combined image.

[0022] Here, the composite image generation system may be configured such that the correction means includes a first conversion means that converts only the first color difference signal and the second color difference signal of the second image to represent the first color difference signal and the second color difference signal of the second image in which the gains of the R component, the G component, and the B component have been corrected by the R correction gain, the B correction gain, and the G correction gain, thereby generating a color difference-corrected second image, and a second conversion means that converts the gain of a luminance signal of the color difference-corrected second image into a luminance signal corrected by the luminance correction gain, thereby generating the corrected second image.

[0023] In this case, the predetermined format is a YCbCr format in which the luminance signal is a Y signal, the first color difference signal is a Cb signal, and the second color difference signal is a Cr signal, and in the first conversion means:

[0024]

number

[0025] However, Y0, Cb0, and Cr0 on the right side of this conversion equation represent the Y, Cb, and Cr signals of the second image, and Y0, Cb', and Cr' on the left side represent the Y, Cb, and Cr signals of the second image after color difference correction. Furthermore, the composite image generation system may be configured such that the correction means includes second conversion means that extracts a luminance signal of the second image and converts it into a luminance signal whose gain has been corrected by the luminance correction gain; second conversion means that extracts a first color difference signal and a second color difference signal of the second image and converts them into first color difference signals and second color difference signals of the second image whose R component, G component, and B component have been gain corrected by the R correction gain, the B correction gain, and the G correction gain; and combination means that combines the luminance signal converted by the first conversion means and the first color difference signal and second color difference signal converted by the second conversion means as the luminance signal, first color difference signal, and second color difference signal of the corrected second image, to generate the corrected second image.

[0026] In this case, the predetermined format is a YCbCr format in which the luminance signal is a Y signal, the first color difference signal is a Cb signal, and the second color difference signal is a Cr signal, and in the second conversion means:

[0027]

number

[0028] However, Y0, Cb0, and Cr0 on the right side of this conversion equation represent the Y signal, Cb signal, and Cr signal of the second image, and Cb' and Cr' on the left side represent the Cb signal and Cr signal after conversion. In the above synthetic image generation system, the representative value may be an average value. According to the composite image generation system described above, only the first and second color difference signals of the second image are corrected with correction gains a, b, and c calculated from the ratio of the R / G / B representative values, and the luminance signal is corrected with correction gain d calculated from the ratio of the luminance signal representative value. This eliminates the need for the calculation of correcting the luminance signal Y with correction gains a, b, and c as in Equation 6 above, and results in a smaller correction value than when the luminance signal and the first and second color difference signals are all corrected with correction gains a, b, and c.

[0029] Therefore, the first image and the second image can be combined with a relatively small amount of processing while suppressing the occurrence of unnaturalness. The present invention also provides a rear view image display system for use in an automobile, which includes the above-described composite image generation system. The rear view image display system includes a monitor that displays the composite image generated by the composite image generation system, as well as the first camera that captures an image of the rear of the automobile from the side, and the second camera that captures an image of the rear of the automobile from the rear. [Effects of the Invention]

[0030] As described above, according to the present invention, it is possible to synthesize multiple images with partially overlapping shooting areas using a method that uses a luminance signal and two color difference signals to represent color, with a relatively small amount of processing while suppressing the occurrence of unnaturalness. [Brief explanation of the drawings]

[0031] [Figure 1] 1 is a block diagram showing a configuration of an image display system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing the arrangement of various cameras and monitors according to an embodiment of the present invention. [Figure 3] FIG. 1 is a block diagram showing a configuration of a left-hand image generation processing unit according to an embodiment of the present invention. [Figure 4] FIG. 10 is a diagram illustrating an example of the operation of a left image generation processing unit according to an embodiment of the present invention. [Figure 5] FIG. 10 is a diagram illustrating an example of the operation of a left image generation processing unit according to an embodiment of the present invention. [Figure 6] FIG. 10 is a block diagram showing another example of the configuration of the left image generation processing unit according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0032] Hereinafter, an embodiment of the present invention will be described. FIG. 1 shows the configuration of an image display system according to this embodiment. The image display system is a system mounted on an automobile, and includes a left side camera 1, a rear camera 2, a right side camera 3, an image processing device 4, a left side monitor 5, and a right side monitor 6, as shown in the figure. In such an image display system configuration, as shown in Figures 2a1 and 2a2, rear camera 2 is located at the rear of the vehicle and takes pictures of the area behind the vehicle from that position. Left side camera 1 is located on the left side of the vehicle and takes pictures of the area behind the vehicle from that position, and right side camera 3 is located on the right side of the vehicle and takes pictures of the area behind the vehicle from that position. Here, rear camera 2, left side camera 1, and right side camera 3 output images in the YCbCr format, which represents colors using a luminance signal Y signal and two color difference signals Cb signal and Cr signal.

[0033] The imaging areas of the rear camera 2 and the left side camera 1 partially overlap behind the vehicle, and the imaging areas of the rear camera 2 and the right side camera 3 partially overlap behind the vehicle. The left side monitor 5 and the right side monitor 6 are displays that display images. For example, as shown in FIG. 2b, the left side monitor 5 is positioned to the left of the center of the driver's seat in the left-right direction, and the right side monitor 6 is positioned to the right of the center of the driver's seat in the left-right direction.

[0034] Returning to Figure 1, the image processing device 4 has a left image processing unit 41 and a right image processing unit 42. The left image processing unit 41 combines the image taken by the left side camera 1 with the image taken by the rear camera 2 and displays it on the left side monitor 5, and the right image processing unit 42 combines the image taken by the right side camera 3 with the image taken by the rear camera 2 and displays it on the right side monitor 6.

[0035] The left image processing unit 41 and the right image processing unit 42 will be described below. The left image processing unit 41 and the right image processing unit 42 have the same configuration and perform the same operation. Therefore, the left image processing unit 41 will be used as a representative in the following description. The description of the right image processing unit 42 will be obtained by swapping the left and right sides of the description of the left image processing unit 41 below. FIG. 3 shows the configuration of the left image processing unit 41. As shown in the figure, the left image processing unit 41 includes a first comparison image extraction unit 4111, a first Y extraction unit 4112, a second Y extraction unit 4113, a Y gain calculation unit 4114, a second comparison image extraction unit 4115, a first RGB conversion unit 4116, a second RGB conversion unit 4117, an RGB gain calculation unit 4118, a CbCr correction unit 4119, a Y correction unit 4120, and a synthesis unit 4121.

[0036] The image captured by the left side camera 1 is input as the first image to the left image processing unit 41, and the image captured by the rear camera 2 is input as the second image. However, it is also possible to configure the system so that the image captured by the rear camera 2 is input as the first image and the image captured by the left side camera 1 is input as the second image.

[0037] The first comparison image extraction unit 4111 has a first comparison area preset to correspond to a portion of the shooting area of ​​the left side camera 1 that overlaps with the shooting area of ​​the rear camera 2, and the first comparison image extraction unit 4111 extracts an image within the first comparison area of ​​the first image as the first comparison image.

[0038] In addition, the second comparison image extraction unit 4115 has pre-set as the second comparison area an area in the second image that corresponds to the first comparison area in the first image, and therefore an area in the second image that contains the same subject as the first comparison area in the first image, and the second comparison image extraction unit 4115 extracts an image within the second comparison area of ​​the second image as the second comparison image.

[0039] The first Y extraction unit 4112 extracts the Y signal of the first comparison image, and the second Y extraction unit 4113 extracts the Y signal of the second comparison image. The Y gain calculation unit 4114 calculates the ratio of the representative value of the Y signal of the first comparison image to the representative value of the Y signal of the second comparison image as a correction gain d. In this embodiment, the average value of the Y signal is used as the representative value of the Y signal, but other representative values ​​such as the median value of the Y signal or the mode value of the Y signal can also be used as the representative value depending on the application and purpose of the image display system. Next, the first RGB converter 4116 converts the Y, Cb, and Cr signals of the first comparison image into R, G, and B signals, thereby converting the first comparison image into a first comparison image in the RGB format. Furthermore, the second RGB converter 4117 converts the Y, Cb, and Cr signals of the second comparison image into R, G, and B signals, thereby converting the second comparison image into a second comparison image in the RGB format.

[0040] The RGB gain calculation unit 4118 calculates a correction gain a, which is the ratio of the representative value of the R signal of the first comparison image in the RGB method to the representative value of the R signal of the second comparison image in the RGB method, calculates a correction gain b, which is the ratio of the representative value of the G signal of the first comparison image in the RGB method to the representative value of the G signal of the second comparison image in the RGB method, and calculates a correction gain c, which is the ratio of the representative value of the B signal of the first comparison image in the RGB method to the representative value of the B signal of the second comparison image in the RGB method.

[0041] In this embodiment, the average value of the R / G / B signals is used as the representative value of the R / G / B signals, but other representative values ​​such as the median value of the R / G / B signals or the mode value of the R / G / B signals can also be used as the representative value of the R / G / B signals depending on the application and purpose of the image display system. The CbCr correction unit 4119 corrects the Cb and Cr signals of the second image so that the gain correction of the R, G, and B components using the correction gains a, b, and c calculated by the RGB gain calculation unit 4118 is reflected. This correction corresponds to passing Y0 through as it is to Y' in the above-mentioned equation 6, and is therefore expressed by the conversion shown in equation 7.

[0042]

number

[0043] Here, in Equation 7, Y0, Cb0, and Cr0 on the right side represent the Y signal, Cb signal, and Cr signal of the second image, and Y0, Cb', and Cr' on the left side represent the Y signal, Cb signal, and Cr signal of the second image after correction. The Y correction unit 4120 corrects the gain of the Y signal of the second image corrected by the CbCr correction unit 4119 using the correction gain d calculated by the Y gain calculation unit 4114. The correction by the Y correction unit 4120 is to perform gain correction on only the Y signal with the correction gain d, and is therefore expressed by the conversion shown in Equation 8.

[0044]

number

[0045] In addition, in equation 8, Y0, Cb', and Cr' on the right side are the Y, Cb, and Cr signals of the second image corrected by the CbCr correction unit 4119, and Y', Cb', and Cr' on the left side are the Y, Cb, and Cr signals of the second image corrected by the Y correction unit 4120. Then, the synthesis unit 4121 converts the second image corrected by the Y correction unit 4120 into an image showing the rear view from the viewpoint of the left side camera 1, synthesizes the embedded image, which is the part of the viewpoint-converted second image used for synthesis with the first image, with the first image to form a single continuous image showing the rear view of the automobile, and outputs and displays it on the left side monitor 5.

[0046] Now, as described above, when only Cb and Cr of the second image are corrected with the correction gains a, b, and c, and Y is corrected with the correction gain d calculated from the luminance ratio, the amount of calculation required to correct the second image is smaller than when all of Y, Cb, and Cr are corrected with the correction gains a, b, and c according to Equation 6, because the calculation to correct Y with the correction gains a, b, and c is eliminated, as shown by comparing Equations 7 and 8 with Equation 6 above.

[0047] Furthermore, since the first term on the right side of Equation 7 converts the input Y signal directly into the output Y signal, the actual calculation processing of Equation 7 is to perform the calculation of the second term on the right side of Equation 7 and output it together with the input Y signal. Furthermore, when handling an image format in which the number of pixels having the color difference signals CbCr is smaller than the number of pixels having the luminance signal Y, such as 422 in which the number of pixels having the color difference signals CbCr is half the number of pixels having the luminance signal Y, or 420 in which the number of pixels having the color difference signals CbCr is one-fourth the number of pixels having the luminance signal Y, it is sufficient to perform the calculation of this second term only for pixels having the Cb and Cr signals.

[0048] Furthermore, the calculation process of Equation 8 is essentially a process of multiplying the luminance signal Y by the correction gain d and outputting the result together with the input Cb and Cr signals, so the amount of processing required for the calculation is small. Therefore, according to this embodiment, it is possible to synthesize the first image and the second image with a relatively small amount of processing while suppressing the occurrence of unnaturalness. An example of processing by the left image processing unit 41 will now be described. Now, it is assumed that a first image is taken by the left side camera 1 as shown in FIG. 4a1, and a second image is taken by the rear camera 2 as shown in FIG. 4b1. Then, the first comparison image extraction unit 4111 extracts from the first image of Figure 4a1 an image portion of the first comparison area C1, which is set as an area corresponding to part of the overlapping area between the shooting area of ​​the left side camera 1 and the shooting area of ​​the rear camera 2, as the first comparison image as shown in Figure 4a2, and the second comparison image extraction unit 4115 extracts from the second image of Figure 4b1 an image portion of the second comparison area C2, which is set as an area corresponding to the first comparison area C1, as the second comparison image as shown in Figure 4b2.

[0049] Then, from the first comparison image in Figure 4a2 and the second comparison image in Figure 4b2, the ratio of the average value of the first comparison image to the average value of the second comparison image for each of the R, G, B, and Y components is calculated as correction gains a, b, c, and d. Then, using the set correction gains, the Cb and Cr signals of the second image shown in Figure 5a1 are corrected using correction gains a, b, and c, and the Y signal is corrected using correction gain d, generating the corrected second image shown in Figure 5a2. Then, the synthesis unit 4121 converts the viewpoint of the corrected second image shown in Figure 5a2 and synthesizes it with the first image in Figure 5b to form a single continuous image showing the rear of the car, as shown in Figure 5c, and displays it on the left side monitor 5. Here, this synthesis is performed, for example, by extracting as an embedded image S a portion of the second image that shows the occluded area H, which is the area behind the vehicle that is blocked from the left side camera 1 by the vehicle, as shown in the figure, and then replacing with the embedded image S an area in the first image that would show the occluded area if the vehicle were transparent, by overwriting or the like.

[0050] FIG. 6 shows another example of the configuration of the left image processing unit 41 of the image processing device 4. In FIG. In the following description of the left image processing unit 41 in FIG. 6, a configuration in which the left and right are reversed will be another example of the configuration of the right image processing unit . The left image processing unit 41 shown in Figure 6, like the left image processing unit 41 shown in Figure 3, is equipped with a first comparison image extraction unit 4111, a first Y extraction unit 4112, a second Y extraction unit 4113, a Y gain calculation unit 4114, a second comparison image extraction unit 4115, a first RGB conversion unit 4116, a second RGB conversion unit 4117, and an RGB gain calculation unit 4118, and the operation of these units is the same as that of the left image processing unit 41 shown in Figure 3.

[0051] On the other hand, the left image processing unit 41 shown in Figure 6 includes a parallel Y correction unit 4122, a parallel CbCr correction unit 4123, and a combining unit 4124 instead of the CbCr correction unit 4119 and Y correction unit 4120 of the left image processing unit 41 shown in Figure 3. The parallel Y correction unit 4122 extracts the Y signal of the second image, corrects it with the correction gain d calculated by the Y gain calculation unit 4114, and outputs it. The parallel CbCr correction unit 4123 extracts the Cb and Cr signals of the second image, corrects them so that the gain correction of the R, G, and B components using the correction gains a, b, and c is reflected, and outputs the corrected signals. Here, the correction by the parallel-type Y corrector 4122 and the correction by the parallel-type CbCr corrector 4123 are performed in parallel. Furthermore, the correction by the parallel Y corrector 4122 is expressed by the conversion of Equation 9, and the correction by the parallel CbCr corrector 4123 is expressed by the conversion of Equation 10.

[0052]

number

[0053]

number

[0054] Note that Y0, Cb0, and Cr0 on the right-hand sides of equations 9 and 10 represent the Y signal, Cb signal, and Cr signal of the second image, Y' on the left-hand side of equation 9 is the Y signal corrected by the parallel Y correction unit 4122, and Cb' and Cr' on the left-hand side of equation 10 are the Cb signal and Cr signal corrected by the parallel CbCr correction unit 4123.

[0055] Then, the combining unit 4124 combines the Y signal corrected by the parallel-type Y corrector 4122 with the Cb and Cr signals corrected by the parallel-type CbCr corrector 4123 and outputs the combined signal to the synthesis unit 4121 . Here, this combination is expressed by Equation 11.

[0056]

number

[0057] Note that Y' on the right side of equation 11 is the Y signal corrected by the parallel Y correction unit 4122, Cb' and Cr' are the Cb signal and Cr signal corrected by the parallel CbCr correction unit 4123, and Y', Cb', and Cr' on the left side of equation 11 are the Y signal, Cb signal, and Cr signal of the second image output from the combining unit 4124 to the synthesis unit 4121.

[0058] The synthesis unit 4121 then converts the second image output from the combination unit 4124 into an image showing the rear view from the viewpoint of the left side camera 1, and synthesizes the embedded image, which is the part of the viewpoint-converted second image used for synthesis with the first image, with the first image to form a single continuous image showing the rear view of the vehicle, and outputs and displays it on the left side monitor 5.

[0059] As described above, in the left image processing unit 41 shown in FIG. 6, when only Cb and Cr of the second image are corrected with correction gains a, b, and c, and Y is corrected with correction gain d calculated from the luminance ratio, and the two are combined, the amount of calculation required to correct the second image is smaller than when all of Y, Cb, and Cr are corrected with correction gains a, b, and c according to Equation 6, because the calculation to correct Y with correction gains a, b, and c is eliminated, as shown by comparing Equations 9-11 with Equation 6 above.

[0060] Furthermore, when handling an image format in which the number of pixels having color difference signals CbCr, such as 422 or 420, is smaller than the number of pixels having a luminance signal Y, it is sufficient to perform the calculation of Equation 10 only for pixels having Cb and Cr signals. Furthermore, the calculation process of Equation 9 is essentially a process of multiplying the luminance signal Y by the correction gain d, and the calculation process of Equation 11 is simply a process of combining Y, Cb, and Cr, so the amount of processing required for the calculation is small. Therefore, the left image processing unit 41 shown in FIG. 6 can also synthesize the first image and the second image with a relatively small amount of processing while suppressing the occurrence of unnaturalness. The embodiments of the present invention have been described above. Here, in the above, we have shown a case where an image showing the rear view is generated by combining images taken by a left side camera 1 and a right side camera 3 mounted on the side of the vehicle with an image taken by a backup camera 2 mounted on the rear of the vehicle. However, this embodiment can be similarly applied to any combination of images taken by multiple cameras with partially overlapping shooting areas, including cases where the cameras are not mounted on the vehicle, as long as they combine images taken by multiple cameras with partially overlapping shooting areas, such as when images taken by multiple cameras with partially overlapping shooting areas that capture the area around the vehicle are converted from a viewpoint and combined to create an overhead (bird's-eye) image. [Explanation of symbols]

[0061] 1...left side camera, 2...back camera, 3...right side camera, 4...image processing device, 5...left side monitor, 6...right side monitor, 41...left image processing unit, 42...right image processing unit, 4111...first comparison image extraction unit, 4112...first Y extraction unit, 4113...second Y extraction unit, 4114...Y gain calculation unit, 4115...second comparison image extraction unit, 4116...first RGB conversion unit, 4117...second RGB conversion unit, 4118...RGB gain calculation unit, 4119...CbCr correction unit, 4120...Y correction unit, 4121...combination unit, 4122...parallel Y correction unit, 4123...parallel CbCr correction unit, 4124...combination unit.

Claims

1. A composite image generation system that combines a first image captured by a first camera with a second image captured by a second camera whose capture area partially overlaps with that of the first camera to generate a composite image that is an image obtained by joining together at least a portion of the first image and at least a portion of the second image, the first image and the second image are images of a predetermined format that represent colors using a luminance signal, a first color difference signal, and a second color difference signal, The synthetic image generation system includes: a comparison image extraction means for extracting a portion of the first image in which the range where the photographed area partially overlaps or a part of that portion as a first comparison image, and a portion of the second image in which the range that appears in the first comparison image is the same as that of the second comparison image as a second comparison image; a brightness correction gain calculation means for calculating a ratio of a representative value of a brightness signal of the first comparison image extracted by the comparison image extraction means to a representative value of a brightness signal of the second comparison image as a brightness correction gain; an RGB conversion means for converting the first comparative image and the second comparative image into an RGB format image that represents colors using R signals, G signals, and B signals; an RGB correction gain calculation unit that calculates an R correction gain as a ratio of a representative value of an R signal of the first comparison image converted into the RGB format to a representative value of an R signal of the second comparison image converted into the RGB format, a G correction gain as a ratio of a representative value of a G signal of the first comparison image converted into the RGB format to a representative value of the G signal of the second comparison image converted into the RGB format, and a B correction gain as a ratio of a representative value of a B signal of the first comparison image converted into the RGB format to a representative value of the B signal of the second comparison image converted into the RGB format; a correction means for correcting a gain of a luminance signal of the second image by the luminance correction gain, and correcting a first color difference signal and a second color difference signal of the second image so as to represent the first color difference signal and the second color difference signal of the second image in which the gains of the R component, the G component, and the B component have been corrected by the R correction gain, the B correction gain, and the G correction gain, thereby generating a corrected second image; and a synthesis unit for synthesizing the corrected second image with the first image to generate the synthesized image.

2. 2. The synthetic image generation system of claim 1, The correction means a first conversion means for converting only the first color difference signal and the second color difference signal of the second image so as to represent the first color difference signal and the second color difference signal of the second image in which the gains of the R component, the G component, and the B component have been corrected by the R correction gain, the B correction gain, and the G correction gain, thereby generating a color difference-corrected second image; and second conversion means for converting the gain of the luminance signal of the color difference corrected second image into a luminance signal corrected with the luminance correction gain to generate the corrected second image.

3. 2. The synthetic image generation system of claim 1, The correction means a second conversion means for extracting a luminance signal of the second image and converting the extracted luminance signal into a luminance signal whose gain has been corrected by the luminance correction gain; a second conversion means for extracting a first color difference signal and a second color difference signal of the second image and converting them into a first color difference signal and a second color difference signal of the second image in which the gains of the R component, the G component, and the B component have been corrected by the R correction gain, the B correction gain, and the G correction gain; a combining means for combining the luminance signal converted by the first converting means and the first and second color difference signals converted by the second converting means as the luminance signal, the first color difference signal, and the second color difference signal of the corrected second image, thereby generating the corrected second image.

4. 3. The synthetic image generation system according to claim 2, the predetermined format is a YCbCr format in which the luminance signal is a Y signal, the first color difference signal is a Cb signal, and the second color difference signal is a Cr signal; The first conversion means [Equation 1] The conversion is expressed by the conversion formula: Y on the right side of the conversion equation 0 , Cb 0 , Cr 0 represents the Y signal, Cb signal, and Cr signal of the second image, and the Y on the left side 0 , Cb', Cr' represent the Y signal, Cb signal, Cr signal of the second image after color difference correction.

5. 4. The synthetic image generation system according to claim 3, the predetermined format is a YCbCr format in which the luminance signal is a Y signal, the first color difference signal is a Cb signal, and the second color difference signal is a Cr signal; The second conversion means [Equation 2] The conversion is expressed by the conversion formula: Y on the right side of the conversion equation 0 , Cb 0 , Cr 0 represents the Y signal, Cb signal, and Cr signal of the second image, and Cb' and Cr' on the left side represent the Cb signal and Cr signal after conversion.

6. 6. A synthetic image generation system according to claim 1, 2, 3, 4 or 5, The synthetic image generating system is characterized in that the representative value is an average value.

7. A rear view image display system mounted on an automobile, comprising the synthetic image generation system according to claim 1, 2, 3, 4, 5 or 6, a monitor for displaying the composite image generated by the composite image generation system; The first camera is a camera that photographs the rear of the vehicle from the side of the vehicle, A rear view image display system characterized in that the second camera is a camera that is mounted at the rear of the vehicle and captures an image of the area behind the vehicle.

Images