Image processing device and image processing method

Abstract

Image processing device comprising the following: a fusion processing unit (32) which, using a high-quality image as a reference, performs distortion on a wide-angle image with a lower image quality and a wider angle of view than the high-quality image, wherein the fusion processing unit (32) performs fusion processing with a high-quality image reference, which generates a fusion image by aligning the positions of overlapping image areas of the high-quality image and the wide-angle image after distortion; wherein the fusion processing unit (32) performs the distortion based on parallax information that specifies a parallax between the high-quality image and the wide-angle image, and The fusion processing unit (32) interpolates parallax information in a non-overlapping image area that is not included in the high-quality image in the wide-angle image using parallax information to the overlapping image area.

Description

Technical area

[0001] This technology refers to an image processing apparatus and method, and enables generation of a captured image with a wide viewing angle and high resolution using captured images each having different viewing angles and resolutions without deteriorating the image quality. State of the art

[0002] In conventional portable electronic devices such as data processing terminals, such as smartphones, the image quality of the associated imaging units deteriorates compared to SLR cameras and the like due to downsizing and reduction in thickness. For this reason, Patent Document 1, for example, discloses that multiple imaging units are provided and multiple images with different image qualities are simultaneously generated, for example, images with a first image angle and a second image angle smaller than the first image angle.

[0003] Patent Document 2 describes an image pickup device and a digital zooming method in which a scene is captured with a primary and secondary lens to generate primary and secondary images. These images are subjected to image correction to obtain corresponding corrected images. By detecting and matching feature points in the corrected images, the overlapping areas are identified, and pixel shifts and depth maps are calculated. Based on these depth maps and pixel shifts, the images are magnified and distorted to achieve a match. Subsequently, the overlapping areas of the distorted images are fused using a weighted blending method to generate a digitally magnified image. Citation listPatent document Patent document 1: JP 2013 - 219 525 A Patent document 2: US 2016 / 0142627 A1 Summary of the inventionProblems to be solved by the invention

[0004] However, it is impossible to obtain a captured image exceeding the performance of the imaging units only by providing the plurality of imaging units as in Patent Document 1.

[0005] This technology therefore has the object of providing an image processing apparatus and an image processing method capable of generating a captured image with a wide image angle and high image quality using a plurality of captured images each having different image angles and image qualities, without deteriorating the image quality. Solutions to the problems

[0006] The object is achieved by an image processing device according to claim 1 and an image processing method according to claim 14.

[0007] In this technology, the fusion processing unit performs fusion processing with a high-quality image reference. The fusion processing unit performs warping with the high-quality image as a reference on the wide-angle image, which has lower image quality and a wider image angle than the high-quality image. Warping is performed based on parallax information regarding the high-quality image and the wide-angle image, and parallax information in a non-overlapping image region of the wide-angle image that is not included in the high-quality image is interpolated using parallax information of the overlapping image region. Furthermore, the fusion processing unit generates the fusion image by aligning the positions of the overlapping image regions of the high-quality image, which has higher image quality than the wide-angle image, and the wide-angle image after warping with each other.

[0008] Further, based on a determination result of a determination unit, the fusion processing unit performs fusion processing with a high-quality image reference when a predetermined condition is met, and performs fusion processing with a wide-angle image reference when the predetermined condition is not met. In the fusion processing with a wide-angle image reference, warping is performed with the wide-angle image as a reference using the high-quality image, and a fusion image is generated by aligning the positions of the overlapping image areas of the high-quality image after warping and the wide-angle image on which warping is not performed.

[0009] For example, the determination unit sets as a predetermined condition that an object distance is greater than a determination threshold, wherein the object distance is determined based on the parallax information for an object positioned in the non-overlapping image area in the wide-angle image that is not included in the high-quality image. Furthermore, the determination unit sets as a predetermined condition that a luminance difference between average luminances of an image area in the overlapping image area within a predetermined distance from a boundary with the non-overlapping image area is, for example, less than or equal to a determination threshold.

[0010] Furthermore, the fusion processing unit performs narrow-angle fusion processing. The fusion processing unit performs warping with the high-quality image as a reference using the wide-angle image and generates the fusion image with an image angle of the high-quality image by aligning the positions of the overlapping image areas of the high-quality image and the wide-angle image after warping.

[0011] The fusion processing unit performs high-quality image reference fusion processing or wide-angle image reference fusion processing based on the determination result of the determination unit in a case where the high-quality image and the wide-angle image are moving images, and a fusion image having an image angle of the wide-angle image is generated. The fusion processing unit performs high-quality image reference fusion processing, wide-angle image reference fusion processing, or narrow-angle fusion processing in a case where the high-quality image and the wide-angle image are still images. The fusion processing unit performs wide-angle image reference fusion processing or narrow-angle fusion processing based on the determination result of the determination unit in a case where the predetermined condition is not met.Furthermore, the fusion processing unit performs the narrow-angle fusion processing in a case where it is determined by the determination unit that the mode selection for automatically adjusting an image angle of the fusion image is performed in a case where the predetermined condition is not met. Further, the determination unit determines a user selection operation of the fusion processing, and the fusion processing unit performs the narrow-angle fusion processing regardless of whether the predetermined condition is met or not when it is determined by the determination unit that the narrow-angle fusion processing is selected. Effects of the invention

[0012] According to this technology, warping is performed with the high-quality image as a reference on the wide-angle image with the lower image quality and the wider image angle than the high-quality image, and processing that synthesizes the high-quality image with the higher image quality and the wide-angle image after warping by aligning the positions of the overlapping image areas with each other is performed, so that the fused image is generated. Thus, the captured image with the wide image angle and high image quality can be generated using the multiple captured images each having different image angles and image qualities without deteriorating the image quality. Note that the advantageous effects described in this specification are merely examples, and the advantageous effects of the present technology are not limited to these and may include additional effects. Brief description of the drawings Fig. 1 is a diagram showing an appearance of a device to which an image processing apparatus is applied. Fig. 2 is a diagram illustrating a configuration of a data processing terminal. Fig. 3 is a diagram showing a pixel arrangement of an imaging unit. Fig. Figure 4 is a diagram showing occlusion when a captured monochrome image is a reference. Fig. 5 is a diagram showing captured images used to generate a fusion image. Fig. 6 is a diagram showing a configuration of an embodiment. Fig. 7 is a flowchart illustrating the operation of one embodiment of an image processing unit. Fig. Figure 8 is a flowchart illustrating the fusion processing determination. Fig. 9 is a diagram schematically showing a positional relationship between imaging units and objects. Fig. 10 is a diagram showing a relationship between an object distance and captured images acquired by the imaging units. Fig. 11 is a diagram showing a near vision detection area. Fig. Figure 12 is a flowchart illustrating reference image determination based on distance. Fig. 13 is a diagram for explaining the reference image determination based on the luminance difference. Fig. 14 is a diagram showing a luminance calculation range. Fig. Figure 15 is a flowchart illustrating reference image determination based on luminance difference. Fig. 16 is a diagram to explain the integration determination. Fig. Figure 17 is a diagram explaining parallax detection. Fig. Fig. 18 is a diagram showing the performance of the fusion image generated based on a fusion determination result. Fig. 19 is a block diagram showing an example of a schematic configuration of a vehicle control system. Fig. 20 is an explanatory diagram showing an example of installation positions of a vehicle exterior information detection unit and an imaging unit. Mode of carrying out the invention

[0013] The following is a description of embodiments for practicing the present technology. Note that the descriptions are presented in the following order. 1. Configuration of the device to which the image processing device is applied 2. Embodiment of an image processing device 2-1. Configuration of the embodiment 2-2. Operation of the embodiment 3. Other embodiments 4. Application example <1. Configuration of the device to which the image processing device is applied>

[0014] Fig. Figure 1 shows an appearance of a device to which an image processing device of this technology is applied. Note that in the following description, the image processing device is applied to a data processing terminal, for example. The front side of a data processing terminal 10 is shown in (a) of Fig. 1, and a display unit 53, a touch panel 54, and an operation unit 55 are provided on the front side. The rear side of the data processing terminal 10 is shown in (b) of Fig. 1 and several imaging units, for example two imaging units 21-H and 21-W, are provided on the back.

[0015] Fig. Figure 2 shows a configuration of the data processing terminal. The data processing terminal 10 includes a plurality of imaging units, for example, two imaging units 21-H and 21-W, an image processing unit 30, a sensor unit 51, a communication unit 52, a display unit 53, a touch panel 54, an operation unit 55, a storage unit 56, and a control unit 60. The image processing unit 30 corresponds to the image processing device of this technology.

[0016] The imaging units 21-H and 21-W are provided on the same side of the data processing terminal 10, as shown in (b) of Fig. 1. The imaging units 21-H and 21-W are each formed using an imaging element such as a complementary metal oxide semiconductor (CMOS) image sensor, perform photoelectric conversion of light captured by a lens (not shown), generate image data of a captured image, and output the image data to the image processing unit 30. The imaging units 21-H and 21-W have a characteristic difference: the imaging unit 21-H has higher image quality than the imaging unit 21-W, and the imaging unit 21-W has a wider angle of view than the imaging unit 21-H.

[0017] Fig. 3 shows a pixel arrangement of the imaging unit. The pixel arrangement of the imaging unit 21-H is shown in (a) of Fig. 3. For all pixels, the imaging unit 21-H is formed by white (W) pixels that output electrical signals based on the amount of incident light across the entire wavelength range of visible light. Thus, the imaging unit 21-H generates image data of a captured monochrome image.

[0018] The pixel arrangement of the imaging unit 21-W is shown in (b) by Fig. 3. The imaging unit 21-W is formed, for example, using a color filter in which red (R) pixels, blue (B) pixels, and green (G) pixels are arranged in a Bayer array. In the Bayer array, two pixels at diagonal positions in a 2×2 pixel unit are green (G) pixels, and the remaining pixels are one red (R) pixel and one blue (B) pixel. In other words, the imaging unit 21-W is formed by color pixels that output electrical signals based on the amount of incident light in one of the color components red, blue, and green. Thus, the imaging unit 21-W generates image data of a captured color image in which each pixel indicates one of the three primary color components (RGB components).

[0019] The image processing unit 30 performs image processing using a captured image with high image quality generated by the imaging unit 21-H and a captured image with a wide image angle generated by the imaging unit 21-W to generate a captured image with the wide image angle and high image quality, and outputs the image to the display unit 53 and the storage unit 56. Note that details of the configuration and operation of the image processing unit 30 will be described later.

[0020] The sensor unit 51 is configured using a gyro sensor and the like and detects a wobble occurring in the data processing terminal 10. The sensor unit 51 outputs information about the detected wobble to the control unit 60.

[0021] The communication unit 52 communicates with a device via a network, for example a local area network (LAN) or the Internet.

[0022] The display unit 53 displays a captured image based on image data supplied from the image processing unit 30, and displays a menu screen, various application screens, and the like based on an information signal from the control unit 60. Furthermore, the touch panel 54 is arranged on the display surface side of the display unit 53 and is configured to utilize a GUI function.

[0023] The operation unit 55 is configured using an operation switch and the like, generates an operation signal corresponding to a user operation, and outputs the signal to the control unit 60.

[0024] The storage unit 56 stores information generated by the data processing terminal 10, such as the image data supplied from the image processing unit 30, and various types of information used to execute communications and applications in the data processing terminal 10.

[0025] The control unit 60 includes a central processing unit (CPU), a read-only memory (ROM), a random access memory (RAM) (not shown), and the like. The control unit 60 executes a program stored in the ROM or RAM and controls the operation of each unit so that an operation corresponding to the user operation on the touch panel 54 or the operation unit 55 is performed in the data processing terminal 10.

[0026] It should be noted that the data processing terminal 10 is not limited to the Fig. 2, and the data processing terminal 10 may include, for example, an encoding processing unit for encoding image data and storing the encoded image data in the storage unit 56, a resolution conversion unit for adjusting the image data to the resolution of the display unit, and the like. <2. Embodiment of the image processing device><2-1. Configuration of the embodiment>

[0027] The image processing unit 30 performs fusion processing using the captured monochrome image with high image quality acquired by the imaging unit 21-H and the captured color image with a wide angle of view acquired by the imaging unit 21-W. The image processing unit 30 performs high-quality image reference fusion processing as fusion processing.In a case where the fusion processing is performed with a high-quality image reference, the image processing unit 30 performs warping with the captured color image having a lower image quality and a wider image angle than the captured monochrome image as the viewpoint of the captured monochrome image, with the captured monochrome image having the high image quality as a reference, and generates a fusion image by aligning positions of overlapping image areas of the captured monochrome image and the captured color image after warping with each other.The image processing unit 30 performs the fusion processing with high-quality image reference and generates the fusion image with an image angle of the imaging unit 21-W in which an overlapping image area, which is an object area captured by the imaging unit 21-H and the imaging unit 21-W, is provided with a high image quality corresponding to the performance of the imaging unit 21-H.

[0028] Since the viewpoints between the imaging unit 21-H and the imaging unit 21-W differ from each other, the occlusion increases in a near view compared to a far view. Fig. 4 shows the occlusion when the captured monochrome image acquired by the imaging unit 21-H is the reference. When the occlusion occurs due to parallax, there is no image data corresponding to an occlusion region in the captured color image acquired by the imaging unit 21-W. For this reason, in the fused image generated by the fusion processing using the captured monochrome image as the reference, color information in the occlusion region is missing, and the image quality of the fused image may be deteriorated compared to that in the captured color image acquired by the imaging unit 21-W.

[0029] Furthermore, the viewing angles differ between the imaging unit 21-H and the imaging unit 21-W. As shown in Fig. 5, captured images used to generate the fusion image thus include a non-overlapping image area (hereinafter referred to as a “frame area”) ARa indicated only by the captured color image with the wide angle of view acquired by the imaging unit 21-W, and an overlapping image area ARb indicated by a captured monochrome image Gbw-H with the high image quality acquired by the imaging unit 21-H and a captured color image Gcr-W acquired by the imaging unit 21-W.Further, when the luminance differs between the captured monochrome image Gbw-H and the captured color image Gcr-W due to a sensitivity difference, an exposure difference, or the like between the imaging unit 21-H and the imaging unit 21-W, in the fusion image generated by the fusion processing with the captured monochrome image as a reference, a boundary due to a luminance difference between the frame area ARa and the overlapped image area ARb becomes conspicuous.

[0030] The image processing unit 30 therefore determines whether a predetermined condition is satisfied, and in a case where it is determined that the predetermined condition is satisfied, in other words, in a case where it is determined that image quality deterioration due to object distance, luminance difference, or the like does not occur, the image processing unit 30 performs fusion processing with a high-quality image reference. In a case where it is determined that the predetermined condition is not satisfied, in other words, in a case where it is determined that image quality deterioration occurs due to object distance, luminance difference, and / or the like, the image processing unit 30 performs fusion processing with a wide-angle image reference.In a case where fusion processing is performed with a wide-angle image reference, the image processing unit 30 performs warping with the captured color image with high image quality as the focal point, and the captured color image with the wide image angle as the reference, and generates a fusion image by aligning the positions of the overlapping image areas of the captured color image and the captured monochrome image after warping. The image processing unit 30 performs such fusion processing with a wide-angle image reference to prevent image quality deterioration due to object distance, luminance difference, or the like.

[0031] Fig. 6 shows a configuration of one embodiment. The image processing unit 30 includes preprocessing units 31-H and 31-W, a fusion processing unit 32, and a determination unit 33. Furthermore, the fusion processing unit 32 includes a parallax detection unit 321, a frame area interpolation unit 322, distortion units 323-H and 323-W, and an image synthesis unit 324. The determination unit 33 includes a distance measuring unit 331, a distance determination unit 332, a luminance difference determination unit 333, and an integration determination unit 334.

[0032] The pre-processing unit 31-H performs correction processing such as lens distortion correction and defective pixel correction on the image data of the captured monochrome image acquired by the imaging unit 21-H. The pre-processing unit 31-H outputs the corrected image data (hereinafter referred to as "monochrome image data") to the fusion processing unit 32.

[0033] The pre-processing unit 31-W performs correction processing such as lens distortion correction and defective pixel correction on the image data of the captured color image acquired by the imaging unit 21-W. Furthermore, the pre-processing unit 31-W performs demosaicing processing using the image data after the correction processing. In the demosaicing processing, three primary color image data indicating respective color components of red, blue, and green are generated for each pixel from image data in which each pixel indicates one of the color components of red, blue, and green. Furthermore, the pre-processing unit 31-W performs color space conversion of the three primary color image data, generates color image data including color difference data and luminance data corresponding to a component of a monochrome image, and outputs the generated data to the fusion processing unit 32.

[0034] The parallax detection unit 321 of the fusion processing unit 32 performs parallax detection based on the monochrome image data supplied from the pre-processing unit 31-H and the color image data supplied from the pre-processing unit 31-W, and generates parallax information indicating a detected parallax. Since the imaging unit 21-H and the imaging unit 21-W perform imaging from different viewpoint positions, as shown in (b) of Fig. As shown in Figure 1, the captured images acquired by the imaging unit 21-H and the imaging unit 21-W are images with parallax. Thus, the parallax detection unit 321 generates parallax information indicating a parallax for each pixel based on the image data supplied from the pre-processing unit 31-H and the pre-processing unit 31-W.

[0035] The parallax detection unit 321 performs parallax detection using the monochrome image data and the color image data, and generates parallax information indicating a parallax between the captured monochrome image and the captured color image. Further, when generating the parallax information, the parallax detection unit 321 uses the captured monochrome image or the captured color image as a reference based on the fusion determination information provided by the determination unit 33, as described later. For example, in a case where the fusion determination information indicates that fusion processing is performed using the captured color image as a reference (fusion processing using a wide-angle image as a reference), the parallax detection unit 321 generates parallax information using the captured color image as a reference image.In a case where the fusion determination information indicates that fusion processing is performed using the captured monochrome image as a reference (fusion processing with high-quality image reference), the parallax detection unit 321 generates parallax information using the captured monochrome image as a reference image. The parallax detection unit 321 performs point correspondence detection processing such as block matching to generate parallax information. For example, the parallax detection unit 321 detects a block area on the other captured image that is most similar to a reference block area, using a target position on a captured reference image as a reference. Further, the parallax detection unit 321 calculates a parallax vector indicating a difference between positions of the detected block area and the reference block area.Furthermore, the parallax detection unit 321 calculates a parallax vector using each pixel on the reference detection image as the target position and generates parallax information indicating the parallax vector calculated for each pixel. The parallax detection unit 321 outputs the generated parallax information to the frame area interpolation unit 322, the distortion unit 323-H, and the determination unit 33.

[0036] In a case where the fusion determination information provided by the determination unit 33 indicates that high-quality image reference fusion processing is performed using the captured monochrome image as a reference, the frame area interpolation unit 322 generates parallax information in the frame area through interpolation processing. Since the captured color image has a wider angle of view than the captured monochrome image, the parallax information generated by the parallax detection unit 321 does not include parallax information about the frame area in the captured color image that is not included in the captured monochrome image.Thus, the frame area interpolation unit 322 generates the parallax information in the frame area by interpolation processing using the parallax information generated with the captured monochrome image as a reference, so that warping can be performed with the captured color image as the image of the viewpoint of the captured monochrome image.

[0037] The distortion unit 323-H distorts the captured monochrome image based on the fusion determination information from the determination unit 33. In a case where the fusion determination information indicates that fusion processing is performed with a wide-angle image reference, the distortion unit 323-H performs distortion using the captured monochrome image as the viewpoint image of the captured color image. The distortion unit 323-H generates monochrome image data of the viewpoint of the captured color image from the monochrome image data and outputs the generated data to the image synthesis unit 324. Furthermore, in a case where the fusion determination information indicates that fusion processing is performed with a high-quality image reference, the distortion unit 323-H outputs the monochrome image data to the image synthesis unit 324 before changing the viewpoint.

[0038] The distortion unit 323-W distorts the captured color image based on the fusion determination information from the determination unit 33. In a case where the fusion determination information indicates that fusion processing is performed with a high-quality image reference, the distortion unit 323-W performs distortion using the captured color image as the viewpoint image of the captured monochrome image. The distortion unit 323-W generates color image data of the viewpoint of the captured monochrome image from the color image data and outputs the generated data to the image synthesis unit 324. Furthermore, in a case where the fusion determination information indicates that fusion processing is performed with a wide-angle image reference, the distortion unit 323-W outputs the color image data to the image synthesis unit 324 before changing the viewpoint.

[0039] The image synthesis unit 324 performs synthesis of the captured monochrome image and the captured color image. The image synthesis unit 324 aligns the positions of overlapping image areas of the monochrome image data output by the distortion unit 323-H and the color image data output by the distortion unit 323-W, and then synthesizes luminance image data of the monochrome image data and the color image data. The image synthesis unit 324 outputs the color image data, including the color difference data and the luminance data, after the synthesis as image data of the fused image.

[0040] The determination unit 33 determines whether the predetermined condition regarding image quality deterioration is met during generation of the fused image, generates fusion determination information based on a determination result, and outputs the fusion determination information to the fusion processing unit 32. Furthermore, the determination unit 33 generates the fusion determination information based on setting information corresponding to the user operation. For example, the determination unit 33 sets, as a predetermined condition, that the object distance determined based on the parallax information is greater than the determination threshold. Further, the determination unit 33 sets, for example, as a predetermined condition that a luminance difference between the high-quality image and the wide-angle image is less than or equal to the determination threshold.

[0041] The distance measuring unit 331 of the determination unit 33 generates distance measurement information of the object. The distance measuring unit 331 may be provided in the imaging unit 21-H or the imaging unit 21-W, or may be provided separately from the imaging units. Further, the distance measuring unit 331 may generate the distance measurement information using either a passive method or an active method. In a case where the distance measuring unit using the passive method is provided in the imaging unit, the imaging unit uses, for example, an imaging element provided with an image-plane phase difference detection pixel on its image plane. The image-plane phase difference detection pixel individually generates image signals of one image and the other image divided by pupil division.The distance measuring unit detects a phase difference between one image and the other image using the image signals generated by the image-plane phase difference detection pixel, and calculates the object distance, which is a distance to the object, based on the phase difference. For example, in a case where the distance measuring unit using the passive method is provided separately from the imaging unit, in the distance measuring unit, a pair of line sensors are provided separately from the imaging unit, and the one image and the other image divided by pupil division are formed on the pair of line sensors. The distance measuring unit detects a phase difference between the images generated on the pair of line sensors and calculates the object distance based on the phase difference.Moreover, in a case where the active method is used, the distance measuring unit 331 outputs light or radio waves and measures the object distance, which is a distance from the object, based on reflected light or radio waves.

[0042] The distance measuring unit 331 outputs the distance measurement information indicating the object distance to the distance determination unit 332. Furthermore, the distance measuring unit 331 may generate a depth map indicating an imaging depth for each pixel in the captured image and output the depth map as distance measurement information to the distance determination unit 332.

[0043] The distance determination unit 332 determines the reference image in the fusion processing based on the distance measurement information generated by the distance measurement unit 331. The distance determination unit 332 determines that the reference image is the captured color image when the object distance in a predetermined near-vision detection range is less than or equal to the determination threshold, and determines that the reference image is the captured monochrome image when the object distance is greater than the determination threshold. Note that the near-vision detection range will be described later. The distance determination unit 332 outputs a reference image determination result based on the object distance to the integration determination unit 334.

[0044] The luminance difference determination unit 333 determines the reference image in the fusion processing based on the captured monochrome image output from the warping unit 323-H and the captured color image output from the warping unit 323-W. The luminance difference determination unit 333 sets a predetermined range from the boundary with the frame area in the overlapping image area as the luminance calculation area and calculates, for example, an average luminance value of the luminance calculation area for the captured monochrome image and the captured color image.Furthermore, the luminance difference determination unit 333 determines that the reference image is the captured color image when a difference between the average luminance values ​​calculated for the captured monochrome image and the captured color image is greater than the determination threshold, and determines that the reference image is the captured monochrome image when the difference between the average luminance values ​​is less than or equal to the determination threshold. The luminance difference determination unit 333 outputs a reference image determination result based on the luminance difference to the integration determination unit 334.

[0045] The integration determination unit 334 determines which type of fusion processing is performed based on the reference image determination results of the distance determination unit 332 and the luminance difference determination unit 333, and the setting information corresponding to the user operation provided from the control unit. Based on the reference image determination result and the setting information, the integration determination unit 334 determines which processing is performed from among high-quality image reference fusion processing, wide-angle image reference fusion processing, or narrow-angle fusion processing. Based on the reference image determination result, the integration determination unit 334 further determines, for example, which processing is performed from among high-quality image reference fusion processing or wide-angle image reference fusion processing during the imaging operation.Note that in narrow-angle fusion processing, warping is performed with the captured monochrome image as a reference using the captured color image. After warping, the captured monochrome image and the captured color image are synthesized by aligning the positions of the overlapping image areas, thereby generating the fused image with an image angle of the captured monochrome image. The integration determination unit 334 outputs the fusion determination information indicating a determination result of the fusion processing to the fusion processing unit 32.Thus, in the fusion processing unit 32, based on the determination result of the determination unit 33, in other words, the fusion determination information, the high-quality image reference fusion processing, the wide-angle image reference fusion processing, or the narrow-angle fusion processing is selectively performed, so that the fusion image with the high image quality can be generated. <2-2. Operation of the embodiment>

[0046] Fig. 7 is a flowchart illustrating the operation of one embodiment of the image processing unit. In step ST1, the image processing unit performs preprocessing. The preprocessing units 31-H and 31-W of the image processing unit 30 acquire the image data of the captured images from the imaging units 21-H and 21-W, respectively, and perform correction processing such as lens distortion correction and defect pixel correction. Furthermore, the preprocessing unit 31-W performs demosaicing, color space conversion, and the like using the image data of the color captured image acquired by the imaging unit 21-W, and generates color image data including the luminance data representing the component equal to the captured monochrome image acquired by the imaging unit 21-H and the color difference data.The preprocessing units 31-H and 31-W of the image processing unit 30 perform preprocessing using image data of the captured monochrome image and the captured color image and proceed to step ST2.

[0047] In step ST2, the image processing unit performs fusion processing determination. The determination unit 33 of the image processing unit 30 determines, based on the user's setting operation and the determination results of the distance determination unit and the luminance difference determination unit, which processing is to be executed from among high-quality image reference fusion processing, wide-angle image reference fusion processing, or narrow-angle fusion processing, and generates fusion determination information indicating the determination result. Fig. Figure 8 is a flowchart illustrating the fusion processing determination.

[0048] In step ST11, the image processing unit acquires the setting information. The determination unit 33 of the image processing unit 30 acquires the setting information corresponding to the user operation from the control unit 60, such as information indicating whether a moving image or a still image is being generated, information indicating a generation mode of the fused image, and the like, and proceeds to step ST12.

[0049] In step ST12, the image processing unit performs a reference image determination based on the distance. Fig. 9 and Fig. 10 are diagrams for explaining the reference image determination based on the distance. Fig. 9 schematically shows a positional relationship between the imaging units and objects, where the imaging unit 21-H is arranged on the left side of an object OBa, and the imaging unit 21-W is arranged on the right side of the object OBa. Furthermore, an object OBb is present at a distant position. An angle of view CVs of the imaging unit 21-H is a normal angle of view, and an angle of view CVw of the imaging unit 21-W is wider than the normal angle of view and is a wide angle of view.

[0050] Fig. 10 shows a relationship between the object distance and the captured images acquired by the imaging units. As a distance to the object OBa becomes shorter, a position of the object OBa in the captured monochrome image Gbw-H obtained by the imaging unit 21-H moves to the right, and the position of the object OBa in the captured color image Gcr-W obtained by the imaging unit 21-W moves to the left. As the distance to the object OBa becomes shorter, the object OBb in the captured color image Gcr-W is occluded by the object OBa. In a case where the fusion image with the wide angle of view is generated using the captured monochrome image Gbw-H as a reference, the image processing unit needs to use the captured color image Gcr-W for the frame area Ara indicated by the dashed line.Here, when the distance to the object OBa is short and the object OBb is occluded by the object OBa in the acquired color image Gcr-W, the object OBb cannot be displayed in the frame area ARa in the fusion image. Thus, the image processing unit sets the frame area ARa as shown in . Fig. 11, the system sets a near-vision detection area ARd for the captured color image Gcr-W, that is, the wide angle of view, and switches the reference image depending on whether the object distance of the object included in the near-vision detection area ARd is greater than the predetermined determination threshold. Note that in a case where the fusion image having the wide angle of view is generated using the captured monochrome image as a reference, the captured color image is used for the frame area. Thus, the frame area is set as the near-vision detection area, and the determination threshold is set in advance, so that it is possible to determine whether there is a possibility that a distant object is occluded by a near object in the frame area.

[0051] Fig. 12 is a flowchart illustrating distance-based reference image determination. In step ST21, the determination unit detects the object distance of the object in the near-vision detection area. The determination unit 33 detects the object distance of the object in the near-vision detection area (e.g., the frame area) in the captured color image using the distance measuring sensor with the passive method or the active method, the depth map, or the like, and proceeds to step ST22.

[0052] In step ST22, the determination unit determines whether the object distance is greater than the determination threshold. The determination unit 33 proceeds to step ST23 if the object distance detected in step ST21 is greater than the determination threshold, and to step ST24 if the distance is less than or equal to the determination threshold.

[0053] In step ST23, the determination unit determines that the captured monochrome image is the reference image. In the case where the object distance is greater than the determination threshold, the fusion image generated using the captured monochrome image as the reference image has a smaller occlusion area because the object is removed, so an error in an image of the frame area is inconspicuous. Thus, the determination unit 33 determines that the captured monochrome image is the reference image.

[0054] In step ST24, the determination unit determines that the captured color image is the reference image. In a case where the object distance is less than or equal to the determination threshold and the object is close, in the fusion image generated using the captured monochrome image as the reference image, the object is close and the occlusion area becomes larger, causing defects in the image in the frame area to become conspicuous. Thus, the determination unit 33 determines that the captured color image is the reference image.

[0055] As described above, the image processing unit determines whether the captured monochrome image or the captured color image is set as the reference image based on the object distance in the near vision detection area, and goes to step ST13 in Fig. 8 over.

[0056] In step ST13, the image processing unit performs the reference image determination based on the luminance difference. Fig. Figure 13 is a diagram for explaining the reference image determination based on the luminance difference. In Fig. 13 shows the captured monochrome image and the captured color image, and for example, the captured monochrome image Gbw-H is a brighter image than the captured color image Gcr-W.

[0057] In a case where the captured monochrome image with high image quality and the captured color image with a wide angle of view are synthesized in the image processing unit, if the luminance of the captured monochrome image Gbw-H differs from the luminance of the captured color image Gcr-W, the luminance difference at the boundary between the frame area ARa and the overlapping image area ARb becomes remarkable in the fusion image. Thus, as shown in Fig. 14, a predetermined range from the boundary with the frame area ARa in the overlapping image area ARb is set as the luminance calculation area ARe, and the image processing unit calculates, for example, an average luminance value of the luminance calculation area ARe in the captured monochrome image Gbw-H and the captured color image Gcr-W. Further, the image processing unit performs setting of the reference image depending on whether the luminance difference, which is a difference between an average luminance value calculated from the captured monochrome image and an average luminance value calculated from the captured color image, is less than or equal to the predetermined determination threshold.

[0058] Fig. 15 is a flowchart illustrating the reference image determination based on the luminance difference. In step ST31, the determination unit performs alignment processing. The determination unit 33 performs alignment between the captured monochrome image and the captured color image so that the positions of the overlapping image areas of the captured monochrome image and the captured color image are aligned, and proceeds to step ST32.

[0059] In step ST32, the determination unit calculates the average luminance using the captured monochrome image. The determination unit 33 calculates the average luminance of the luminance calculation area ARe in the captured monochrome image and proceeds to step ST33.

[0060] In step ST33, the determination unit calculates the average luminance using the captured color image. The determination unit 33 calculates the average luminance of the luminance calculation area ARe in the captured color image and proceeds to step ST34.

[0061] In step ST34, the determination unit calculates the luminance difference. The determination unit 33 calculates the luminance difference between the average luminance calculated in step ST32 and the average luminance calculated in step ST33 and proceeds to step ST35.

[0062] In step ST35, the determination unit determines whether the luminance difference is less than or equal to the determination threshold. The determination unit 33 proceeds to step ST36 in a case where the luminance difference calculated in step ST34 is less than or equal to the determination threshold, and proceeds to step ST37 in a case where the luminance difference is greater than the determination threshold.

[0063] In step ST36, the determination unit determines that the captured monochrome image is the reference image. If the luminance difference is less than or equal to the determination threshold, the fusion image generated using the captured monochrome image as the reference image has a smaller luminance difference between an overlapping image portion and a non-overlapping image portion, and the boundary is not conspicuous. Thus, the determination unit 33 determines that the captured monochrome image is the reference image.

[0064] In step ST37, the determination unit determines that the captured color image is the reference image. If the luminance difference is greater than the determination threshold, the fusion image generated using the captured monochrome image as the reference image has a large luminance difference between the overlapping image area and the non-overlapping image area (frame area), so that the boundary of the area is conspicuous. Thus, the determination unit 33 determines that the captured color image is the reference image.

[0065] As described above, the image processing unit determines whether the captured monochrome image or the captured color image is set as the reference image based on the luminance difference between the captured monochrome image and the captured color image, and goes to step ST14 in Fig. 8 over.

[0066] In step ST14, the image processing unit performs an integration determination. The determination unit 33 of the image processing unit 30 determines which type of fusion processing is being performed based on the setting information acquired in step ST11 and the determination results obtained in steps ST12 and ST13, and generates the fusion determination information.

[0067] Fig. 16 is a diagram to explain the integration determination and (a) of Fig. 16 shows a case where the moving image is generated as the fusion image, and (b) of Fig. 16 shows a case where the still image is generated as a fusion image.

[0068] The determination unit 33 generates the fusion determination information based on (a) of Fig. 16 in a case where it is determined according to the acquired setting information that the user has selected the generation of the moving image, and generates the fusion determination information on the basis of (b) of Fig. 16 in a case where it is determined that the user has selected the generation of the still image.

[0069] When generating the moving image, the angle of view is not changed during imaging. Thus, in a case where the user has selected the moving image generation and a mode (wide-angle mode) for generating the fused image with the wide angle of view, the determination unit 33 sets the angle of view to the wide angle of view and switches the reference image based on the distance determination result and the luminance difference determination result. For example, in a case where it is determined that the monochrome captured image is the reference image in both the distance determination result and the luminance difference determination result, the determination unit 33 sets, for example, fusion processing with a high-quality image reference as the fusion processing, the captured monochrome image as the reference image, and the wide angle as the angle of view.In a case where the captured color image is set as the reference image in the determination results of the distance and / or the determination results of the luminance difference, the determination unit 33 sets the fusion processing with wide-angle image reference as the fusion processing, the captured color image as the reference image, and the wide angle as the angle of view.

[0070] Furthermore, when generating the moving image, in a case where the image quality cannot be allowed to deteriorate by setting the captured color image as the reference image compared to a case where the captured monochrome image is set as the reference image, the determination unit 33 is activated to select an image quality priority mode. Regardless of whether the predetermined condition is met or not, in the image quality priority mode, narrow-angle fusion processing is set as the fusion processing, the captured monochrome image is set as the reference image, and a narrow angle of view (a normal angle, which is an angle of view of the captured monochrome image) is set as the angle of view.When the angle of view is set to the narrow angle of view as described above, the fusion image is not affected by the near object in the near vision detection area and the luminance difference between the overlapping image area and the frame area.

[0071] When generating the still image, the reference image and the angle of view can be adjusted using the analysis result of the captured image to generate the fused image. For example, the still image generation is provided with a mode (wide-angle mode) for generating the fused image with a wide angle of view and a mode (normal-angle mode) for generating the fused image with the angle of view. Furthermore, the user can select a mode (automatic angle mode) for automatically adjusting the angle of view, etc.

[0072] In a case where the user has selected the wide-angle mode, the determination unit 33 sets the wide angle of view as the angle of view and switches the reference image based on the distance determination result and the luminance difference determination result. In a case where both the distance determination result and the luminance difference determination result determine that the captured monochrome image is the reference image, the determination unit 33 sets, for example, high-quality image reference fusion processing as the fusion processing, the captured monochrome image as the reference image, and the wide angle of view as the angle of view.In a case where the captured color image is set as the reference image in the determination results of the distance and / or the determination results of the luminance difference, the determination unit 33 sets the fusion processing with wide-angle image reference as the fusion processing, the captured color image as the reference image, and the wide angle of view as the angle of view.

[0073] In a case where the user has selected the normal angle mode, the determination unit 33 sets the normal angle of view (narrow angle of view) as the angle of view of the fused image. When the angle of view is set to the normal angle of view, the fused image is not affected by the near object in the near-vision detection area and the luminance difference between the overlapping image area and the frame area. Thus, in the case where the user has selected the normal angle mode, the determination unit 33 sets the narrow-angle fusion processing as the fusion processing, the captured monochrome image as the reference image, and the normal angle of view as the angle of view, regardless of whether the predetermined condition is met or not.

[0074] In a case where the user has selected the automatic angle mode, in other words, the mode for automatically adjusting the angle of view of the fused image, the determination unit 33 switches the reference image and adjusts the angle of view based on the distance determination result and the luminance difference determination result. In a case where it is determined that the captured monochrome image is the reference image in both the distance determination result and the luminance difference determination result, the determination unit 33 sets, for example, the high-quality image reference fusion processing as the fusion processing, the captured monochrome image as the reference image, and the wide angle of view as the angle of view.In a case where the captured color image is set as the reference image in the distance determination results and / or the luminance difference determination results, the determination unit 33 sets the narrow-angle fusion processing as the fusion processing, the captured monochrome image as the reference image, and the normal angle of view as the angle of view. As described above, if the automatic angle mode can be selected, the reference image and the angle of view can be automatically adjusted so that the overlapping image area has high image quality.

[0075] The determination unit 33 determines, on the basis of the setting information, the determination result of the distance and the determination of the luminance difference, what kind of fusion processing is performed, generates the fusion determination information indicating the determination result, and proceeds to step ST3 in Fig. 7 continued.

[0076] In step ST3, the image processing unit performs parallax detection. The parallax detection unit 321 of the image processing unit 30 detects a parallax of another captured image with respect to the reference image based on the fusion determination information generated in step ST2. The parallax detection unit 321 detects a parallax for each pixel of the reference image. In a case where the fusion determination information indicates that the reference image is a monochrome image and the image angle is a wide image angle, the frame area interpolation unit 322 performs interpolation processing that generates a parallax of the frame area and sets the parallax of the frame area as the parallax of a boundary portion in the overlapping image area. Fig. Fig. 17 is a diagram for explaining parallax detection. The parallax detection unit 321 detects the parallax of the other captured image with respect to the reference image based on the fusion determination information using the captured monochrome image Gbw-H and the captured color image Gcr-W shown in (a) of Fig. 17. Here, in a case where the fusion determination information indicates that the reference image is the captured monochrome image and the angle of view is the wide angle of view, the parallax of the frame area ARa is not detected as shown in (b) of Fig. 17. Note that in the figure, the detected parallax (the parallax vector) is schematically indicated by arrows. Thus, the frame area interpolation unit 322 uses the parallax of a pixel positioned in the overlapping image area ARb at the boundary with the frame area ARa as the parallax of the frame area ARa, as shown in (c) of Fig. 17 and proceeds to step ST4.

[0077] In step ST4, the image processing unit performs warping. The warping units 323-H and 323-W of the image processing unit 30 perform the warping based on the fusion determination information generated in step ST2 and the parallax detected in step ST3.

[0078] In a case where the fusion determination information indicates that the captured monochrome image is the reference image and the angle of view is the wide angle of view, the warping unit 323-W generates a captured color image, which is a viewpoint of the captured monochrome image, from the captured color image based on a parallax amount detected by the parallax detection unit 321 and a parallax amount of the frame area interpolated by the frame area interpolation unit 322, and outputs the generated image to the image synthesis unit 324.In a case where the fusion determination information indicates that the captured monochrome image is the reference image and the angle of view is the normal angle of view, the distortion unit 323-W generates a captured color image, which is a viewpoint of the captured monochrome image, from an image of the overlapping image area in the captured color image, based on the parallax amount detected by the parallax detection unit, and outputs the generated image to the image synthesis unit 324. Further, in a case where the fusion determination information indicates that the captured monochrome image is the reference image, the distortion unit 323-H outputs the captured monochrome image supplied from the pre-processing unit 31-H to the image synthesis unit 324.

[0079] In a case where the fusion determination information indicates that the captured color image is the reference image, the distortion unit 323-W outputs the captured color image supplied from the preprocessing unit 31-W to the image synthesis unit 324. Further, in a case where the fusion determination information indicates that the captured color image is the reference image, the distortion unit 323-H generates a captured monochrome image, which is a viewpoint of the captured color image, from the captured monochrome image based on the parallax amount detected by the parallax detection unit 321, and outputs the generated image to the image synthesis unit 324.

[0080] The image processing unit 30 performs the warping based on the fusion determination information and the parallax amount, generates the captured image of the viewpoint of the reference image from the captured image different from the reference image, and proceeds to step ST5.

[0081] In step ST5, the image processing unit performs image synthesis. The image synthesis unit 324 of the image processing unit 30 synthesizes the monochrome image supplied by the distortion unit 323-H and a luminance component image in the captured color image supplied by the distortion unit 323-W by aligning the positions of the overlapping image areas. Furthermore, the image processing unit 30 generates and outputs color image data including the color difference data and the luminance data, which are the components equal to the captured monochrome image.It should be noted that in a case where the moving image is generated, in a case where the pipeline processing or parallel processing is not limited to sequence processing that performs the processing in the order of steps, the image processing unit 30 may start generating the fusion image of the next frame before determining that the synthesis processing of all the lines is completed, for example.

[0082] As described above, according to the embodiment of the present invention, by using the plurality of captured images each having different image angles and image qualities, the captured image with the wide image angle and the high image quality can be generated without deteriorating the image quality. Fig. Figure 18 shows the performance of the fusion image generated based on the fusion determination information and fusion image examples. Note that (b) of Fig. 18 represents the color image Gcr-W taken in, and (c) to (e) of Fig. 18 each represent a fusion image GFu. The fusion image generated by the fusion processing with high-quality image reference is shown in (c) of Fig. 18, the fusion image generated by the fusion processing with wide-angle image reference is shown in (d) of Fig. 18, and the fusion image generated by the narrow-angle fusion processing is shown in (e) of Fig. 18 shown.

[0083] In a case where the fusion determination information indicates that fusion processing is performed with high-quality image reference, the fusion image with the wide angle of view is generated using the acquired monochrome image as the reference image. Since the fusion image is generated using the acquired monochrome image as the reference image, the image quality of the overlapping image area is the high image quality (double circle mark). Furthermore, in generating the fusion image, the acquired color image is used for the frame area, so the image quality of the frame area is an image quality equivalent to that of the acquired color image, and the angle of view is the wide angle of view, which is an image angle of the acquired color image.

[0084] In a case where the fusion determination information indicates that fusion processing with wide-angle image reference is performed, the fusion image with the wide image angle is generated using the captured color image as the reference image. Since the fusion image is generated using the captured color image as the reference image, the image quality of the overlapped image area is a lower image quality (circle mark) compared to that of the fusion processing with high-quality image reference. Furthermore, in generating the fusion image, the captured color image is used for the frame area, so the image quality of the frame area is an image quality equivalent to that of the captured color image, and the image angle is the wide image angle, which is an image angle of the captured color image.

[0085] In a case where the fusion determination information indicates that narrow-angle fusion processing is performed, the fusion image with the narrow angle of view (normal angle of view) is generated using the acquired monochrome image as the reference image. Because the fusion image is generated using the acquired monochrome image as the reference image, the image quality of the overlapped image area is a high image quality (double circle mark), similar to that of fusion processing with a high-quality image reference. Since the angle of view is the narrow angle of view, there is no frame area, and the fusion image has the normal angle of view, which is an image angle of the acquired monochrome image. <3. Other embodiments>

[0086] However, the image processing unit is not limited to one having a configuration that performs the determination of the distance and the luminance difference based on the captured image generated during the generation of the moving image, and performs fusion processing of the subsequently generated captured image based on the fusion determination information generated depending on the determination result. For example, the image processing unit may perform the determination of the distance and the luminance difference based on the captured image, and perform fusion processing of the captured image used to determine the distance and the luminance difference based on the fusion determination information generated depending on the determination result.In this case, the determination unit 33 in the image processing unit 30 outputs the fusion determination information to the image synthesis unit 324. Further, the distortion unit 323-H and the distortion unit 323-W each generate the captured monochrome image of the viewpoint of the captured color image and the captured color image of the viewpoint of the captured monochrome image, and output the captured monochrome image and the captured color image, as well as the captured monochrome image and the captured color image after viewpoint conversion, to the image synthesis unit 324. The image synthesis unit 324 generates a fusion image using the captured monochrome image as a reference or a fusion image using the captured color image as a reference based on the fusion determination information from the determination unit 33.By performing such processing, the image processing unit 30 can perform the fusion processing of the captured image used for the determination of the distance and the luminance difference depending on the determination result of the distance and the luminance difference.

[0087] Furthermore, in the above-described embodiment, a case was described where the captured image with the high image quality is the monochrome captured image and the captured image with the wide angle of view is the color captured image; however, the captured images are not limited to those in the above-described embodiment. For example, the captured image with the high image quality may be a color captured image with a high resolution, or a color captured image with a high resolution and a different color component. Furthermore, the captured image with the high image quality may be a captured image formed by an imaging unit having a sensitivity to a wavelength range different from that of the color captured image.For example, an imaging unit with sensitivity to the infrared range is used as an imaging unit with sensitivity to a wavelength range different from that of the captured color image. When the captured monochrome image is captured using the imaging unit with sensitivity to the infrared range as the imaging unit 21-H, a fused image with a wide angle of view can be generated, ensuring that the object positioned in the overlapped image area has high image quality even in the evening, at night, and the like. <4. Application example>

[0088] The technology according to the present disclosure can be applied to various products. The technology according to the present disclosure can be implemented not only as a data processing terminal, but also as a device mounted on any type of mobile body, such as a car, an electric car, a hybrid electric car, a motorcycle, a bicycle, a personal transporter, an aircraft, a drone, a ship, a robot, a construction machine, an agricultural machine (tractor), and the like.

[0089] Fig. 19 is a block diagram illustrating a schematic configuration example of a vehicle control system 7000, which is an example of a mobile body control system to which the technology according to the present disclosure can be applied. The vehicle control system 7000 includes a plurality of electronic control units connected to each other via a communication network 7010. In the Fig. In the example shown in Figure 19, the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, a vehicle exterior information detection unit 7400, a vehicle interior information detection unit 7500, and an integrated control unit 7600. The communication network 7010 connecting these multiple control units may be, for example, an in-vehicle communication network conforming to any standard, such as a controller area network (CAN), a local interconnection network (LIN), a local area network (LAN), or FlexRay (registered trademark).

[0090] Each control unit includes a microcomputer that performs arithmetic processing according to various programs, a storage unit that stores programs executed by the microcomputer, parameters used for various calculations, or the like, and a drive circuit that drives devices to be controlled. Each control unit includes a network interface for communicating with other control units via the communication network 7010 and a communication interface for communicating with devices inside and outside a vehicle, a sensor, or the like via wired communication or wireless communication. Fig. 19 shows, as the functional configuration of the integrated control unit 7600, a microcomputer 7610, a general-purpose communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, an in-vehicle device I / F 7660, an audio image output unit 7670, an in-vehicle network I / F 7680, and a storage unit 7690. Similarly, the other control units each include a microcomputer, a communication I / F, a storage unit, and the like.

[0091] The drive system control unit 7100 controls the operation of devices related to a drive system of a vehicle according to various programs. For example, the drive system control unit 7100 functions as a control device of a driving force generating device for generating a driving force of the vehicle, such as an internal combustion engine or a drive motor, a driving force transmitting mechanism for transmitting a driving force to wheels, a steering mechanism for adjusting a steering angle of the vehicle, a braking device for generating a braking force of the vehicle, and the like. The drive system control unit 7100 may have a function as a control device such as an anti-lock brake system (ABS) or an electronic stability control (ESC).

[0092] The drive system control unit 7100 is connected to a vehicle state detection unit 7110. The vehicle state detection unit 7110 includes, for example, a gyro sensor that detects the angular velocity of the axle rotation of a vehicle body, an acceleration sensor that detects vehicle acceleration, or a sensor for detecting an accelerator pedal operation amount, a brake pedal operation amount, a steering angle of the steering wheel, an engine speed, a wheel speed, or the like. The drive system control unit 7100 performs arithmetic processing using a signal input from the vehicle state detection unit 7110 and controls the internal combustion engine, the drive motor, the electric power steering device, the brake device, or the like.

[0093] The body system control unit 7200 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 7200 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as a headlight, tail lamp, brake lamp, turn signal, and fog lamp. In this case, a radio wave input from a portable device that replaces a key or signals from various switches can be input to the body system control unit 7200. The body system control unit 7200 accepts the input of these radio waves or signals and controls the door lock device, power window device, lamp, and the like of the vehicle.

[0094] The battery control unit 7300 controls a secondary battery 7310, which is a power supply source of the drive motor, according to various programs. For example, information such as a battery temperature, a battery output voltage, or a remaining battery capacity from a battery device including the secondary battery 7310 is input to the battery control unit 7300. The battery control unit 7300 performs arithmetic processing using these signals and performs temperature adjustment control of the secondary battery 7310 or control of a cooling device or the like provided in the battery device.

[0095] The vehicle exterior information detection unit 7400 detects information related to the exterior of the vehicle on which the vehicle control system 7000 is mounted. For example, at least one imaging unit 7410 or a vehicle exterior information detection unit 7420 is connected to the vehicle exterior information detection unit 7400. The imaging unit 7410 includes at least one time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, or other cameras. For example, at least one environmental sensor for detecting the current climate or weather, or an environmental information detection sensor for detecting another vehicle, an obstacle, a pedestrian, or the like in the vicinity of the vehicle on which the vehicle control system 7000 is mounted, is mounted on the vehicle exterior information detection unit 7420.

[0096] The environmental sensor may be, for example, a raindrop sensor that detects rainy weather, a fog sensor that detects fog, a sunshine sensor that detects solar irradiance, and / or a snow sensor that detects snowfall. The environmental information detection sensor may be an ultrasonic sensor, a radar device, and / or a light detection and ranging (LIDAR) device. The imaging unit 7410 and the vehicle exterior information detection unit 7420 may each be provided as independent sensors or devices, or may be provided as a device in which multiple sensors or devices are integrated.

[0097] Here shows Fig. 20 shows an example of installation positions of the imaging unit 7410 and the vehicle exterior information detection unit 7420. Imaging units 7910, 7912, 7914, 7916, and 7918 are provided, for example, at least at one position of a vehicle 7900 under the front nose, the side mirror, the rear bumper, the tailgate, the upper part of the windshield inside the vehicle. The imaging unit 7910 is provided at the front nose, and the imaging unit 7918 is provided at the upper part of the windshield inside the vehicle to mainly capture images in front of the vehicle 7900. The imaging units 7912 and 7914 are provided at the side mirrors to mainly capture images on the sides of the vehicle 7900. The imaging unit 7916 is provided on the rear bumper or the tailgate to mainly capture an image behind the vehicle 7900.The imaging unit 7918 provided at the upper part of the windshield in the vehicle interior is mainly used for detecting a preceding vehicle, a pedestrian, an obstacle, a traffic signal, a road sign, a roadway, or the like.

[0098] It should be noted that Fig. 20 shows an example of imaging ranges of the imaging units 7910, 7912, 7914, and 7916. An imaging range a indicates an imaging range of the imaging unit 7910 provided at the front nose, imaging ranges b and c respectively indicate the imaging ranges of the imaging units 7912 and 7914 provided at the side mirrors, and an imaging range d indicates an imaging range of the imaging unit 7916 provided at the rear bumper or tailgate. For example, the data acquired by the imaging units 7910, 7912, 7914, and 7916 are superimposed, thereby obtaining a top view of the vehicle 7900 viewed from above.

[0099] The vehicle exterior information detection units 7920, 7922, 7924, 7926, 7928, and 7930, which are provided at the front, rear, sides, corners, and upper part of the windshield in the vehicle interior of the vehicle 7900, may be, for example, ultrasonic sensors or radar devices. The vehicle exterior information detection units 7920, 7926, and 7930, which are provided at the front nose, rear bumper, tailgate, and upper part of the windshield in the vehicle interior of the vehicle 7900, may be, for example, LIDAR devices. These vehicle exterior information detection units 7920 to 7930 are mainly used to detect a preceding vehicle, a pedestrian, an obstacle, or the like.

[0100] With further reference to Fig. The description continues with reference to Figure 19. The vehicle exterior information detection unit 7400 causes the imaging unit 7410 to capture an image outside the vehicle and receives the captured image data. Furthermore, the vehicle exterior information detection unit 7400 receives detected information from the connected vehicle exterior information detection unit 7420. In a case where the vehicle exterior information detection unit 7420 is an ultrasonic sensor, a radar device, or a LIDAR device, the vehicle exterior information detection unit 7400 transmits ultrasonic waves, electromagnetic waves, or the like and receives information related to the received reflected waves.The vehicle exterior information detection unit 7400 can perform object detection processing or distance detection processing on a person, a car, an obstacle, a sign, a letter on the road surface, or the like based on the received information. The vehicle exterior information detection unit 7400 can calculate a distance to an object outside the vehicle based on the received information.

[0101] Furthermore, the vehicle exterior information detection unit 7400 may perform distance detection processing or image recognition processing for detecting a person, a car, an obstacle, a character, a letter on the road surface, or the like based on the received image data. The vehicle exterior information detection unit 7400 may perform processing such as distortion correction or alignment on the received image data and synthesize the image data captured by various imaging units 7410 to generate a plan view or a panoramic image. The vehicle exterior information detection unit 7400 may perform image angle conversion processing using the image data captured by the various imaging units 7410.

[0102] The vehicle interior information detection unit 7500 detects information related to the vehicle interior. The vehicle interior information detection unit 7500 is connected, for example, to a driver state detection unit 7510 that detects a state of a driver. The driver state detection unit 7510 may include a camera that captures an image of the driver, a biometric sensor that detects biological information of the driver, a microphone that collects sounds in the vehicle interior, and the like. The biometric sensor is provided, for example, on a seat surface, a steering wheel, or the like, and detects biological information of an occupant sitting on a seat or a driver holding the steering wheel.The vehicle interior information detection unit 7500 can calculate a driver's fatigue level or concentration level based on the detected information input from the driver state detection unit 7510 and can determine whether the driver is asleep or not. The vehicle interior information detection unit 7500 can perform noise suppression processing or the like on a collected sound signal.

[0103] The integrated control unit 7600 controls the overall operation in the vehicle control system 7000 according to various programs. The integrated control unit 7600 is connected to an input unit 7800. The input unit 7800 is implemented by a device such as a touch panel, a button, a microphone, a switch, a lever, or the like, on which an input operation can be performed by the occupant. Data obtained by performing voice recognition on the sound input through the microphone can be input to the integrated control unit 7600. The input unit 7800 may be, for example, a remote control device using infrared rays or other radio waves, or an external connection device such as a mobile phone or a personal digital assistant (PDA) adaptable to the operation of the vehicle control system 7000.The input unit 7800 may be, for example, a camera, in which case the occupant can input information through gestures. Alternatively, data obtained by detecting a movement of a portable device worn by the occupant may be input. Furthermore, the input unit 7800 may include, for example, an input control circuit or the like that generates an input signal based on the information or the like input by the occupant using the input unit 7800 and outputs the input signal to the integrated control unit 7600. By operating the input unit 7800, the occupant or the like inputs various data to the vehicle control system 7000 or issues a command to perform the processing operation.

[0104] The storage unit 7690 may include a read-only memory (ROM) that stores various programs executed by the microcomputer, and a random access memory (RAM) that stores various parameters, calculation results, sensor values, or the like. Furthermore, the storage unit 7690 may be implemented by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, an optical magnetic storage device, or the like.

[0105] The general-purpose communication I / F 7620 is a general-purpose communication I / F that facilitates communication with various devices present in an outdoor environment 7750. The general-purpose communication I / F 7620 can implement a cellular-based communication protocol such as Global System for Mobile Communications (GSM), WiMAX, Long Term Evolution (LTE), or Advanced LTE (LTE-A), or other wireless communication protocols such as wireless LAN (also referred to as WiFi (registered trademark) and Bluetooth (registered trademark)). For example, the general-purpose communication I / F 7620 can be connected to a device (e.g., an application server or a control server) present in an external network (e.g., the Internet, a cloud network, or an enterprise-specific network) via a base station or an access point. Furthermore, the general-purpose communication I / F 7620 can be connected to a terminal device present near the vehicle (e.g., a mobile phone).B. a terminal of a driver, a pedestrian, or a shop or a terminal for machine-type communication (MTC terminal), for example using a peer-to-peer technology (P2P technology).

[0106] The 7630 Dedicated Communications I / F is a communication I / F that supports a communication protocol designed for use in vehicles. For example, the 7630 Dedicated Communications I / F can implement a standard protocol such as Wireless Access in Vehicular Environments (WAVE), which is a combination of the lower layer IEEE 802.11p and the upper layer IEEE 1609, dedicated short-range communication (DSRC), or a cellular-based communication protocol. The 7630 Dedicated Communications I / F typically performs V2X communication, which is a concept that includes vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and / or vehicle-to-pedestrian communication.

[0107] For example, the positioning unit 7640 receives a global navigation satellite system (GNSS) signal (e.g., a global positioning system (GPS) signal from a GPS satellite) from a GNSS satellite to perform positioning, and generates position information including the latitude, longitude, and altitude position of the vehicle. Note that the positioning unit 7640 may specify the current position by exchanging signals with a wireless access point or may obtain the position information from a terminal device such as a mobile phone, a PHS, or a smartphone with a positioning function.

[0108] The beacon receiving unit 7650 receives radio waves or electromagnetic waves transmitted from a wireless station or the like, for example, installed along a road, and acquires information such as the current position, traffic jams, road closures, and required time. Note that the function of the beacon receiving unit 7650 can be included in the dedicated communication I / F 7630 described above.

[0109] The in-vehicle device I / F 7660 is a communication interface that mediates a connection between the microcomputer 7610 and various in-vehicle devices 7760 present in the vehicle. The in-vehicle device I / F 7660 can establish a wireless connection using a wireless communication protocol such as a wireless LAN, Bluetooth (registered trademark), near-field communication (NFC), or wireless USB (WUSB). Furthermore, the in-vehicle device I / F 7660 can establish a wired connection such as a wireless universal bus (USB), a high-definition multimedia interface (HDMI), or a mobile high-definition connection (MHL) via a connection port (and a cable, if applicable) not shown.The in-vehicle device 7760 may be, for example, a mobile device and / or a portable device possessed by the occupant, or an information device carried in or attached to the vehicle. Furthermore, the in-vehicle device 7760 may include a navigation device that performs a route search to an arbitrary destination. The in-vehicle device I / F 7660 exchanges control signals or data signals with these in-vehicle devices 7760.

[0110] The in-vehicle network interface 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network interface 7680 sends and receives signals and the like according to a predetermined protocol supported by the communication network 7010.

[0111] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs based on information acquired through the universal communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and / or the in-vehicle network I / F 7680. For example, the microcomputer 7610 can calculate a control target value of the driving force generating device, the steering mechanism, or the braking device based on information acquired inside and outside the vehicle and output a control command to the drive system control unit 7100.For example, the microcomputer 7610 can perform cooperative control for implementing advanced driver assistance system (ADAS) functions, including vehicle collision avoidance or shock mitigation, vehicle distance-based following, vehicle speed maintenance, vehicle collision warning, vehicle lane departure warning, or the like. Furthermore, the microcomputer 7610 can perform cooperative control aimed at automatic driving or the like, which performs autonomous driving independent of the driver's operation by controlling the driving force generation device, steering mechanism, braking device, or the like based on detected information around the vehicle.

[0112] The microcomputer 7610 can generate three-dimensional distance information between the vehicle and an object such as a surrounding structure or a person based on information obtained via the general-purpose communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, or the in-vehicle network I / F 7680, and generate local map information including environmental information of the current position of the vehicle. Furthermore, based on the acquired information, the microcomputer 7610 can predict a hazard such as a collision of a vehicle, an approach of a pedestrian, or the like, or entering a closed road, and generate a warning signal. The warning signal may be, for example, a signal for generating a warning sound or turning on a warning light.

[0113] The audio-image output unit 7670 sends an output signal of audio and / or image to an output device that can visually or audibly inform an occupant inside or outside the vehicle of information. In the example of Fig. 19, an audio speaker 7710, a display unit 7720, and an instrument panel 7730 are shown as output devices. The display unit 7720 may include, for example, an on-board display and / or a head-up display. The display unit 7720 may have an augmented reality (AR) display function. The output device may be a device other than these devices, for example, a lamp, a projector, or a wearable device such as a headset and a display glasses worn by the occupant. In a case where the output device is a display device, the display device visually displays results obtained by the various processing performed by the microcomputer 7610 or information received from the other control units in various formats such as text, image, table, or diagram.In a case where the output device is an audio output device, the audio output device converts an audio signal including reproduced audio data, acoustic data, and the like into an analog signal to acoustically output the analog signal.

[0114] It should be noted that in the Fig. 19, at least two control units connected to each other via the communication network 7010 may be integrated into one control unit. Alternatively, each control unit may be formed by a plurality of control units. Furthermore, the vehicle control system 7000 may include another control unit not shown. Furthermore, in the above description, some or all of the functions performed by one of the control units may be performed by another control unit. That is, as long as information is sent and received via the communication network 7010, predetermined computing processing may be performed by any of the control units. Likewise, a sensor or device connected to one of the control units may be connected to another control unit, and a plurality of control units may mutually send and receive information acquired via the communication network 7010.

[0115] In the vehicle control system 7000 described above, the mapping units 7410, 7910, 7912, 7914, 7916 and 7918 are designed to use a plurality of mapping units, for example, those shown in Fig. 2 illustrated imaging units 21-H and 21-W. Furthermore, the image processing unit 30 is in the integrated control unit 7600 of the Fig. 19 is provided. With such a configuration, even if the imaging units 7410, 7910, 7912, 7914, 7916, and 7918 are downsized and reduced in thickness, the captured image can be captured with high image quality and a wide angle of view, so that the captured captured image can be used for driving assistance, driving control, and the like. It should be noted that the image processing unit 30 is incorporated in a module (e.g., a single-chip integrated circuit module) for the Fig.19 shown integrated control unit 7600 can be implemented.

[0116] The sequence of processing steps described in the specification can be executed by hardware, software, or a combination of both. In a case where the processing is executed by software, a program recording a processing sequence is installed and executed in a memory in a computer integrated with dedicated hardware. Alternatively, the program can be installed and executed in a general-purpose computer capable of performing various types of processing.

[0117] For example, the program can be pre-recorded in a read-only memory (ROM), a solid-state drive (SSD), or a hard disk as the recording medium. Alternatively, the program can be temporarily or permanently stored (recorded) in a removable recording medium, such as a flexible disk, a compact disc (CD-ROM), a magneto-optical disk (MO disk), a DVD, a Blu-ray Disc (registered trademark), a magnetic disk, or a semiconductor memory card. Such a removable recording medium can be provided as so-called packaged software.

[0118] In addition to installing the program from the removable recording medium on the computer, it can also be transferred wirelessly or wired from a download location to the computer via a network, such as a local area network (LAN) or the Internet. The computer can receive the program transferred in this way and install the program on a recording medium, such as a built-in hard disk.

[0119] It should be noted that the advantageous effects described in this specification are merely examples, and the advantageous effects of the present technology are not limited to these examples and may include additional effects not described herein. Furthermore, the present technology should not be considered limited to the above-described embodiments of the technology. The embodiments of the present invention, as defined in the claims, disclose the present technology by way of examples, and it should be obvious that those skilled in the art can modify or replace these embodiments with other embodiments without departing from the scope of the technology. Industrial applicability

[0120] In the image processing device and method of this technology, warping is performed using the high-quality image as a reference on the wide-angle image with lower image quality and wider image angle than the high-quality image, and processing that synthesizes the high-quality image with higher image quality than the wide-angle image and the wide-angle image after warping by aligning the positions of the overlapping image areas to each other, thereby generating the fused image. Therefore, the captured image with a wide image angle and high image quality can be generated using multiple captured images each with different image angles and image qualities without deteriorating the image quality.Thus, this technology is suitable for a device that uses an imaging unit and requires downsizing and reducing the thickness of the imaging unit. List of reference symbols 10 Data processing terminal 21-H, 21-W imaging unit 30 Image processing unit 31-H, 31-W preprocessing unit 32 Fusion Processing Unit 33 Determination unit 51 Sensor unit 52 Communication unit 53 Display unit 54 Touch field 55 Control unit 56 storage unit 60 control unit 321 Parallax detection unit 322 Frame area interpolation unit 323-H, 323-W distortion unit 324 Image synthesis unit 331 distance measuring unit 332 Distance determination unit 333 Luminance difference determination unit 334 Integration determination unit

Claims

[1] Image processing device comprising: a fusion processing unit (32) which, with a high-quality image as a reference, performs warping on a wide-angle image having a lower image quality and a wider image angle than the high-quality image, wherein the fusion processing unit (32) performs fusion processing with a high-quality image reference, which generates a fusion image by aligning positions of overlapping image areas of the high-quality image and the wide-angle image after warping with each other; wherein the fusion processing unit (32) performs the warping based on parallax information indicating a parallax between the high-quality image and the wide-angle image, and the fusion processing unit (32) interpolates parallax information in a non-overlapping image area not included in the high-quality image in the wide-angle image using parallax information about the overlapping image area. [2] The image processing apparatus according to claim 1, wherein the fusion processing unit (32) performs warping with the wide-angle image as a reference using the high-quality image, and performs wide-angle image reference fusion processing that generates a fusion image by aligning positions of the overlapping image portions of the high-quality image after warping and the wide-angle image on which warping is not performed, or performs the high-quality image reference fusion processing. [3] An image processing apparatus according to claim 2, further comprising: a determination unit (33) which determines whether the predetermined condition is met or not, wherein the fusion processing unit (32) performs the wide-angle image reference fusion processing in a case where the predetermined condition is not satisfied, and the high-quality image reference fusion processing in a case where the predetermined condition is satisfied. [4] The image processing apparatus according to claim 3, wherein the determination unit (32) sets as a predetermined condition that an object distance determined on the basis of parallax information is greater than a determination threshold. [5] The image processing apparatus according to claim 4, wherein the object distance is an object distance of an object positioned in a non-overlapping image area not included in the high-quality image in the wide-angle image. [6] The image processing apparatus according to claim 3, wherein the determination unit (32) sets as a predetermined condition that a luminance difference between the high-quality image and the wide-angle image is less than or equal to a determination threshold. [7] An image processing apparatus according to claim 6, wherein the luminance difference is a difference between average luminances of an image area within a predetermined distance from a boundary with the non-overlapping image area in the overlapping image area. [8] The image processing apparatus according to claim 3, wherein the fusion processing unit (33) performs warping with the high-quality image as a reference using the wide-angle image, and depending on a determination result of the determination unit (32), performs narrow-angle fusion processing that generates a fusion image having an image angle of the high-quality image by adjusting positions of the overlapping image areas of the high-quality image and the wide-angle image after warping. [9] The image processing apparatus according to claim 8, wherein, in a case where the high-quality image and the wide-angle image are moving images and a fused image having an image angle of the wide-angle image is generated, the fusion processing unit (33) performs the high-quality image reference fusion processing or the wide-angle image reference fusion processing based on the determination result of the determination unit. [10] The image processing apparatus according to claim 8, wherein the fusion processing unit (33) performs the high-quality image reference fusion processing, the wide-angle image reference fusion processing, or the narrow-angle fusion processing in a case where the high-quality image and the wide-angle image are still images. [11] The image processing apparatus according to claim 10, wherein the fusion processing unit (33) performs the wide-angle image reference fusion processing or the narrow-angle fusion processing in a case where the predetermined condition is not satisfied. [12] The image processing apparatus according to claim 11, wherein the fusion processing unit (33) performs the narrow-angle fusion processing in a case where it is determined by the determination unit in a case where the predetermined condition is not satisfied that the mode selection for automatically adjusting an image angle of a fusion image is performed. [13] Image processing apparatus according to claim 8, wherein the determination unit (32) determines a user selection operation of the fusion processing, and the fusion processing unit (33) performs the narrow-angle fusion processing in a case where it is determined by the determination unit that the narrow-angle fusion processing is selected, regardless of whether the predetermined condition is satisfied or not. [14] Image processing method comprising: Performing warping with a high-quality image as a reference on a wide-angle image having a lower image quality and a wider image angle than the high-quality image, and performing fusion processing with a high-quality image reference that generates a fusion image by aligning positions of overlapping image areas of the high-quality image and the wide-angle image after warping with each other, by a fusion processing unit; wherein the warping is performed based on parallax information indicating a parallax between the high-quality image and the wide-angle image; and wherein parallax information in a non-overlapping image area not included in the high-quality image in the wide-angle image is interpolated using parallax information about the overlapping image area.

Images