COMPLETE EYE IMAGE DEVICE, IMAGE PROCESSING METHOD, PROGRAM AND RECORDING MEDIUM

Abstract

A compound eye image recording device comprising (100): a camera device (1) comprising: an image capture element (11) comprising several image capture areas (15a, 15b, ...); several lenses (12a, 12b, ...) arranged to correspond to the several image acquisition areas (15a, 15b, ...) and configured to capture images of the same field of view onto the corresponding image acquisition areas (15a, 15b, ...); and several optical filters (13a, 13b, ...), wherein the multiple image capture areas (15a, 15b, ...) include at least one image capture area (15a) of a first type and multiple image capture areas of a second type (15b, 15c, ...) which are of the same size and smaller than the at least one image capture area (15a) of the first type, wherein the multiple lenses (12a, 12b, ...) have a focal length such that lenses corresponding to a larger image capture area have a longer focal length, and wherein the multiple optical filters (13a, 13b, ...) are provided such that an image captured in each of the image acquisition areas (15b, 15c, ...) of the second type represents a type of information that is different from that of an image captured in each of the at least one image acquisition area (15a) of the first type; and a processor (20) comprising at least one resolution enhancement unit (30a), wherein the at least one resolution enhancement unit (30a) comprises: a first filter processor (311) for receiving, as a low-resolution image (D1), the image captured in one of the at least one image acquisition area (15b, 15c,...) of the second type, and for extracting, from the low-resolution image (D1), a low-frequency component (D1L) and a high-frequency component (D1H) of the low-resolution image (D1); a second filter processor (312) for receiving, as a reference image (D0), the image which is captured in one of the at least one image acquisition area (15a) of the first type, for extracting, from the reference image (D0), a low-frequency component and a high-frequency component of the reference image (D0); a low-frequency component combiner (313) to enlarge the low-resolution low-frequency component (D1L) of the low-resolution image (D1) to the same resolution as the reference image (D0) and to combine the enlarged low-frequency component and the low-frequency component (D0L) of the reference image (D0) by weighted addition to produce a combined low-frequency component (D313); a high-frequency component combiner (314) to enlarge the high-frequency component (D1H) of the low-resolution image (D1) to the same resolution as the reference image (D0) and to combine the enlarged high-frequency component and the high-frequency component (D0H) of the reference image (D0) by weighted addition to produce a combined high-frequency component (D314); and a component combiner (315) to combine the combined low frequency component (D313) and the combined high frequency component (D314) to produce a high resolution image (D30).

Description

Technical field

[0001] The present invention relates to a camera device, a compound eye image acquisition device and an image processing method.

[0002] In particular, the present invention relates to a camera device for capturing images of the same field of view in multiple image acquisition areas and for capturing multiple images representing different types of information in the multiple image acquisition areas. The present invention further relates to a compound eye image acquisition device that includes a processor for improving the resolution of the multiple images captured by the aforementioned camera device. The present invention further relates to an image processing method implemented by the aforementioned compound eye image acquisition device.

[0003] The present invention further relates to a program for causing a computer to perform the processing of the aforementioned compound eye image recording device or the image processing method, and a recording medium for storing the program. State of the art

[0004] In recent years, the demands placed on image capture devices have become increasingly diverse, with a growing desire to capture not only a visible RGB image but also additional information. Near-infrared light, in particular, is suitable for surveillance, object detection, and similar applications. It is gaining traction in the fields of surveillance cameras, in-vehicle cameras, and similar devices due to its higher transmission compared to night sky illumination and its lack of visibility. Furthermore, images obtained solely from light with a specific polarization direction are useful for eliminating light reflected from windows, road surfaces, and the like, and for detecting poorly visible objects such as...black objects or transparent objects, and are of particular interest in the fields of in-vehicle cameras and inspection cameras used in factory automation (FA).

[0005] Traditionally, such different pieces of information have usually been captured instead of normal color images. The simplest means of simultaneously capturing a color image and different information is a camera array containing multiple cameras; however, this presents problems because the cameras must be precisely positioned, the apparatus becomes larger, and installation and maintenance costs are increased.

[0006] Furthermore, RGB-X sensors, i.e., sensors equipped with filters that only allow near-infrared light to pass through, or with other elements in color filter arrangements, and thus capable of simultaneously capturing different types of information, have recently emerged. However, such sensors require considerable effort and time for design and development and present many manufacturing challenges.

[0007] As a small device to solve these problems, a device has been proposed which has an image acquisition element divided into several image acquisition areas and which captures respective images on the several image acquisition areas (see patent reference 1). This device is equipped with different optical filters for the several image acquisition areas and is therefore capable of simultaneously capturing different information. The device from patent reference 1 has the advantage that it can be manufactured by placing a lens on a single image acquisition element and can be easily miniaturized. List of prior art patent literature

[0008] Patent literature 1: Japanese patent application JP 2001 - 61 109 A (sections 0061 and 0064)

[0009] Further state of the art exists as follows: US 2012 / 0281072A1 discloses a method and devices for capturing and reproducing images with focused plenoptic cameras, in which different filters are used on different microlenses.

[0010] EP 3 171587 A1 discloses a faceted camera imaging device comprising: a plurality of faceted optics designed to be arranged in a two-dimensional configuration opposite an object; a picture element configured to comprise – subdivided into several facets – a plurality of pixels which receive the light focused by the plurality of faceted optics and generate image signals, wherein spatial pixel displacement is carried out by shifting the optical axes of the multiple faceted optics, and wherein the optical axes of the facets with green filters are shifted in opposite directions, while the optical axes of the facets with a red filter and a blue filter are shifted in the same direction; and a signal processing unit configured to generate an image corresponding to the object based on the image signals generated by the picture element.

[0011] WO 2006 / 039 486 A2 teaches an imaging device that acquires light data corresponding to the light passing through a specific focal plane. The light data is acquired using a method that allows the determination of the direction from which different components of the light incident on a specific area of ​​the focal plane originate. Using this directional information in conjunction with the brightness value of the light, as detected by photosensors, an image represented by the light is to be selectively focused and / or corrected.

[0012] WO 2013 / 043751A1 teaches the use of pixel apertures to control the degree of aliasing in captured images of a scene. One embodiment comprises a plurality of focal planes, a control circuit configured to manage the capture of image information by the pixels within the focal planes, and a sampling circuit configured to convert pixel outputs into digital pixel data. Furthermore, the pixels in the plurality of focal planes comprise a pixel stack containing a microlens and an active area, wherein light incident on the surface of the microlens is focused by the microlens onto the active area, and the active area samples the incident light to capture image information. The pixel stack defines a pixel area and includes a pixel aperture, the size of which is smaller than the pixel area. Summary of the invention: Technical problem

[0013] Although the device described in patent literature 1 allows the subdivision of the image capture element to simultaneously capture multiple images from a single image capture element, it suffers from the problem that increasing the number of image capture areas reduces the number of pixels per image capture area, i.e., the resolution.

[0014] The present invention has been made to solve the above problems and is intended to provide a compound eye camera capable of capturing different types of information and capturing high-priority information with high resolution. Solution to the task

[0015] This problem is solved by the subject matter of the attached independent claims. Advantageous further developments are specified in the pending claims. Advantageous effects of the invention.

[0016] With the camera device of the present invention, it is possible, since the camera device contains the multiple image acquisition areas, to capture images representing different types of information. Furthermore, since the multiple image acquisition areas comprise at least one image acquisition area of ​​the first type and multiple image acquisition areas of the second type, which have smaller areas and a smaller number of pixels than the image acquisition area of ​​the first type, it is possible to capture such an image at high resolution by allocating a relatively large image acquisition area to high-priority information, i.e., an image for which it is more desirable to be captured at high resolution, among the different types of information.

[0017] With the compound eye image acquisition device of the present invention, since the resolution enhancement unit improves the resolution of the low-resolution image, even if the image acquisition area is relatively small, it is possible to obtain an image that has a high resolution. Brief description of the drawings Fig. 1A is a perspective exploded view showing a configuration of a compound eye camera forming a camera device according to a first embodiment of the present invention, and Fig. Figure 1B is a diagram showing the sizes of images captured by the aforementioned compound eye camera. Fig. 2A to 2D are schematic diagrams that illustrate different examples of how an image acquisition area of ​​an image acquisition element of the compound eye camera of Fig. 1A is to be subdivided. Fig. 3A and Fig. Figure 3B are schematic diagrams that illustrate different examples of how the image acquisition area of ​​the image acquisition element of the compound eye camera of Fig. 1A is to be subdivided. Fig. Figure 4 is a block diagram illustrating a compound eye image acquisition device according to a second embodiment of the present invention. Fig. Figure 5 is a block diagram that illustrates an example of a resolution enhancement unit used in the second embodiment. Fig. Figure 6 is a block diagram that provides another example of the resolution enhancement unit used in the second embodiment. Fig. Figure 7 is a block diagram that shows another example of a processor used in the second embodiment. Fig. Figure 8 is a block diagram that shows another example of the processor used in the second embodiment. Fig. Figure 9 is a block diagram that illustrates an example of a processor used in a third embodiment of the present invention. Fig. Figure 10 is a block diagram that provides an example of a combiner used in the third embodiment. Fig. Figure 11 is a block diagram that provides another example of the processor used in the third embodiment. Fig. Figure 12 is a block diagram representing a processor used in a fourth embodiment of the present invention. Fig. Figure 13 is a diagram that illustrates an example of interpolation of image information by the processor in the fourth embodiment. Fig. Figure 14 is a block diagram representing a processor used in a fifth embodiment of the present invention. Fig. Figure 15 is a block diagram illustrating a compound eye image acquisition device according to a sixth embodiment of the present invention. Fig. Figure 16 is a perspective exploded view showing configurations of a compound eye camera and a monocular camera according to the sixth embodiment. Fig. 17A and Fig. Figures 17B are each a schematic diagram illustrating an example of how an image acquisition area of ​​an image acquisition element of the compound eye camera of Fig. It is to be divided into 16 parts. Fig. Figure 18 is a flowchart representing a procedure for processing an image processing method according to a seventh embodiment of the present invention. Description of embodiments: First embodiment

[0018] Fig. Figure 1A is a perspective exploded view schematically representing a camera device 1 according to a first embodiment of the present invention.

[0019] The in Fig. 1A The camera device 1 shown contains a compound eye camera 10. The compound eye camera 10 includes an image acquisition element 11, a lens assembly 12, a filter assembly 13 and a partition 14.

[0020] The image capture element 11 has a rectangular image capture area 11a, and the image capture area 11a is subdivided, for example, into several rectangular image capture areas 15a to 15f, as shown in Fig. 2A is shown.

[0021] The lens arrangement 12 contains several lenses 12a, 12b, ..., which are arranged to correspond to the respective image acquisition areas 15a, 15b, ... . The several lenses 12a, 12b, ... form a lens group.

[0022] The lenses 12a, 12b, ... are configured so that images of the same field of view are projected onto the respective image acquisition areas 15a, 15b, ... .

[0023] To project images of the same field of view onto image capture areas that have different sizes, lenses corresponding to larger image capture areas, for example, have longer focal lengths.

[0024] The filter arrangement 13 contains the optical filters 13a, 13b, ..., which are intended for one or more of the several image acquisition areas.

[0025] The partition 14 is provided between the image acquisition areas 15a, 15b, ... and prevents each image acquisition area from receiving light from the other lenses that are not the corresponding lens.

[0026] The image acquisition element 11 is preferably a sensor having a CMOS or CCD structure that allows the received image signals to be read out pixel by pixel. Due to the subdivision, an image acquisition element of the global shutter type (simultaneous exposure and simultaneous readout), which does not cause image blurring, is preferable.

[0027] Of the image capture areas, the largest image capture area(s), in particular image capture area 15a, which has the largest number of pixels, is / are designated as the high-resolution image capture area(s), and the other image capture areas 15b, 15c, ... are designated as low-resolution image capture areas. In the Fig. In the example shown in 2A, the image capture area 11a is square (the number of pixels in a vertical direction and the number of pixels in a horizontal direction are equal), the image capture areas 15a to 15f are also square, the image capture areas 15b to 15f are the same size and therefore have the same number of pixels, and the image capture area 15a has dimensions in the vertical and horizontal directions that are twice those of each of the image capture areas 15b to 15f.

[0028] Since the number of pixels in the vertical and horizontal directions of the high-resolution image capture area 15a is twice that of each image from image capture areas 15b, 15c, ..., the size (number of pixels) of an image (high-resolution image) D0 captured in the high-resolution image capture area is four times that of an image (low-resolution image) D1 captured in each low-resolution image capture area, as shown in Fig. 1B is shown.

[0029] As described above, the resolution of the high-resolution image capture area differs from that of the low-resolution image capture areas, with the former exhibiting a higher resolution. To distinguish between them, the former can be referred to as a first-type image capture area, and the latter as a second-type image capture area.

[0030] The optical filters 13a, 13b, ..., which form the filter arrangement 13, contain optical filters that have different optical properties, so that different types of information (images representing different types of information) are captured from the respective image acquisition areas.

[0031] For example, the optical filters 13a, 13b, ..., which have different optical properties, are provided for one or more of the image acquisition areas 15a, 15b, ... so that images representing different types of information are captured in the respective image acquisition areas.

[0032] The optical filters, which have different optical properties, include, for example, at least one spectral filter, one polarizing filter and one neutral density filter (ND filter), and by using them, images resulting from light with different wavelength ranges, images resulting from light with different polarization directions, or images resulting from image capture with different exposure levels are captured in the respective image capture areas.

[0033] These optical filters can be used alone or in combination.

[0034] The “images resulting from light with different wavelength ranges” described above refer to images obtained by photoelectric conversion of light in specific wavelength ranges, and the “images resulting from light with different polarization directions” refer to images obtained by photoelectric conversion of light that has specific polarization directions.

[0035] For example, it is possible to equip the high-resolution image acquisition area 15a with a high-transmittance G (green) transmission filter, an infrared cut filter, or an optical filter of a complementary color system, or it is possible not to provide an optical filter for the high-resolution image acquisition area 15a (to set the high-resolution image acquisition area 15a to be a monochrome area).

[0036] Optical filters of a complementary color system generally exhibit high optical transmission and are preferable in this respect.

[0037] Here, "not providing an optical filter" means that no optical filter intended to capture different types of images is provided, and an optical filter for another purpose may be provided.

[0038] Furthermore, depending on the lens design, in some cases the free aperture of a lens intended for high-resolution image capture is larger than that of the low-resolution image capture areas, resulting in a large exposure value in the high-resolution image capture area. Providing an optical filter, such as an ND filter, which reduces the amount of transmitted light, for the high-resolution image capture area can prevent large differences in exposure value between the high-resolution and low-resolution image capture areas, thus avoiding excessive exposure in the high-resolution area and insufficient exposure in the low-resolution area.

[0039] The method for dividing the image acquisition area 11a into image acquisition areas is not based on the example of Fig. 2A limited.

[0040] The larger the image capture area, the higher the resolution of the captured image. However, increasing the number of image capture areas, increasing the number of filters with different optical properties, or increasing the number of different combinations of filters with different optical properties, thereby increases the number of types of information that can be captured by the image capture element 11.

[0041] Fig. 2B to 2D and Fig. 3A and Fig. 3B represent subdivision methods that differ from those of Fig. 2A are different.

[0042] In the example of Fig. In 2B, the image acquisition area 11a is rectangular, with a vertical to horizontal dimension ratio of 3:4, and is divided into one high-resolution image acquisition area 15a and three low-resolution image acquisition areas 15b to 15d. These image acquisition areas 15a to 15d are each square. The low-resolution image acquisition areas 15b to 15d are arranged vertically in a strip-shaped section that occupies the left quarter of the image acquisition area 11a, and the high-resolution image acquisition area 15a is located in the remaining section.

[0043] A low-resolution center point of the image acquisition area 15c, located vertically, and a high-resolution center point of the image acquisition area 15a are horizontally aligned. This arrangement allows depth information to be obtained by performing stereo matching using parallax shift between the image captured in the low-resolution image acquisition area 15c and the image captured in the high-resolution image acquisition area 15a.

[0044] In the examples of Fig. 2C and Fig. 2D and Fig. 3A and Fig. 3B are the image capture area 11a and the image capture areas 15a, 15b, ... each square.

[0045] In the example of Fig. 2C, the image acquisition area 11a is subdivided into a high-resolution image acquisition area 15a and seven low-resolution image acquisition areas 15b to 15h. Specifically, image acquisition areas 15b to 15h are arranged in a strip-shaped section occupying the left quarter of the image acquisition area 11a and a strip-shaped section occupying the lower quarter of the image acquisition area 11a, and the high-resolution image acquisition area 15a is arranged in the remaining section.

[0046] The image capture area 11a of Fig. 2C is larger than the image acquisition area 11a of Fig. 2A, and the high-resolution image capture area 15a of Fig. 2C is larger than the high-resolution image capture area 15a. Fig. 2A and is capable of capturing a higher-resolution image D0. Furthermore, the center of the low-resolution image acquisition area 15c, located second from the top in the left strip-shaped section, and the center of the high-resolution image acquisition area 15a are horizontally aligned; the center of the low-resolution image acquisition area 15g, located second from the right in the lower strip-shaped section, and the center of the high-resolution image acquisition area 15a are vertically aligned; and it is possible to accurately obtain depth information in both the horizontal and vertical directions by performing stereo matching in multiple views.

[0047] In the example of Fig. In 2D, the image acquisition area 11a is subdivided into a high-resolution image acquisition area 15a and twelve low-resolution image acquisition areas 15b to 15m. Specifically, the low-resolution image acquisition areas 15b to 15m are arranged vertically in the following order: a strip-shaped section occupying the left quarter of the image acquisition area 11a, a strip-shaped section occupying the lower quarter of the image acquisition area 11a, a strip-shaped section occupying the right quarter of the image acquisition area 11a, and a strip-shaped section occupying the upper quarter of the image acquisition areas 11a. The high-resolution image acquisition area 15a is located in the remaining section.

[0048] If the image capture areas 11a in the examples of the Fig. 2C and Fig. 2D images of the same size have a high resolution of 15a. Fig. 2D smaller than the image capture area 15a with high resolution of Fig. 2C. However, a larger number of low-resolution image acquisition areas 15b to 15m are provided, and a larger number of images D1, containing different types of information, can be captured.

[0049] In the example of Fig. 3A contains the image acquisition areas, including several high-resolution and low-resolution image acquisition areas. Specifically, image acquisition area 11a is divided into three high-resolution image acquisition areas 15a to 15c and four low-resolution image acquisition areas 15d to 15g.

[0050] With this subdivision, the image acquisition area 11a is divided into two parts both vertically and horizontally, and the high-resolution image acquisition areas 15a to 15c are located in the upper left quarter, the upper right quarter, and the lower right quarter. Furthermore, the lower left quarter is divided into two parts both vertically and horizontally, and the respective subdivided areas form the four low-resolution image acquisition areas 15d to 15g.

[0051] The example of Fig. 3A can be used, for example, to capture RGB information, which is primary color information, at high resolution in image acquisition ranges 15a to 15c, and to capture wavelength information in other narrow bands or polarization information at low resolution in image acquisition ranges 15d to 15g. This makes it possible to capture images with more natural color information at high resolution. RGB information can be captured at high resolution in image acquisition ranges 15a to 15c by providing an R transmission filter, a G transmission filter, and a B transmission filter for image acquisition ranges 15a to 15c at high resolution.Similarly, wavelength information in a narrow band can be acquired by providing a narrowband transmission filter, and polarization information in a specific direction can be acquired by providing an optical filter that transmits one polarization component in that direction and attenuates polarization components in the other directions.

[0052] In the example of Fig. In addition to a high-resolution image acquisition area 15a, 3B contains two types of image acquisition areas of different sizes. Specifically, each of the image acquisition areas 15j to 15y of a second group has dimensions in the vertical and horizontal directions that are half those of the image acquisition areas 15b to 15i of a first group.

[0053] For the purpose of differentiation, the image acquisition areas 15b to 15i of the first group can be referred to as image acquisition areas with intermediate resolution, and the image acquisition areas 15j to 15y of the second group can be referred to as image acquisition areas with low resolution.

[0054] Furthermore, since the image acquisition areas 15b to 15i of the first group and the image acquisition areas 15j to 15y of the second group have a lower resolution than the high-resolution image acquisition area 15a, they can be collectively referred to as low-resolution image acquisition areas.

[0055] The arrangement of the image capture areas in the configuration of Fig. 3B is described in detail below. The image acquisition area 11a is divided into two parts both vertically and horizontally, and the high-resolution image acquisition area 15a is located in the upper right quarter. The upper left quarter is divided into two parts both vertically and horizontally, and the respective subdivided areas form the four low-resolution image acquisition areas 15b to 15e. Furthermore, the lower right quarter is divided into two parts both vertically and horizontally, and the respective subdivided areas form the four low-resolution image acquisition areas 15f to 15i. Finally, the lower left quarter is divided into four parts both vertically and horizontally, and the respective subdivided areas form the sixteen low-resolution image acquisition areas 15j to 15y.

[0056] Providing many image capture areas in this way makes it possible to capture many different types of information. Furthermore, since image capture areas of varying sizes are provided (excluding the high-resolution image capture area), it is possible to assign image capture areas of different sizes depending on the type of information.

[0057] For example, if the intention is to acquire a multispectral image consisting of several narrowband images, it is possible to assign one or more relatively small image acquisition areas (low-resolution image acquisition areas) 15j to 15y to the narrowband images and equip them with narrowband bandpass filters, and to assign one or more of the relatively large image acquisition areas (intermediate-resolution image acquisition areas) 15b to 15e information such as RGB primary color information, near-infrared information, or polarization information, which require a somewhat higher resolution, and to equip them with optical filters to acquire the respective information.

[0058] As above, in the Fig. 2A to 2D and the Fig. 3A and Fig. In the examples shown in Figure 3B, the image capture areas are all square; however, the present invention is not limited to this, and they can be rectangular. Furthermore, although the high-resolution image capture area(s) may be of a different size than the low-resolution image capture areas, and the low-resolution image capture areas may contain several image capture areas of different sizes, it is preferable that all image capture areas, including the high-resolution and low-resolution image capture areas, have the same aspect ratio.

[0059] As in the Fig. 2A to 2D and the Fig. 3A and Fig. As illustrated in Figure 3B, when multiple image capture areas are formed by subdividing the image capture area 11a, it is difficult to align the fields of view of these multiple image capture areas exactly. This is due to errors in the focal lengths of the lenses, lens aberrations, or similar factors. If the fields of view of the multiple image capture areas do not align exactly, the images captured by the camera can be used, for example, after cropping (removing the edges of the images by cropping).

[0060] The advantages obtained through the camera device 1 of the first embodiment are now described.

[0061] Since in the camera device 1 of the first embodiment the image recording area of ​​the image recording element of the compound eye camera 10, which forms the camera device, is divided into several image recording areas, it is possible to capture images that represent different types of information.

[0062] Furthermore, since the image acquisition areas include a relatively large image acquisition area and a relatively small image acquisition area, it is possible to capture such an image with high resolution by assigning the relatively large image acquisition area to high-priority information, i.e., an image that is more desirable to be captured at high resolution, from the different types of information.

[0063] This makes it possible, for example, to capture primary images, such as RGB images in visible light, at high resolution by assigning relatively large image acquisition areas to these images, and to capture many images that contain additional information, such as narrowband images, near-infrared or ultraviolet images to create a multispectral image, polarization images, or images with different exposure sizes by assigning many relatively small image acquisition areas to these images.

[0064] Furthermore, since the multiple image capture areas are formed in a single image capture element, it is possible to reduce the size of the camera device. Second embodiment

[0065] Fig. Figure 4 is a block diagram illustrating a compound eye image acquisition device 100 according to a second embodiment of the present invention. The Fig. Figure 4 of the illustrated compound eye image acquisition device 100 comprises a camera device 1, an image acquisition control unit 17, an A / D converter 18 and a processor 20. The camera device 1 can be the one described in the first embodiment.

[0066] The image acquisition control unit 17 controls the image acquisition by the camera device 1. For example, it controls the image acquisition time or exposure time.

[0067] An image capture signal output by the camera device 1 is, after undergoing processing such as amplification in an analog processor (not shown), converted into a digital image signal by the A / D converter 18 and input into the processor 20.

[0068] The processor 20 contains an image memory 22 and at least one resolution enhancement unit 30.

[0069] The image memory 22 preferably contains several memory areas 22-1, 22-2, ..., which correspond to the respective image acquisition areas of the compound eye camera 10, which forms the camera device 1. In this case, a digital image signal stored in each memory area represents an image captured in the corresponding image acquisition area.

[0070] From the images stored in the respective memory areas 22-1, 22-2, ... of the image memory 22, the images captured in two image acquisition areas, which have different resolutions, are supplied to the resolution enhancement unit 30.

[0071] For example, a high-resolution image taken in image capture area 15a of Fig. 2A is captured, and a low-resolution image is captured in image capture area 15b of Fig. The image captured in image acquisition area 15a is fed to the resolution enhancement unit 30. The high-resolution image captured in image acquisition area 15a is subsequently designated by reference numeral D0, and the low-resolution image captured in image acquisition area 15b is designated by reference numeral D1.

[0072] The resolution enhancement unit 30 improves (or increases) the resolution of the low-resolution image D1 using the high-resolution image D0 as a reference image to produce a high-resolution image D30.

[0073] Assuming that the high-resolution image D30 to be generated and the reference image D0 correlate with each other in their image features (such as local gradients or patterns), the resolution enhancement unit 30 performs a process of transferring (reflecting) a high-resolution component of an object contained in the reference image D0 onto the low-resolution image D1, thereby generating the high-resolution image D30, which contains the high-resolution component of the object.Furthermore, it is possible to compare the low-resolution image D1 and the reference image D0 and perform adaptive processing, dependent on the position within the image, in such a way as to promote the transmission of the high-resolution component in image areas where the correlation is considered high and to suppress the transmission of the high-resolution component in image areas where the correlation is considered low. This is because the image acquisition conditions (such as wavelength, polarization direction, or exposure magnitude) differ between the low-resolution image D1 and the reference image D0, and thus, depending on the object's reflective properties, a high-resolution component of the low-resolution image D1 and the reference image D0 do not always correlate.

[0074] Fig. Figure 5 represents a configuration example (identified by reference 30a) of the resolution enhancement unit 30.

[0075] The resolution enhancement unit 30a from Fig. 5 performs resolution improvement by a method of separating a low-frequency component and a high-frequency component from each of the low-resolution image D1 and the reference image D0 by filtering and combining according to components.

[0076] The resolution enhancement unit 30a, which is in Fig. Figure 5 shows filter processors 311 and 312, a low-frequency component combiner 313, a high-frequency component combiner 314 and a component combiner 315.

[0077] The filter processor (the first filter processor) 311 extracts a low-frequency component D1L and a high-frequency component D1H from the low-resolution image D1. For example, the filter processor 311 performs smoothing filter processing on the low-resolution image D1 to extract the low-frequency component D1L and generates the high-frequency component D1H of the image by obtaining a difference between the extracted low-frequency component D1L and the original image D1.

[0078] The filter processor (the second filter processor) 312 extracts a low-frequency component D0L and a high-frequency component DOH from the reference image D0. For example, the filter processor 312 performs smoothing filter processing on the reference image D0 to extract the low-frequency component D0L, and generates the high-frequency component DOH of the image by obtaining a difference between the extracted low-frequency component D0L and the original image D0.

[0079] In the smoothing filter processing in the filter processors 311 and 312, a Gaussian filter, a bilateral filter or the like can be used.

[0080] The low-frequency component combiner 313 enlarges the low-frequency component D1L to the same resolution as the reference image D0 and combines the enlarged low-frequency component and the low-frequency component D0L by weighted addition to create a combined low-frequency component D313.

[0081] The high-frequency component combiner 314 enlarges the high-frequency component D1H to the same resolution as the reference image D0 and combines the enlarged high-frequency component and the high-frequency component DOH by weighted addition to create a combined high-frequency component D314.

[0082] The component combiner 315 combines the combined low frequency component D313 and the combined high frequency component D314 to produce the high-resolution image D30.

[0083] In the Fig. In the configuration shown in section 5, it is possible to improve the perceived resolution by performing the combination, while the high-frequency component D0H, which has a high resolution and is included in the high-resolution image D0, is heavily weighted in the combination in the high-frequency component combiner 314.

[0084] Fig. Figure 6 represents another configuration example (identified by reference 30b) of the resolution enhancement unit 30.

[0085] The in Fig. 6. The resolution enhancement unit 30b shown receives the low-resolution image D1 and the reference image D0 and uses a guided filter to improve the resolution of the low-resolution image D1 based on the reference image D0.

[0086] The in Fig. The resolution enhancement unit 30b shown in Figure 6 contains a reduction processor 321, a coefficient calculation unit 322, a coefficient map magnification unit 323 and a linear converter 324.

[0087] The reduction processor 321 reduces the reference image D0 to produce a reduced reference image DOb that has the same resolution as the low-resolution image D1.

[0088] The coefficient calculation unit 322 calculates linear coefficients a m and b m , which approximate a linear relationship between the reduced reference image D0b and the low-resolution image D1.

[0089] The coefficient calculation unit 322 first receives a variance varl(x) of pixel values ​​I(y) of the reduced reference image D0b in a local area Ω(x) centered on the pixel position x, according to equation (1): varI(x)=∑y∈Ω(x)I(y)2−(∑y∈Ω(x)I(y))2.

[0090] In equation (1) I(x) is a pixel value of a pixel at pixel position x of the reduced reference image D0b.

[0091] I(y) is a pixel value of a pixel at pixel position y of the reduced reference image D0b.

[0092] Here, pixel position y is a pixel position in the local area Ω(x) centered on pixel position x.

[0093] The coefficient calculation unit 322 also receives a covariance covlp(x) of the pixel values ​​I(y) of the reduced reference image D0b and the pixel values ​​p(y) of the input image D1 in the local area Ω(x) centered on the pixel position x, according to equation (2): covIp(x)=∑y∈Ω(x)(I(y)⋅p(y))−(∑y∈Ω(x)I(y))⋅(∑y∈Ω(x)p(y)).

[0094] In equation (2) I(x) and I(y) are as described for equation (1).

[0095] p(y) is a pixel value of a pixel at pixel position y of the input image D1.

[0096] The coefficient calculation unit 322 further calculates a coefficient a from the variance varl(x) obtained by equation (1) and the covariance covlp(x) obtained by equation (2), according to equation (3): a(x)=covIp(x)varI(x)+eps

[0097] In equation (3) eps is a constant that determines the degree of boundary conservation and is predetermined.

[0098] The coefficient calculation unit 322 further calculates a coefficient b(x) using the coefficient a(x) obtained by equation (3) according to equation (4): b(x)=∑y∈Ω(x)p(y)−a(x)∑y∈Ω(x)I(y).

[0099] The coefficients a(x) and b(x) are called linear regression coefficients. The coefficient calculation unit 322 also calculates the linear coefficients a m (x) and b m(x) by averaging the coefficients a(x) and b(x) obtained by equations (3) and (4), according to equation (5): am(x)=∑y∈Ω(x)a(y)2,bm(x)=∑y∈Ω(x)b(y)2.

[0100] The coefficient map magnification unit 323 enlarges a coefficient map consisting of the linear coefficients a m (x) consists of the coefficients obtained by equation (5) in the coefficient calculation unit 322, and a coefficient chart consisting of the linear coefficients b m (x) consists of the coefficients obtained by equation (5) in the coefficient calculation unit 322, to the same resolution as the reference image D0. The linear coefficients of the enlarged coefficient maps are given as a mb (x) and b mb (x) marked. A coefficient map is a map in which coefficients corresponding to all pixels that make up an image are arranged in the same positions as the corresponding pixels.

[0101] The linear converter 324 generates a high-resolution image D30, which contains information represented by the low-resolution image D1, based on the linear coefficients a mb and b mb the enlarged coefficient maps and the reference image D0.

[0102] In particular, the linear converter 324 uses the linear coefficients a mb and b mb of the enlarged coefficient maps to derive output values ​​q of the directed filter according to equation (6): q(x)=amb(x)J(x)+bmb(x).

[0103] q(x) is a pixel value of a pixel at pixel position x of the high-resolution image D30.

[0104] J(x) is a pixel value of a pixel at pixel position x of the reference image D0.

[0105] Equation (6) indicates that the output of the guided filter (the pixel value of image D30) q(x) and the pixel value J(x) of the reference image D0 have a linear relationship.

[0106] In the Fig. In the configuration shown in Figure 6, the coefficient calculation unit 322 and the linear converter 324 are basic components of the processing by the guided filter, and the reduction processor 321 and the coefficient map magnification unit 323 are added to reduce the processing load on the coefficient calculation unit 322.

[0107] By calculating the pixel values ​​of the high-resolution image D30, as described above, it is possible to perform smoothing processing only on areas of the reduced reference image D0b where the variance varl(x) is small, while preserving the texture of the other area.

[0108] Although the aforementioned resolution enhancement unit 30b performs processing using a guided filter, the present invention is not limited thereto. The resolution enhancement unit 30b can employ other methods, such as a method for improving the resolution of images containing different types of information based on edge or gradient information of a high-resolution image, such as a method using a common bilateral filter.

[0109] Fig. Figure 7 represents a further example (identified by reference numeral 20b) of the processor 20 used in the second embodiment. The processor 20 is described in Figure 7. Fig. The processor 20b shown in Figure 7 contains an alignment unit 25 in addition to the image memory 22 and the resolution enhancement unit 30.

[0110] If a misalignment exists between the low-resolution image D1 and the high-resolution image D0 output by the camera device 1, the alignment unit 25 performs alignment processing prior to resolution enhancement by the resolution enhancement unit 30. This alignment processing can be performed using a fixed-value alignment based on information about the initial misalignment (correction information), a dynamic alignment, registration (image matching), or similar methods.

[0111] The resolution enhancement unit 30 performs the resolution enhancement using images aligned by the alignment unit 25 as input.

[0112] It is also possible to provide multiple resolution enhancement units, each identical to one of the resolution enhancement units (30a and 30b) described in the Fig. 5 and Fig. 6 are described, and the resolution enhancement units described as their modifications are to input different low-resolution images into the respective resolution enhancement units and to perform resolution enhancement on the input low-resolution image using a high-resolution image as a reference image in each resolution enhancement unit.

[0113] The high-resolution images used as reference images by the multiple resolution enhancement units can be the same or different.

[0114] It is also possible to use several combinations of the alignment unit 25 and the resolution enhancement unit 30, which are in Fig. 7 is to be provided.

[0115] Fig. Figure 8 represents a further example (identified by reference numeral 20c) of the processor used in the second embodiment.

[0116] The in Fig. 8 The processor 20c shown contains an image memory 22, a resolution enhancement unit 30c and the image enlargers 31r and 31b.

[0117] The example of Fig. Figure 8 assumes a case in which the image acquisition area 11a contains one high-resolution image acquisition area 15a and three or more low-resolution image acquisition areas, for example as in Fig. 2A, and the high-resolution image acquisition area 15a captures G-information, and the three low-resolution image acquisition areas (e.g. 15b, 15c and 15d) capture an R-image, a G-image and a B-image respectively.

[0118] In this case, three low-resolution images, D1-r, D1-g, and D1-b, represent the R, G, and B images, respectively, which have low resolution. Additionally, the G image, which is captured at high resolution in image acquisition area 15a, is identified by the reference symbol D0.

[0119] The image enlargers 31r and 31b enlarge the images D1-r and D1-b to the same resolution as the high-resolution image D0 in order to produce the enlarged images D31-r and D31-b respectively.

[0120] The resolution enhancement unit 30c replaces the image D1-g with the image D0 and outputs the image obtained by the replacement as a high-resolution image D30-g.

[0121] Such processing can drastically reduce the computational effort or processing time for image processing and reduce the hardware costs of the processor.

[0122] In the configuration of Fig. 8. It is also possible to use the alignment unit 25 (shown in Fig. 7) to be provided in front of the resolution enhancement unit 30c. It is also possible to provide similar alignment units in front of the image enlargers 31r and 31b.

[0123] The advantages obtained through the compound eye image acquisition device 100 of the second embodiment are now described.

[0124] In the compound eye image acquisition device 100 of the second embodiment, several images, which have different types of information, are captured from several image acquisition areas of the camera device, wherein these images include an image which has a relatively low resolution and an image which has a relatively high resolution, and the resolution of the image which has the relatively low resolution is improved using a component with the high resolution of the image which has the relatively high resolution, and thus it is possible to obtain several images which have a high resolution and have different types of information.

[0125] Thus, even if the image capture area for capturing images containing different types of information is small, a high-resolution image can be produced, and it is possible to obtain high-resolution images containing different types of information while reducing the size of the camera device. Third embodiment

[0126] Fig. Figure 9 represents an example (designated by reference numeral 20d) of a processor 20 used in a compound eye imaging device according to a third embodiment of the present invention. The processor 20 is described in Figure 9. Fig. The processor 20d shown in Figure 9 contains an image memory 22, several, specifically first to Nth (where N is an integer equal to or greater than 2), resolution enhancement units 30-1 to 30-N, and a combiner 40. Apart from the processor 20d, the compound eye image acquisition device of the third embodiment is configured in the same way as, for example, Fig. 4 configured.

[0127] The first to Nth resolution enhancement units 30-1 to 30-N are each provided so that they cover N low-resolution image acquisition areas (e.g., 15b, 15c, ... in the example of Fig. 2A) correspond to receiving low-resolution images D1-1 to D1-N, which are captured in the corresponding low-resolution image acquisition areas, and receiving one high-resolution image D0, which is captured in the high-resolution image acquisition area 15a.

[0128] The resolution enhancement unit 30-n (where n is one of the integers in the range from 1 to N) improves the resolution of the low-resolution image D1-n using the high-resolution image D0 as a reference image, producing a high-resolution image D30-n. This processing is performed by all resolution enhancement units 30-1 to 30-N, resulting in multiple images D30-1 to D30-N, each containing different types of information.

[0129] Each of the resolution enhancement units 30-1 to 30-N is, for example, as in Fig. 5, Fig. 6 or Fig. 8 described and configured.

[0130] The combiner 40 receives the multiple high-resolution images D30-1 to D30-N, which contain different types of information, and produces one or more combined high-resolution images D40-a, D40-b, ...

[0131] In particular, the combiner 40 combines the high-resolution images D30-1 to D30-N, which contain the different types of information and are generated by the resolution enhancement units 30-1 to 30-N, and produces the combined high-resolution images D40-a, D40-b, ...

[0132] The combination processing in the combiner 40 can be performed, for example, by pansharpening, weighted addition of images, intensity combination, or area selection. Area selection can, for instance, be performed based on the visibility of an image, estimated using a local variance as an index.

[0133] Among the above, pansharpening techniques are used in satellite image processing (remote acquisition) or the like, and pansharpening processing involves converting an RGB color image into an HSI (hue, saturation, intensity) image, replacing the I-values ​​of the HSI image resulting from the conversion with pixel values ​​of a monochrome image, which is a high-resolution image, and converting the HSI image back into an RGB image with the replaced pixel values.

[0134] Fig. 10 represents an example (marked with reference 40b) of the combiner 40.

[0135] The in Fig. The combined unit 40b shown contains a luminance / color separator 411, a luminance separator 412, a weighting adder 413 and a luminance / color combiner 414 and performs the combination by weighted addition of luminance.

[0136] The combiner 40b receives an R-image D30-r, a G-image D30-g, a B-image D30-b, a polarization image D30-p, and a NIR (near-infrared) image D30-i, which are further enhanced by the resolution improvement provided by the multiple resolution improvement units 30-1, 30-2, ... (each of which is equal to the one in Fig. 5, Fig. 6 or Fig. 8 described or the like) are subjected to.

[0137] The images D30-r and D30-b, which were produced by the image enlargers 31r and 31b, which are in Fig. The enlarged images shown in Figure 8 can be entered into the combiner 40b and used instead of the high-resolution images. Thus, the combiner 40b can be configured to combine one or more high-resolution images with one or more enlarged images.

[0138] The luminance / color separator 411 receives the R image D30-r, the G image D30-g and the B image D30-b and separates them into a luminance component D411-y and color components (components of the colors R, G and B) D411-r, D411-g and D411-b.

[0139] The luminance separator 4123 receives the polarization image D30-p and separates a luminance component D412.

[0140] The weighting adder 413 weights the luminance component D412 of the polarization image, which is output from the luminance separator 412, and the NIR image D30-i, which is input into the combiner 40b, and adds them to the luminance component D411-y, which is output from the luminance / color separator 411, thereby obtaining a combined luminance component D413.

[0141] The luminance / color combiner 414 combines the color components D411-r, D411-g and D411-b, which are output from the luminance / color separator 411, and the combined luminance component D413, which is output from the weighting adder 413, thereby producing an R-image D40-r, a G-image D40-g and a B-image D40-b.

[0142] The R-image D40-r, the G-image D40-g and the B-image D40-b, which are output from the luminance / color combiner 414, have luminance information enhanced by the luminance component of the polarization image D30-p and the NIR image D30-i.

[0143] In weighted addition using the weighting adder 413, it is possible to use a method for adding a pixel value multiplied by a gain depending on an image.

[0144] Alternatively, it is also possible to extract high-frequency components from the respective images (the luminance component D411-y, which is output from the luminance / color separator 411, the luminance component D412, which is output from the luminance separator 412, and the NIR image D30-i, which is input into the combiner 40b) by filter processing, to weight and add them together and to obtain the weighted mean.

[0145] Fig. 11 represents a further example (identified by reference numeral 20e) of the processor 20 used in the third embodiment.

[0146] The in Fig. The processor 20e shown in Figure 11 contains a camera information input port 23 in addition to an image memory 22, several resolution enhancement units 30-1 to 30-N and a combiner 40 and receives compound eye camera information Dinfo from the compound eye camera 10 via the port 23 and sends the compound eye camera information Dinfo to the resolution enhancement units 30-1 to 30-N and the combiner 40.

[0147] The compound eye camera information Dinfo contains information specifying the wavelengths captured in the respective image acquisition areas, information specifying polarization directions, information specifying the positions (the positions in the image acquisition surfaces) of the respective image acquisition areas, and the like.

[0148] By inputting the compound eye camera information Dinfo into the resolution enhancement units 30-1 to 30-N and the combiner 40, it is possible to improve the accuracy of the resolution enhancement, the combination processing or the like, or to increase the information obtained through these processes.

[0149] For example, if the compound eye camera information Dinfo specifies the spectral properties of the optical filters provided for the respective image acquisition areas, it is possible to extract a near-infrared image from RGB images and a monochrome image in combination processing.

[0150] As described above, the third embodiment makes it possible to produce a more useful image for the intended use by improving the resolution of several images containing different types of information captured by the camera device and then combining them. Fourth embodiment

[0151] Fig. Figure 12 represents a configuration example (identified by reference numeral 20f) of a processor 20 used in a compound eye imaging device according to a fourth embodiment of the present invention. The processor 20 is described in Figure 12. Fig. The processor 20f shown in Figure 12 contains an image memory 22, several, specifically first to Nth, resolution enhancement units 30-1 to 30-N, and a combiner 41. Apart from the processor 20f, the compound eye image acquisition device of the fourth embodiment is configured in the same way as, for example, Fig. 4 configured. The resolution enhancement units 30-1 to 30-N are the same as those used, for example, in Fig. 9 are described.

[0152] The combiner 41 receives high-resolution images D30-1 to D30-N, which are output by the resolution enhancement units 30-1 to 30-N, and generates from them, by interpolation, the high-resolution images D41-a, D41-b, ..., which represent information that differs from them in its type.

[0153] In this case, it is assumed that the images D1-1, D1-2, ..., which are located in the multiple high-resolution image capture areas (e.g., 15b, 15c, ... in the example of Fig. 2A) are captured, contain multiple images that have different types or values ​​of at least one parameter from one or more parameters that specify the image capture conditions, and thus the high-resolution images D30-1, D30-2, ... that are generated from these low-resolution images D1-1, D1-2, ... contain multiple images that have different types or values ​​of at least one parameter from one or more parameters that specify the image capture conditions.

[0154] The combiner 41 generates (reconstructs) by interpolation from the multiple high-resolution images D30-1, D30-2, ... the high-resolution images D41-a, D41-b, ... which have a type or value of at least one parameter that differs from that of any of the multiple high-resolution images D30-1, D30-2, ...

[0155] For example, a recovery technique used in compressed capture can be applied to interpolation.

[0156] An example of image generation through interpolation is given below with reference to Fig. 13 described. In the example of Fig. 13 are different types of parameters: wavelength and exposure magnitude. For wavelength, the values ​​of the parameters R, G, and B are (a representative wavelength in an R wavelength range, a representative wavelength in a G wavelength range, and a representative wavelength in a B wavelength range), and for exposure magnitude, the values ​​of the parameters are 1 / 1000, 1 / 100, 1 / 10, and 1. These values ​​are relative values ​​based on the case where no optical filter is used.

[0157] It is assumed that images with the parameter combinations defined by the circles in Fig. The images specified in section 13 are entered into combiner 41 as high-resolution images D30-1, D30-2, ... For example, high-resolution image D30-1 is an image obtained by improving the resolution of an image captured at an exposure time of 1 / 1000 in an image capture area equipped with an optical filter transparent to light in an R wavelength range. Similarly, high-resolution image D30-2 is an image obtained by improving the resolution of an image captured at an exposure time of 1 / 1000 in an image capture area equipped with an optical filter transparent to light in a B wavelength range.

[0158] The combiner 41 generates images D41-a to D41-f from the high-resolution images D30-1 to D30-6 by interpolation, corresponding to the parameter combinations indicated by the triangles. For example, image D41-a is an image assumed to be produced by improving the resolution of an image taken at an exposure size of 1 / 1000 in an image acquisition area equipped with an optical filter transparent to light in a G-wavelength range.

[0159] The combiner 41 outputs not only the generated images D41-a to D41-f, but also the input images D30-1 to D30-6.

[0160] Performing such processing will yield a larger number of high-resolution images D30-1 to D30-6 and D40-a to D40-f, which contain different types of information.

[0161] The advantages obtained through the compound eye image acquisition device of the fourth embodiment are now described.

[0162] The fourth embodiment makes it possible to generate a larger number of images containing different types of information from a relatively small number of images captured by recording, each containing different types of information. Thus, even if the number of image capture areas is not large, many types of information can be obtained. Fifth embodiment

[0163] Fig. Figure 14 represents a configuration example (designated by reference numeral 20g) of a processor 20 used in a compound eye imaging device according to a fifth embodiment of the present invention. The Fig. The processor 20g shown in Figure 14 contains an image memory 22, a combiner 42, and a resolution enhancement unit 32. Apart from the processor 20g, the compound eye image acquisition device of the fifth embodiment is configured in the same way as, for example, Fig. 4 configured.

[0164] The combiner 42 performs combination processing on images D1-1 to D1-N, which are located in the image acquisition areas (e.g., 15b, 15c, ... in the example of Fig. 2A) are captured at low resolution and contain different types of information, thereby producing one or more combined images (low-resolution combined images) D42-a, D42-b, ...

[0165] The resolution enhancement unit 32 performs resolution enhancement on one or more combined images from the combined images D42-a, D42-b, ..., which are output from the combiner 42, using the reference image D0, thereby producing high-resolution images (high-resolution combined images) D32-a, D32-b, ...

[0166] If the number of combined images D42-a, D42-b, ..., generated by the combination using combiner 42, is less than the number of input images D1-1 to D1-N, performing the resolution enhancement after the combination can reduce the processing required for the resolution enhancement and reduce the overall computational effort. Sixth embodiment

[0167] Fig. Figure 15 is a block diagram illustrating a compound eye image acquisition device 102 according to a sixth embodiment of the present invention.

[0168] The compound eye image acquisition device 102 according to the sixth embodiment includes a camera device 50, an image acquisition control unit 17, A / D converters 18 and 19 and a processor 20h.

[0169] Fig. Figure 16 is a perspective exploded view of the camera device 50. The camera device 50 contains a compound eye camera 60 and a monocular camera 70.

[0170] As described in detail below, the compound eye camera 60 uses one that has an image capture area divided into several image capture areas, like the compound eye camera 10, which is in Fig. 1A is shown, and on the other hand, the monocular camera 70 has an image acquisition area that is not subdivided, and an image taken by the monocular camera 70 is used instead of the image taken in the high-resolution image acquisition area of ​​the compound eye camera 10. Fig. 1A is recorded and used.

[0171] The compound eye camera 60 includes an image acquisition element 61, a lens assembly 62, a filter assembly 63 and a partition 64.

[0172] The image acquisition element 61 has a rectangular image acquisition area 61a, and the image acquisition area 61a is subdivided, for example, into several, e.g., nine, image acquisition areas 65a to 65i, as shown in Fig. Figure 17A shows that these nine image acquisition areas, 65a to 65i, are all the same size and are arranged in three rows and three columns.

[0173] Fig. 17B provides another example of how the image capture area 61a is to be subdivided. In the Fig. In the example shown in Figure 17B, the image acquisition area 61a of the image acquisition element 61 of the compound eye camera 60 is divided into four image acquisition areas 65a to 65d. These four image acquisition areas 65a to 65d are of the same size and are arranged in two rows and two columns.

[0174] Although the image acquisition areas 65a, 65b, ... of the compound eye camera 60 can be the same size as in the examples of Fig. 17A and Fig. 17B, ​​the present invention is not limited thereto, and the image acquisition areas 65a, 65b, ... can have different sizes.

[0175] Even if they are of different sizes, they preferably have the same aspect ratio.

[0176] The lens arrangement 62 contains the lenses 62a, 62b, ..., which are provided such that they correspond to the respective image acquisition areas 65a, 65b, ... and capture images of the same field of view onto the respective corresponding image acquisition areas.

[0177] The filter arrangement 63 contains the optical filters 63a, 63b, ..., which are intended for one or more image acquisition areas from the multiple image acquisition areas.

[0178] The partition 64 is provided between the image acquisition areas 65a, 65b, ... and prevents each image acquisition area from receiving light from lenses that are not the corresponding lens.

[0179] The monocular camera 70 contains an image acquisition element 71, a lens 72 and an optical filter 73.

[0180] The image capture element 71 also has a rectangular image capture area 71a. The entire image capture area 71a forms a single image capture region 75.

[0181] The Fig. 17A and Fig. Figure 17B schematically represents the positional relationship of the image acquisition area 75 of the monocular camera 70 to the image acquisition areas 65a, 65b, ... of the compound eye camera 60.

[0182] The image acquisition area 75, which is formed by the entire image acquisition surface 71a of the image acquisition element 71 of the monocular camera 70, has more pixels than any of the image acquisition areas 65a, 65b, ... of the compound eye camera 60. Thus, the resolution of the image acquisition area 75 of the monocular camera 70 is higher than the resolution of the largest image acquisition area from the image acquisition areas 65a, 65b, ... of the compound eye camera 60.

[0183] Image capture area 75 has the same aspect ratio as image capture areas 65a, 65b, ...

[0184] The image acquisition area 75 of the monocular camera 70 is an image acquisition area whose resolution differs from that of the image acquisition areas 65a, 65b, ... of the compound eye camera, and the former has a higher resolution than the latter. For the sake of distinction, the former can be referred to as an image acquisition area of ​​the first type, and the latter can be referred to as image acquisition areas of the second type.

[0185] The lens 72 of the monocular camera 70 is designed so that an image of the same field of view as images on the respective image acquisition areas of the compound eye camera 60 is projected onto the image acquisition area 75.

[0186] The lenses 62a, 62b, ... of the compound eye camera 60 and the lens 72 of the monocular camera 70 form a lens group.

[0187] The optical filters 63a, 63b, ..., which form the filter arrangement 63, and the optical filter 73 contain optical filters that have different optical properties, so that different types of information (images representing different types of information) are captured from the respective image acquisition areas.

[0188] For example, it is possible to select the optical filters for the respective image capture areas, assuming that the image capture area 15a has a high resolution of Fig. 2A of the first embodiment is replaced by the image acquisition area 75 of the monocular camera 70, and the image acquisition areas 15b, 15c, ... are of low resolution. Fig. 2A are replaced by the image acquisition areas 65a, 65b, ... of the compound eye camera 60. That is, it is possible to use an optical filter that has the same properties as the one for the high-resolution image acquisition area 15a. Fig. 2A provides optical filters for the image acquisition area 75 of the monocular camera 70 in the sixth embodiment and optical filters having the same properties as the optical filters provided for the low-resolution image acquisition areas 15b, 15c, ... Fig. 2A are provided for the image acquisition areas 65a, 65b, ... of the compound eye camera 60 in the sixth embodiment.

[0189] It is possible not to provide any optical filters for one or more image capture areas, e.g. image capture area 75, from image capture areas 75, 65a, 65b, ... (to set the one or more image capture areas as monochrome areas).

[0190] The image acquisition control unit 17 controls the image acquisition by the compound eye camera 60 and the image acquisition by the monocular camera 70. For example, it controls the timing or exposure parameters of the image acquisition by the two cameras. The image acquisition timing control is implemented so that the two cameras perform the image acquisition essentially simultaneously.

[0191] The processor 20h of the compound eye image acquisition device 102 of this embodiment receives images captured in the multiple image acquisition areas of the compound eye camera 60 and an image captured by the monocular camera 70 via the A / D converters 18 and 19, respectively.

[0192] The processor 20h contains an image memory 22 and at least one resolution enhancement unit 30.

[0193] The image memory 22 preferably has several memory areas 22-1, 22-2, ... which correspond to the several image acquisition areas of the compound eye camera 60 and the image acquisition area of ​​the monocular camera 70 on a one-to-one basis.

[0194] The resolution enhancement unit 30 receives the high-resolution image D0, captured by the monocular camera 70, as a reference image, receives a low-resolution image D1, captured in one of the image acquisition areas of the compound eye camera 60, and enhances the resolution of the low-resolution image D1 using a high-resolution component contained in the reference image D0, thereby producing a high-resolution image D30.

[0195] As above, when the camera device 1 of the first embodiment is used, an image captured in a high-resolution image acquisition area (15a or the like) of the compound eye camera 10 is used as the reference image to improve the resolution of an image captured in the low-resolution image acquisition area of ​​the same compound eye camera 10; and, on the other hand, when the camera device 50 of the sixth embodiment is used, an image captured by the image acquisition element of the monocular camera 70 is used as the reference image to improve the resolution of a low-resolution image captured by the compound eye camera 60.

[0196] Otherwise, the sixth embodiment is identical to the second embodiment. For example, the processing of the resolution enhancement unit 30 can be carried out in the same way as described in the second embodiment with reference to Fig. 5, Fig. 6, Fig. 8 or similar.

[0197] Although it has been described that a processor identical to the one described in the second embodiment is used as processor 20h of the sixth embodiment, it is also possible to use a processor identical to that of the third, fourth, or fifth embodiment instead. In any case, the image captured in the image acquisition area of ​​the monocular camera 70 should be used as the reference image instead of the high-resolution image captured in the high-resolution image acquisition area of ​​the first embodiment.

[0198] In the example of Fig. 17A are a center of image acquisition area 65i, located at the center of image acquisition area 61a, and a center of image acquisition area 75, horizontally aligned. Such an arrangement makes it possible to obtain depth information by performing stereo matching using displacement due to parallax between the image captured in image acquisition area 65i and the image captured in image acquisition area 75.

[0199] Advantages obtained through the compound eye image acquisition device 102 of the sixth embodiment will now be described.

[0200] In the compound eye image acquisition device 102 of the sixth embodiment, the monocular camera 70 is provided separately from the compound eye camera 60, and a high-resolution image can be captured by the monocular camera 70. Thus, it is possible to improve the resolution of an image captured in each image acquisition area of ​​the compound eye camera 60.

[0201] Furthermore, it is possible to manufacture the image acquisition areas 65a, 65b, ... of the compound eye camera 60 so that they all have the same shape, as shown in the example in the Fig. 17A and Fig. 17B is shown, and thereby reduce manufacturing costs.

[0202] Furthermore, since the distance between the centers of the compound eye camera 60 and the monocular camera 70 is relatively large, the parallax shift between the image captured by the monocular camera 70 and the images captured by the compound eye camera 60 is greater than the parallax shift between the images captured in different image acquisition areas of the compound eye camera 10 of the first embodiment. It is also possible to obtain depth information more accurately by utilizing the parallax.

[0203] In the first embodiment, a camera device is formed solely by a compound-eye camera (10), and in the sixth embodiment, a camera device is formed by a compound-eye camera (60) and a monocular camera (70); however, in short, it is sufficient that a camera device comprising multiple image acquisition areas (one image acquisition area of ​​a first type and image acquisition areas of a second type) of different sizes, and a group of filters are provided such that images representing different types of information are captured in the multiple image acquisition areas. The different image acquisition areas can be formed in a single image acquisition element (11) as in the first embodiment and can be formed in multiple image acquisition elements (61 and 71) as in the sixth embodiment.The multiple image capture areas, which have different sizes, contain at least one image capture area of ​​a first type and multiple image capture areas of a second type, which have smaller areas and a smaller number of pixels than the image capture area of ​​the first type.

[0204] If the image acquisition area of ​​the first type and the image acquisition areas of the second type are both formed in a single image acquisition element, as in the first embodiment, the image acquisition area of ​​the first type and the image acquisition areas of the second type are formed by subdividing an image acquisition surface of the single image acquisition element, and a lens group includes a lens that is contained in a lens arrangement provided for the image acquisition surface.

[0205] In this case, a partition is provided to prevent each of the multiple image capture areas from receiving light from lenses that are not the corresponding lens.

[0206] If the image acquisition area of ​​the first type and the image acquisition areas of the second type are formed in different image acquisition elements, as in the sixth embodiment, the “one or more image acquisition areas of the first type” described above are formed by a single image acquisition area formed by the entire image acquisition surface of a first image acquisition element (71), the multiple image acquisition areas of the second type are formed by subdividing the image acquisition surface of a second image acquisition element (61), and a lens group includes a lens provided for the image acquisition surface of the first image acquisition element and an objective included in a lens arrangement provided for the image acquisition surface of the second image acquisition element.

[0207] In this case, a partition is provided to prevent each of the multiple image capture areas of the second image capture element from receiving light from lenses that are not the corresponding lens. Seventh embodiment

[0208] The processors of the compound eye image-taking devices described in the second to sixth embodiments can be dedicated processors or can be CPUs of a computer that execute programs stored in memory.

[0209] As an example, a processing procedure is described in the case where a computer is caused to perform image processing implemented by a compound eye image capture device that utilizes the processor of Fig. 9, below with reference to Fig. 18 described.

[0210] First, in step ST1, an image is captured, for example by the in Fig. 1A Camera device 1 shown, and several low-resolution images D1-1 to D1-N representing different types of information, and one high-resolution image D0 are captured and stored in a memory (which has the same role as the image memory 22 of Fig. 9) are stored here. Images D1-1 to D1-N, with low resolution, and image D0, with high resolution, are examples of images containing different types of information, captured in the multiple image capture areas 15a, 15b, 15c, ... of the in Fig. 1A camera device 1 is captured, onto which images of the same field of view are mapped, and the images, wherein images which have a relatively low resolution than the low-resolution images D1-1, D1-2, ... are captured, and an image which has a relatively high resolution than the high-resolution image D0 is captured.

[0211] Next, in step ST2, the resolution enhancement processing is performed on each of the low-resolution images D1-1, D1-2, ... using the high-resolution image D0 as a reference image, so that high-resolution images D30-1, D30-2, ... are generated. The resolution enhancement processing is the same as that performed, for example, for the resolution enhancement unit of the Fig. 5, Fig. 6 or Fig. 8 is described.

[0212] Next, in step ST3, the multiple high-resolution images D30-1, D30-2, ... are combined to create one or more combined images D40-a, D40-b. The combination processing is performed as for combiner 40. Fig. 9 described and executed.

[0213] As described above, the present invention makes it possible to capture images in several image acquisition areas which have different types of information and different resolutions, and to obtain a high-resolution image from relatively low-resolution images from the captured images.

[0214] Although compound eye image acquisition devices have been described above in the second to sixth embodiments, image processing methods implemented by the compound eye image acquisition devices also form part of the present invention. Furthermore, programs for causing a computer to execute the processing of the aforementioned compound eye image acquisition devices or image processing methods, and computer-readable recording media for storing such programs, also form part of the present invention. Reference symbol list

[0215] 1 Camera assembly, 10 Compound eye camera, 11 Image acquisition element, 11a Image acquisition area, 12 Lens assembly, 12a, 12b, ... Lens, 13 Filter assembly, 13a, 13b, ... Optical filter, 14 Partition, 15a High-resolution image acquisition area, 15b, 15c, ... Low-resolution image acquisition area, 17 Image acquisition control unit, 18, 19 A / D converter, 20, 20a to 20h Processor, 22 Image memory, 25 Alignment unit, 30, 30a to 30c, 30-1 to 30-N Resolution enhancement unit, 31r, 31b Image magnification unit, 32 Resolution enhancement unit, 40, 42 Combiner, 50 Camera assembly, 60 Compound eye camera, 61 Image acquisition element, 61a Image acquisition area, 62 Lens arrangement, 62a, 62b, ... Lens, 63 Filter arrangement, 63a, 63b, ... Optical filter, 64 Partition, 65a, 65b, ...Low-resolution image acquisition area, 70 Monocular camera, 71 Image acquisition element, 71a Image acquisition area, 72 Lens, 73 Optical filter, 75 High-resolution image acquisition area, 100, 102 Compound eye image acquisition device, 311, 312 Filter processor, 313 Low-frequency component combiner, 314 High-frequency component combiner, 315 Component combiner, 321 Reduction processor, 322 Coefficient calculation unit, 323 Coefficient map magnification unit, 324 Linear converter, 411 Luminance / color separator, 412 Luminance separator, 413 Weighting adder, 414 Luminance / color combiner.

Claims

[1] A compound eye image recording device comprising (100): a camera device (1) comprising: an image capture element (11) comprising several image capture areas (15a, 15b, ...); several lenses (12a, 12b, ...) arranged to correspond to the several image acquisition areas (15a, 15b, ...) and configured to capture images of the same field of view onto the corresponding image acquisition areas (15a, 15b, ...); and several optical filters (13a, 13b, ...), wherein the multiple image capture areas (15a, 15b, ...) include at least one image capture area (15a) of a first type and multiple image capture areas of a second type (15b, 15c, ...) which are of the same size and smaller than the at least one image capture area (15a) of the first type, wherein the multiple lenses (12a, 12b, ...) have a focal length such that lenses corresponding to a larger image capture area have a longer focal length, and wherein the multiple optical filters (13a, 13b, ...) are provided such that an image captured in each of the image acquisition areas (15b, 15c, ...) of the second type represents a type of information that is different from that of an image captured in each of the at least one image acquisition area (15a) of the first type; and a processor (20) comprising at least one resolution enhancement unit (30a), wherein the at least one resolution enhancement unit (30a) comprises: a first filter processor (311) for receiving, as a low-resolution image (D1), the image captured in one of the at least one image acquisition area (15b, 15c,...) of the second type, and for extracting, from the low-resolution image (D1), a low-frequency component (D1L) and a high-frequency component (D1H) of the low-resolution image (D1); a second filter processor (312) for receiving, as a reference image (D0), the image which is captured in one of the at least one image acquisition area (15a) of the first type, for extracting, from the reference image (D0), a low-frequency component and a high-frequency component of the reference image (D0); a low-frequency component combiner (313) to enlarge the low-resolution low-frequency component (D1L) of the low-resolution image (D1) to the same resolution as the reference image (D0) and to combine the enlarged low-frequency component and the low-frequency component (D0L) of the reference image (D0) by weighted addition to produce a combined low-frequency component (D313); a high-frequency component combiner (314) to enlarge the high-frequency component (D1H) of the low-resolution image (D1) to the same resolution as the reference image (D0) and to combine the enlarged high-frequency component and the high-frequency component (D0H) of the reference image (D0) by weighted addition to produce a combined high-frequency component (D314); and a component combiner (315) to combine the combined low frequency component (D313) and the combined high frequency component (D314) to produce a high resolution image (D30). [2] A compound eye image recording device comprising (100): a camera device (1) comprising: an image capture element (11) comprising several image capture areas (15a, 15b, ...); several lenses (12a, 12b, ...) arranged to correspond to the several image acquisition areas (15a, 15b, ...) and configured to capture images of the same field of view onto the corresponding image acquisition areas (15a, 15b, ...); and several optical filters (13a, 13b, ...), wherein the multiple image capture areas (15a, 15b, ...) include at least one image capture area (15a) of a first type and multiple image capture areas of a second type (15b, 15c, ...) which are of the same size and smaller than the at least one image capture area (15a) of the first type, wherein the multiple lenses (12a, 12b, ...) have a focal length such that lenses corresponding to a larger image capture area have a longer focal length, and wherein the multiple optical filters (13a, 13b, ...) are provided such that an image captured in each of the image acquisition areas (15b, 15c, ...) of the second type represents a type of information that is different from that of an image captured in each of the at least one image acquisition area (15a) of the first type; and a processor (20) comprising at least one resolution enhancement unit (30a), wherein the at least one resolution enhancement unit (30a) comprises: a reduction processor (321) for receiving, as a reference image (D0), the image that is captured in one of the at least one image acquisition area (15a) of the first type, and for reducing the reference image (D0) to produce a reduced reference image (D0b) that has the same resolution as a low-resolution image (D1) that is captured in one of the image acquisition areas (15b, 15c,...) of the second type; a coefficient calculation unit (322) to calculate linear coefficients that approximate a linear relationship between the reduced reference image (D0b) and the low-resolution image (D1); a coefficient map magnification unit (323) to enlarge a coefficient map containing the linear coefficients calculated by the coefficient calculation unit (322) to the same resolution as the reference image (D0); and a linear converter (324) to generate a high-resolution image (D30) containing information represented by the low-resolution image (D1) based on linear coefficients of the enlarged coefficient map and the reference image (D0). [3] Compound eye image acquisition device according to claim 1 or 2, wherein the at least one resolution enhancement unit comprises several resolution enhancement units (30-1 to 30-N), and wherein the processor (20d) further comprises a combiner (40) to combine the multiple high-resolution images (D30-1 to D30-N) produced by the multiple resolution enhancement units (30-1 to 30-N) to produce one or more combined high-resolution images (D40-a, D-40-b, ...). [4] Compound eye image acquisition device according to claim 3, wherein the multiple low-resolution images (D1-1 to D1-N) that are subjected to resolution enhancement by the multiple resolution enhancement units (30-1 to 30-N) have different types or values ​​of at least one parameter from one or more parameters that specify the image acquisition conditions, and thus the multiple high-resolution images (D30-1 to D30-N) that are generated by the multiple resolution enhancement units (30-1 to 30-N) have different types or values ​​of at least one parameter from one or more parameters that specify the image acquisition conditions, and wherein the combiner (41) interpolates an image (D41-a, D41-b,...) from the multiple high-resolution images (D30-1 to D30-N).) produced with high resolution, which has a type or value of at least one parameter that is different from that of any of the multiple high-resolution images (D30-1 to D30-N). [5] Compound eye image acquisition device according to claim 1, wherein the processor contains (20g): a combiner (42) to combine the images (D1-1 to D1-N) captured in the multiple image acquisition areas (15b, 15c, ...) of the second type to produce at least one low-resolution combined image (D42-a, D42-b, ...); and a resolution enhancement unit (32) to receive the image captured in one of the at least one image acquisition area (15a) of the first type as a reference image (D0) and to enhance the resolution of the at least one low-resolution combined image (D42-a, D42-b,...) using a high-resolution component contained in the reference image (D0) in order to produce at least one high-resolution combined image (D32-a, D32-b, ...). [6] Compound eye image capture device according to any one of claims 1 to 5, wherein none of the optical filters (13a, 13b, ...) for capturing the images, which represent the different types of information, is provided to the image capture area (15a) of the first type. [7] Program for causing a computer to perform the processing in the compound eye image acquisition device according to any one of claims 1 to 6. [8] Computer-readable recording medium that stores the program according to claim 7. [9] Image processing method comprising the following: a step (ST1) for capturing multiple images, which are captured by recording in multiple image acquisition areas onto which images of the same field of view are mapped, wherein the multiple images have different types of information and different resolutions, a resolution enhancement step (ST2) for performing resolution enhancement processing on images from the multiple images with different resolutions that have a relatively low resolution, using a high-resolution component contained in one image from the multiple images with different resolutions that has a relatively high resolution, thereby generating high-resolution images that contain different types of information; and a combination step (ST3) to combine the high-resolution images that contain the different types of information, thereby generating one or more combined high-resolution images, the resolution enhancement step (ST3) includes: Extracting low-frequency and high-frequency components from images that have a relatively low resolution; Extracting, from the image which has a relatively high resolution, a low-frequency component and a high-frequency component of the image; Enlarging the low-frequency components of images with a relatively low resolution to the same resolution as an image with a relatively high resolution, Combining the enlarged low-frequency components and the low-frequency component of the image, which has a relatively high resolution, by weighted addition to generate a combined high-frequency component; and Combining the combined low-frequency components and the combined high-frequency components to create a high-resolution image.

Images