Imaging device, method, and computer program

Abstract

An imaging device (1) having a color filter for achieving high spatial resolution and high temporal resolution, comprising: a V * H pixel array unit (11) for receiving light through pixels, where the pixels comprise n * m divided pixels; a sampling unit (12) for generating sampled data (30, 501, 511, 521, 531, 541, 701, 711, 901, 911, 931, 941) such that the positional relationships of the divided pixels of each pixel (111, 112, 113, 114) included in the pixel group used in one sampling are different from each other; and a reconstruction unit (13) for reconstructing the sampling data (30, 501, 511, 521, 531, 541, 701, 711, 901, 911, 931, 941) for each specific wavelength component.

Description

Technical Field

[0001] The present invention relates to the field of imaging devices, methods and computer programs. Background Art

[0002] Non-patent document 1 (Michael et al., “INCREASING IMAGING RESOLUTION BY COVERING YOUR SENSOR”, 2011 18th IEEE International Conference on Image Processing, 2011) discloses a non-regular sampling method for improving resolution without increasing pixel throughput in an image sensor system. Non-patent document 2 (Markus Jonscher et al., “Dynamic Non-Regular Sampling Sensor Using Frequency Selective Reconstruction”, IEEE Transactions on Circuits and Systems for Video Technology, October 2019, Vol. 29, No. 10) discloses a method in which different pixel positions are read in different frames in an image sensor system by dynamic non-regular sampling. Non-patent document 3 (“Sony Releases Smartphone with Industry’s Highest 48-Megapixel Effective Pixels Equipped with Stacked CMOS Image Sensor”, [Online], July 23, 2018, [Search Time: February 23, 2018], Internet,<https: / / www.sony.net / SonyInfo / News / Press / 201807 / 18-060E / > ) discloses a color filter array in which adjacent 2×2 pixels have the same color to achieve high resolution in a smartphone image sensor. Summary of the invention

[0003] The imaging process of many applications, such as smartphone cameras and surveillance cameras, requires high spatial resolution and high temporal resolution. However, due to the limitation of pixel throughput, there has always been a trade-off problem in the existing technology, such as high spatial resolution but low temporal resolution (or vice versa).

[0004] The present invention is proposed in view of the above circumstances. An object of the present invention is to provide an imaging device, method and computer program that achieve high spatial resolution and high temporal resolution in imaging processing.

[0005] In order to solve the above-mentioned problem, according to a first aspect of the present invention, there is provided an imaging device, comprising: a V×H pixel array unit for receiving light through a plurality of pixels, wherein the plurality of pixels include a color filter for light having a specific wavelength component, and each pixel is composed of n×m segmented pixels receiving light having the same wavelength, n≥2, m≥2; a sampling unit for sampling light having a specific wavelength component corresponding to each pixel through a segmented pixel and generating sampling data, wherein the positional relationship of the segmented pixels of each pixel included in a pixel group used in one sampling is different from each other; and a reconstruction unit for reconstructing the sampling data for each specific wavelength component and acquiring pixel data corresponding to all segmented pixels.

[0006] According to this aspect, compared with an image sensor having a color filter array (in which adjacent n×m pixels have the same color), the throughput can be reduced to 1 / (n×m), and the occurrence of aliasing can be prevented. Therefore, a high-quality imaging system with both high spatial resolution and high temporal resolution can be realized.

[0007] According to a second aspect of the present invention related to the first aspect, the sampling unit is further configured to perform n×m samplings, wherein in each of the n×m samplings, positional relationships of divided pixels of a specific pixel are different from each other.

[0008] According to this aspect, aliasing caused by regular sampling is reduced, and high-resolution imaging processing can be achieved.

[0009] According to a third aspect of the present invention related to the first aspect or the second aspect, the sampling unit is also used to project previous sampling data onto the sampling data and generate first projection data; the reconstruction unit is also used to reconstruct the first projection data for each specific wavelength component and obtain pixel data corresponding to all segmented pixels.

[0010] According to this aspect, since the pixel throughput does not increase as the amount of data to be sampled increases, the reconstruction accuracy can be improved and a high-quality imaging system can be realized.

[0011] According to a fourth aspect of the present invention related to the first aspect or the second aspect, the sampling unit is further used to generate second projection data based on previous sampling data, the sampling data and a motion vector, wherein the motion vector is obtained based on the following data: the sampling data and pixel data obtained based on the previous sampling data; the reconstruction unit is further used to reconstruct the second projection data for each specific wavelength component and obtain pixel data corresponding to all segmented pixels.

[0012] According to this aspect, since the movement of a camera equipped with an imaging device (camera shake, etc.) and the movement of a subject can be reflected in projection data, the reconstruction accuracy can be further improved, and a high-quality imaging system can be realized.

[0013] According to a fifth aspect of the present invention, a method for imaging includes: receiving light through a plurality of pixels, wherein the plurality of pixels include a color filter for light having a specific wavelength component, each pixel being composed of n×m segmented pixels receiving light having the same wavelength, n≥2, m≥2; sampling light having a specific wavelength component corresponding to each pixel through a segmented pixel and generating sampling data, wherein a positional relationship of the segmented pixels of each pixel included in a pixel group used in one sampling is different from each other; reconstructing the sampling data for each specific wavelength component and acquiring pixel data corresponding to all segmented pixels.

[0014] According to a sixth aspect of the present invention, a computer program for imaging causes a computer of an imaging device to perform the following operations: receiving light through a plurality of pixels, wherein the plurality of pixels include a color filter for light having a specific wavelength component, each pixel being composed of n×m segmented pixels receiving light having the same wavelength, n≥2, m≥2; sampling light having a specific wavelength component corresponding to each pixel through a segmented pixel and generating sampling data, wherein a positional relationship of the segmented pixels of each pixel included in a pixel group used in one sampling is different from each other; and reconstructing the sampling data for each specific wavelength component and acquiring pixel data corresponding to all segmented pixels.

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly describes the drawings necessary for illustrating the embodiments. Obviously, the drawings described below only illustrate some embodiments of the present application, and those skilled in the art can make other drawings based on these drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram showing a functional configuration of an imaging device according to a first embodiment;

[0017] Figure 2 is a schematic diagram showing an image sensor according to a first embodiment;

[0018] Figure 3 is a schematic diagram describing sampling through a color filter according to a first embodiment;

[0019] Figure 4 is a schematic diagram showing a processing flow of an imaging device according to a first embodiment;

[0020] Figure 5is a schematic diagram describing acquisition of pixel data according to the first embodiment;

[0021] Figure 6 is a schematic diagram showing a processing flow of an imaging device according to a second embodiment;

[0022] Figure 7 is a schematic diagram describing acquisition of pixel data according to the second embodiment;

[0023] Figure 8 is a schematic diagram showing a processing flow of an imaging device according to a third embodiment;

[0024] Fig. 9 is a schematic diagram describing acquisition of pixel data according to the third embodiment. DETAILED DESCRIPTION

[0025] First embodiment

[0026] In the following, reference Figures 1 to 5 An imaging device 1 according to a first embodiment is described.

[0027] (Functional Configuration of Imaging Device 1)

[0028] Figure 1 1 is a schematic diagram showing the functional configuration of an imaging device 1 according to a first embodiment. Figure 2 and Figure 3 The functional configuration of the imaging device 1 is described, wherein: Figure 2 shows the configuration of the image sensor, Figure 3 Describes sampling through color filters.

[0029] The imaging device 1 includes a V×H (vertical×horizontal) pixel array unit 11, a sampling unit 12, and a reconstruction unit 13. The V×H pixel array unit 11 includes a plurality of pixels for receiving light, each pixel including a color filter (wavelength selection element). The pixel group includes p×q pixels (p≥2, q≥2) for light having a specific wavelength component. The light having a specific wavelength component includes light having red (R), blue (B), and green (Gr or Gb) wavelength components. Each pixel includes n×m segmented pixels (n≥2 and m≥2) that receive light having the same wavelength.

[0030] For example, you can Figure 2The V×H pixel array unit 11 and the sampling unit 12 are implemented as shown. That is, the image sensor 20 includes the pixel array unit 11 (the pixel array unit 11 includes a color filter), a pixel controller 121, and an analog-to-digital converter (ADC) 125. The pixel array unit 11 stores the charge acquired by the photoelectric conversion. The pixel array unit 11 includes pixels PC arranged in a grid shape in the row direction (row direction, RD) and the column direction (column direction, CD). The pixel controller 121 includes a timing control circuit 21 and a vertical scanning circuit 22. The timing control circuit 21 controls the timing of reading and accumulating the pixel PC. The vertical scanning circuit 22 is connected to the pixel array unit 11 via the horizontal control line Hlin to control the reading of the pixel PC. The ADC 125 includes a reference voltage generating circuit 23, a load circuit 24, and a column ADC circuit 25. The reference voltage generating circuit 23 outputs a reference voltage VREF to the column ADC circuit. The load circuit 24 is connected to the pixel array unit 11 via the vertical signal line Vlin, and reads the pixel signal according to the pixel PC. The column ADC circuit 25 converts the pixel signal into a digital signal according to the reference voltage VREF of the reference voltage generating circuit 23 and outputs an output signal S. Figure 2 The detailed configuration of the pixel array unit 11 of the image sensor 20 is as follows: Figure 3 This is described in the figure on the left.

[0031] Figure 3 The left figure shows a pixel group, which is a part of the pixel array unit 11. For example, Figure 3 As shown, one pixel group includes four pixels (p=2, q=2) arranged in a grid. That is, one pixel group includes pixel 111, pixel 112, pixel 113, and pixel 114. Pixel 111 includes a color filter that receives light having a red wavelength. Pixel 112 and pixel 113 include color filters that receive light having a green wavelength. Pixel 114 includes a color filter that receives light having a blue wavelength.

[0032] In addition to the above-mentioned 2×2 (p=2, q=2) Bayer arrangement, the pixel array may also be an RGBW array or a RYYB array, etc.

[0033] When n=2 and m=2, the pixel 111 includes four segmented pixels (R0 to R3) arranged in a grid. The pixel 112 includes four segmented pixels (Gr0 to Gr3) arranged in a grid. The pixel 113 includes four segmented pixels (Gb0 to Gb3) arranged in a grid. The pixel 114 includes four segmented pixels (B0 to B3) arranged in a grid.

[0034] The sampling unit 12 is used to sample light having a specific wavelength component corresponding to each pixel through a divided pixel in each pixel and generate sampling data 30. Figure 3 As shown, the sampling unit 12 includes a pixel controller 121 and a circuit. The circuit includes pixel selection circuits 122-1 to 122-4, switching circuits 123-1 and 123-2, and switches 124-1 to 124-4. The pixel controller 121 inputs a signal to the switching circuit 123 and controls the switch 124. The pixel controller 121 selects one of the divided pixels in each pixel by inputting a signal to the pixel selection circuit 122, and samples light having a specific wavelength component corresponding to each pixel.

[0035] The pixel selection circuits 122-1 to 122-4 and the pixels 111 to 114 are configured so that the positional relationship of the split pixels of each pixel included in the pixel group used in one sampling is different from each other. In other words, the pixel selection circuits 122-1 to 122-4 and the pixels 111 to 114 are configured so that the positional relationship of the split pixels of one pixel used for one sampling is different from the positional relationship of the split pixels of other pixels in the pixel group including the one pixel. Here, a plurality of pixels included in one pixel group are adjacent to each other.

[0036] For example, Figure 3 As shown, the pixel selection circuit 122-1 is connected to select the divided pixel R0 of the pixel 111, the divided pixel Gr1 of the pixel 112, the divided pixel Gb3 of the pixel 113, and the divided pixel B2 of the pixel 114. The pixel selection circuit 122-2 is connected to select the divided pixel R2 of the pixel 111, the divided pixel Gr0 of the pixel 112, the divided pixel Gb1 of the pixel 113, and the divided pixel B3 of the pixel 114. The pixel selection circuit 122-3 is connected to select the divided pixel R1 of the pixel 111, the divided pixel Gr3 of the pixel 112, the divided pixel Gb2 of the pixel 113, and the divided pixel B0 of the pixel 114. The pixel selection circuit 122-4 is connected to select the divided pixel R3 of the pixel 111, the divided pixel Gr2 of the pixel 112, the divided pixel Gb0 of the pixel 113, and the divided pixel B1 of the pixel 114.

[0037] Here, it is assumed that a signal is input to the pixel selection circuit 122-1 in one sampling, and the segmented pixels of each pixel are sampled. At this time, the position of the segmented pixel of the pixel 111 is R0 at the upper left of the pixel 111. The position of the segmented pixel of the pixel 112 is Gr1 at the upper right of the pixel 113. The position of the segmented pixel of the pixel 113 is Gb3 at the lower right of the pixel 113. The position of the segmented pixel of the pixel 114 is B2 at the lower left of the pixel 114. As described above, the positional relationship of the segmented pixels of each pixel included in the pixel group used in one sampling is different from each other.

[0038] The connection relationship between the pixel selection circuits 122-1 to 122-4 and the divided pixels in the pixel can be configured so that the positional relationship of the divided pixels of each pixel included in the pixel group used in one sampling is different from each other. That is, the connection relationship between the pixel selection circuits 122-1 to 122-4 and the divided pixels in the pixel is not limited to Figure 3 The connection relationship shown.

[0039] The reconstruction unit 13 is used to reconstruct the sampling data 30 generated by the sampling unit 12 for each specific wavelength component and obtain pixel data 31 corresponding to all the segmented pixels.

[0040] As described above, the functional configuration of the imaging device 1 according to the first embodiment has been described. Further, the imaging device 1 may also include an image signal processor for performing image signal processing (e.g., Bayer conversion) on the acquired pixel data 31 and outputting a color image. However, the imaging device 1 may perform image signal processing on the pixel data 31 acquired by other techniques to achieve the output of a color image, and is not limited to the above-mentioned embodiments. In addition, the imaging device 1 may also include a storage unit for storing data generated by the imaging device 1 (e.g., sampling data 30 or pixel data 31). The storage unit may be, for example, a flash memory, a dynamic random access memory (DRAM), a static random access memory (SRAM), or the like.

[0041] (Processing Flow of Imaging Device 1)

[0042] Figure 4 1 is a schematic diagram showing a process flow of the imaging device 1 according to the first embodiment. Figure 3 and Figure 5 The processing flow of the imaging device 1 is described to describe the acquisition of pixel data.

[0043] First, when the imaging operation of the imaging device 1 is performed, the V×H pixel array unit 11 receives light through a color filter including p×q pixels for light having a specific wavelength component (step S401 ).

[0044] The sampling unit 12 samples light having a specific wavelength component corresponding to each pixel through one divided pixel in each pixel, and generates the sampling data 30 (step S402 ).

[0045] Specifically, when the pixel controller 121 of the sampling unit 12 inputs a signal to the pixel selection circuit 122-1, light having a specific wavelength component corresponding to each pixel can be sampled through the split pixels connected to the pixel selection circuit 122-1. Here, when the pixel controller 121 sequentially inputs signals to the switching circuits 123-1 and 123-2, the switches 124-1 to 124-4 are switched. Then, the sampling unit 12 samples light having a specific wavelength component corresponding to each pixel through each split pixel. According to Figure 3 In the example shown, the sample data 30 is acquired in one sampling at time T=t. The positions of the divided pixels of each of the pixels 111 to 114 in the sample data 30 are different from each other. The divided pixel receiving the light with a red wavelength is R0. The divided pixels receiving the light with a green wavelength are Gr1 and Gb3. The divided pixel receiving the light with a blue wavelength is B2.

[0046] Next, the ADC 125 of the imaging device 1 AD-converts the sampled data 30 (step S403 ).

[0047] The reconstruction unit 13 reconstructs the AD-converted sampled data 30 for each specific wavelength component and acquires the pixel data 31 corresponding to all the segmented pixels (step S404). The reconstruction process can be performed by using a technique of interpolating missing data in the sampled data 30 for each individual color, or by using a technique of identifying the main frequency of the spectrum of each specific wavelength component of the sampled data 30 for each individual color and estimating an appropriate weight.

[0048] The imaging device 1 performs image signal processing (eg, Bayer conversion) on the acquired pixel data 31 and outputs a color image (step S405 ).

[0049] When the imaging process of the imaging device 1 continues at time T=t+1 (step S406 : Yes), the imaging device 1 performs steps S401 to S405 again.

[0050] Specifically, it is assumed that in a sampling at time T=t+1, the pixel controller 121 sends Figure 3 The pixel selection circuit 122-2 in the input signal is sampled. Figure 5 The sample data 511 at time T=t+1 is obtained by configuring Figure 3The pixel array shown is obtained. At time T=t+1, the positions of the split pixels of each of the pixels 111 to 114 in the sampled data 30 are different from each other. The split pixel that receives light with a red wavelength is R2. The split pixels that receive light with a green wavelength are Gr0 and Gb1. The split pixel that receives light with a blue wavelength is B3. The reconstruction unit 13 reconstructs the sampled data 511 for each specific wavelength component and obtains pixel data 512 corresponding to all the split pixels. The imaging device 1 performs image signal processing on the acquired pixel data 512 and outputs a color image.

[0051] If the imaging process continues after time T=t+2 (step S406: Yes), similarly, the imaging device 1 executes steps S401 to S405 again and outputs pixel data (522, 542), such as Figure 5 shown.

[0052] The sampling data 541 at time T=t+4 is generated using the pixel selection circuit 122-1. If the imaging process continues after time T=t+5, the sampling data is generated by repeating the pixel selection circuits 122-1 to 122-4.

[0053] like Figure 5 As shown, when the sampling unit 12 performs n×m samplings from time T=t to T=t+3 ( Figure 5 When n=2, m=2 in , dynamic irregular sampling is performed so that n×m times is one cycle. That is, in each sampling in the n×m samplings, the positional relationship of the split pixels of the pixels of the color filter with the same wavelength of light is different from each other. In other words, in the n×m samplings, the positional relationship of the split pixels of the specific pixels used for one sampling is different from the positional relationship of the split pixels of the specific pixels used for other samplings. Specifically, the position of the split pixels of the pixels of the red (R) light of the sampling data 501 is R0. At this time, the position of the split pixels of the pixels of the red light of the sampling data 511 is R2, which is different from R0. Similarly, the position of the split pixels of the pixels of the red light of the sampling data 521 is R1, which is different from R2 and R0. The position of the split pixels of the pixels of the red light of the sampling data 531 is R3, which is different from R2, R0 and R1. The same applies to green (Gr, Gb) and blue (B) light. Therefore, the aliasing caused by regular sampling is reduced, and high-resolution imaging processing can be achieved.

[0054] If the imaging process of the imaging device 1 does not continue at the time T (step S406 : No), the imaging operation of the imaging device 1 ends.

[0055] As described above, the processing contents from the execution of the imaging operation of the imaging device 1 to the end of the imaging operation are described. Although steps S401 to S406 have been described for the sake of convenience, other operations may be performed in parallel during one operation. For example, at time T=t, the reconstruction process in step S404 or the image signal processing and image output process in step S405 may be performed in parallel with the sampling process at the next time T=t+1.

[0056] In the above description, whether the sampled data to be digitized (eg, reconstructed) is subjected to AD conversion may be omitted. The omission is for convenience, and those skilled in the art may understand that the sampled data to be digitized is subjected to AD conversion.

[0057] It should be noted that each process of the above-mentioned imaging device 1 can be stored in a computer-readable recording medium in the form of a computer program, and the program can be read and executed by the computer (processor or central processing unit (CPU)) of the imaging device 1, and the above-mentioned processing can be performed. Here, the computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, etc.

[0058] Further, the above program can be used to implement a part of the above functions. Further, the program can be a difference file (difference program), which can implement the above functions in combination with the program already recorded in the computer system. Further, the computer can include one computer or multiple computers, which can be connected to each other so as to communicate with each other.

[0059] (Effect)

[0060] The imaging device 1 according to the first embodiment includes: a V×H pixel array unit 11 for receiving light through a plurality of pixels, wherein the plurality of pixels include a color filter for light having a specific wavelength component, and each of the pixels 111 to 114 is composed of n×m divided pixels receiving light having the same wavelength, n≥2 and m≥2; a sampling unit 12 for sampling light having a specific wavelength component corresponding to each pixel through one divided pixel and generating sampling data 30, wherein the positional relationship of the divided pixels of each pixel included in a pixel group used in one sampling is different from each other; and a reconstruction unit 13 for reconstructing the sampling data 30 for each specific wavelength component and acquiring pixel data corresponding to all divided pixels.

[0061] According to this aspect, compared with an imaging sensor having a color filter array (in which adjacent n×m pixels have the same color), the throughput can be reduced to 1 / (n×m), and the occurrence of aliasing can be prevented. Therefore, a high-quality imaging system with both high spatial resolution and high temporal resolution can be realized.

[0062] Second embodiment

[0063] In the following, reference Figure 6 and Figure 7 An imaging device 1 according to a second embodiment is described. Figure 6 is a schematic diagram showing a process flow of the imaging device according to the second embodiment. Figure 7 2 is a schematic diagram showing acquisition of pixel data according to the second embodiment. Unless otherwise specified, the functional configuration of the imaging device 1 according to the second embodiment is configured and functions in the same manner as each component of the imaging device 1 according to the first embodiment.

[0064] (Processing Flow of Imaging Device 1)

[0065] The light receiving process (step S601 ) and the sampling process (step S602 ) after the imaging operation of the imaging device 1 is performed are the same as those of the imaging device 1 in the first embodiment (steps S401 and S402 ).

[0066] Assume that the sampling data 701 has been generated at time T=t. At time T=t, the ADC 125 AD-converts the sampling data 701. The storage unit of the imaging device 1 stores the sampling data 701 that has been AD-converted.

[0067] Next, at time T=t+1, the sampling unit 12 generates the sampling data 711 through the sampling process (step S602). The ADC 125 of the imaging device 1 performs AD conversion on the sampling data 711 (step S603).

[0068] Next, the sampling unit 12 projects the previous sampling data 701 at time T=t onto the sampling data 711 at time T=t+1 to generate the first projection data 712 (step S604). Figure 7 As shown, the first projection data 712 is generated by the logical sum of the previous sampling data 701 and the sampling data 711 which have been AD converted and stored in the storage unit.

[0069] The reconstruction unit 13 reconstructs the first projection data 712 for each specific wavelength component and obtains the pixel data 713 corresponding to all the segmented pixels (step S605). The reconstruction process can be performed by using a technique for interpolating missing data in the first projection data 712 for each individual color, or by using a technique for identifying the main frequency of the spectrum of each specific wavelength component of the first projection data 712 for each individual color and estimating an appropriate weight.

[0070] The imaging device 1 performs image signal processing (eg, Bayer conversion) on the acquired pixel data 713 and outputs a color image (step S606 ).

[0071] When the imaging process of the imaging device 1 continues at time T=t+2 (step S607 : Yes), the imaging device 1 performs steps S601 to S606 again.

[0072] If the imaging process continues after time T=t+3 (step S607: Yes), similarly, the imaging device 1 executes steps S601 to S606 again and outputs pixel data (733, 743), such as Figure 7 shown.

[0073] If the imaging process of the imaging device 1 does not continue at the time T (step S607 : No), the imaging operation of the imaging device 1 ends.

[0074] As described above, the processing contents from the execution of the imaging operation of the imaging device 1 in the second embodiment to the end of the imaging operation are described. Although steps S601 to S607 have been described for the sake of convenience, other operations can be performed in parallel during one operation. For example, at time T=t, the reconstruction process in step S605 or the image signal processing and image output process in step S606 can be performed in parallel with the sampling process at the next time T=t+1.

[0075] In the above description, whether the sampled data to be digitized (such as projection) is subjected to AD conversion may be omitted. The omission is for convenience, and those skilled in the art can understand that the sampled data to be digitized is subjected to AD conversion.

[0076] (Effect)

[0077] In the imaging device 1 according to the second embodiment, the sampling unit 12 is also used to project the previous sampling data 701 onto the sampling data 711 and generate first projection data 712; the reconstruction unit 13 is also used to reconstruct the first projection data 712 for each specific wavelength component and obtain pixel data 713 corresponding to all segmented pixels.

[0078] According to this aspect, since the pixel throughput does not increase as the amount of data to be sampled increases, the reconstruction accuracy can be improved and a high-quality imaging system can be realized.

[0079] Third embodiment

[0080] In the following, reference Figure 8 and Fig. 9 An imaging device 1 according to a third embodiment is described. Figure 8 is a schematic diagram showing a process flow of the imaging device according to the third embodiment. Fig. 9 2 is a schematic diagram showing acquisition of pixel data according to the third embodiment. Unless otherwise specified, the functional configuration of the imaging device 1 according to the third embodiment is configured and functions in the same manner as each component of the imaging device 1 according to the first embodiment.

[0081] The light receiving process (step S801 ) and the sampling process (step S802 ) after the imaging operation of the imaging device 1 is performed are the same as those of the imaging device 1 in the first embodiment (steps S401 and S402 ).

[0082] Assume that the sampling data 901 has been generated at time T=t. At time T=t, the ADC 125 of the imaging device 1 AD-converts the sampling data 901. The storage unit of the imaging device 1 stores the AD-converted sampling data 901. The reconstruction unit 13 reconstructs the AD-converted sampling data 901 for each specific wavelength component and obtains pixel data 903 corresponding to all divided pixels.

[0083] Next, at time T=t+1, the sampling unit 12 generates the sampling data 911 through the sampling process (step S802). The ADC 125 of the imaging device 1 performs AD conversion on the sampling data 911 (step S803).

[0084] Next, the sampling unit 12 acquires a motion vector 91 based on the pixel data 903 acquired based on the AD-converted sampling data 901 stored in the storage unit and the AD-converted sampling data 911. Here, the motion vector is based on the movement of the camera provided with the imaging device 1 (camera shake, etc.) and the movement of the object.

[0085] For example, the sampling unit 12 compares the pixel data 903 and the sample data 911, and calculates a difference between each of R0 to R3 of the pixel data 903 and R2 of the sample data 911. Similarly, the sampling unit 12 calculates a difference between each of Gr0 to Gr3 of the pixel data 903 and Gr0 of the sample data 911. Similarly, the sampling unit 12 calculates a difference between each of Gb0 to Gb3 of the pixel data 903 and Gb1 of the sample data 911. Similarly, the sampling unit 12 calculates a difference between each of B0 to B3 of the pixel data 903 and B3 of the sample data 911. When the difference between Gb0 of the pixel data 903 and Gb1 of the sample data 911 is the smallest among all the differences, or when the difference between B2 of the pixel data 903 and B3 of the sample data 911 is the smallest among all the differences, a rightward motion is detected, as indicated by the motion vector 91.

[0086] The sampling unit 12 generates the second projection data 912 based on the previous sampling data 901 at time T=t, the sampling data 911 at time T=t+1, and the motion vector 91 (step S804). Specifically, the sampling unit 12 projects the acquired data onto the AD-converted sampling data 911 by moving the AD-converted sampling data 901 stored in the storage unit by one divided pixel in the direction indicated by the motion vector 91, so as to generate the second projection data 912. In other words, the sampling unit 12 generates the second projection data 912 by logically summing the data acquired by moving the sampling data 901 by one divided pixel in the direction indicated by the motion vector 91 and the sampling data 911.

[0087] like Fig. 9As shown, the motion vector 91 indicates a rightward movement. At this time, the data obtained by moving the sample data 901 only one split pixel to the right is projected onto the sample data 911. The data obtained by moving the split pixel R0 of the sample data 901 only one split pixel to the right is R1. There is no data of the same color in Gr0 to Gr3, wherein the split pixel Gr1 of the sample data 901 is only one split pixel moved to the right. There is no data of the same color in Gb0 to Gb3, wherein the split pixel Gb3 of the sample data 901 is only one split pixel moved to the right. The data obtained by moving the split pixel B2 of the sample data 901 only one split pixel to the right is B3. Therefore, the data obtained by moving the sample data 901 only one split pixel to the right is (R1, B3). This data is projected onto the sample data 911. At this time, the split pixels corresponding to the red (R) wavelength component in the second projection data 912 include R1 and R2. The segmented pixels corresponding to the green (Gr) wavelength component in the second projection data 912 include Gr0. The segmented pixels corresponding to the green (Gb) wavelength component in the second projection data 912 include Gb1. In the data obtained by shifting the sample data 901 to the right by only one segmented pixel, the segmented pixel corresponding to the blue (B) wavelength component is B3, but B3 already exists in the sample data 911. At this time, the segmented pixel (B3) corresponding to the blue (B) wavelength component in the shifted data is not projected onto the sample data 911. Therefore, the segmented pixels corresponding to the blue (B) wavelength component in the second projection data 912 include B3.

[0088] The reconstruction unit 13 reconstructs the second projection data 912 for each specific wavelength component and obtains the pixel data 913 corresponding to all the segmented pixels (step S805). The reconstruction process can be performed by using a technique for interpolating missing data in the second projection data 912 for each individual color, or by using a technique for identifying the main frequency of the spectrum of each specific wavelength component of the second projection data 912 for each individual color and estimating an appropriate weight.

[0089] The imaging device 1 performs image signal processing (eg, Bayer conversion) on the acquired pixel data 913 and outputs a color image (step S806 ).

[0090] When the imaging process of the imaging device 1 continues at time T=t+2 (step S807: Yes), the imaging device 1 executes steps S801 to S806 again.

[0091] If the imaging process continues after time T=t+3 (step S807: Yes), similarly, the imaging device 1 executes steps S801 to S806 again and outputs pixel data (933, 943), as shown in FIG. Fig. 9 shown.

[0092] exist Fig. 9 In the example shown, the acquired motion vector 94 does not include any directional component (motion vector: No). At this time, in step S804 , the sampling unit 12 directly projects the sampling data 931 onto the sampling data 941 to generate second projection data 942 .

[0093] If the imaging process of the imaging device 1 does not continue at the time T (step S807 : No), the imaging operation of the imaging device 1 ends.

[0094] As described above, the processing contents from the execution of the imaging operation of the imaging device 1 in the third embodiment to the end of the imaging operation are described. Although steps S801 to S807 have been described for the sake of convenience, other operations can be performed in parallel during one operation. For example, at time T=t, the reconstruction process in step S805 or the image signal processing and image output process in step S806 can be performed in parallel with the sampling process at the next time T=t+1.

[0095] In the above description, whether the sampled data to be digitized (such as projection) is subjected to AD conversion may be omitted. The omission is for convenience, and those skilled in the art can understand that the sampled data to be digitized is subjected to AD conversion.

[0096] (Effect)

[0097] In the imaging device 1 according to the third embodiment, the sampling unit 12 is further used to generate second projection data 912 based on the previous sampling data 901, the sampling data 911 and the motion vector 91, wherein the motion vector 91 is obtained based on the following data: the sampling data 911 and the pixel data 903 obtained based on the previous sampling data 901; the reconstruction unit 13 is further used to reconstruct the second projection data 912 for each specific wavelength component and obtain the pixel data 913 corresponding to all the segmented pixels.

[0098] According to this aspect, since the movement of a camera equipped with an imaging device (camera shake, etc.) and the movement of a subject can be reflected in projection data, the reconstruction accuracy can be further improved, and a high-quality imaging system can be realized.

[0099] As mentioned above, some embodiments of the present invention have been described, but all of these embodiments are presented as examples and are not intended to limit the scope of the present invention. These embodiments can be implemented in various other embodiments, and various omissions, substitutions and changes can be made without departing from the gist of the present invention. These embodiments and their variations are included in the scope and gist of the present invention, and are also included in the scope of the present invention described in the claims and their equivalents.

Claims

1. An imaging device, characterized in that: include: A V×H pixel array unit for receiving light through a plurality of pixels, wherein the plurality of pixels include a color filter for light having a specific wavelength component, each pixel being composed of n×m segmented pixels receiving light having the same wavelength, n≥2, m≥2; a sampling unit for sampling light having a specific wavelength component corresponding to each pixel through one divided pixel and generating sampling data, wherein positional relationships of the divided pixels for each pixel included in a pixel group used in one sampling are different from each other; The reconstruction unit is used to reconstruct the sampled data for each specific wavelength component and obtain pixel data corresponding to all segmented pixels.

2. The imaging device according to claim 1, characterized in that The sampling unit is further configured to perform n×m samplings, wherein in each of the n×m samplings, positional relationships of divided pixels of a specific pixel are different from each other.

3. The imaging device according to claim 1 or 2, characterized in that: The sampling unit is also used to project the previous sampling data onto the sampling data and generate first projection data; The reconstruction unit is further configured to reconstruct the first projection data for each specific wavelength component and acquire pixel data corresponding to all segmented pixels.

4. The imaging device according to claim 1 or 2, characterized in that: The sampling unit is further configured to generate second projection data based on previous sampling data, the sampling data and a motion vector, wherein the motion vector is obtained based on the following data: the sampling data and pixel data obtained based on the previous sampling data; The reconstruction unit is further configured to reconstruct the second projection data for each specific wavelength component and acquire pixel data corresponding to all segmented pixels.

5. A method, characterized in that include: receiving light through a plurality of pixels, wherein the plurality of pixels include color filters for light having a specific wavelength component, each pixel being composed of n×m segmented pixels receiving light having the same wavelength, n≥2, m≥2; sampling light having a specific wavelength component corresponding to each pixel by one divided pixel and generating sampling data, wherein positional relationships of the divided pixels for each pixel included in a pixel group used in one sampling are different from each other; The sampled data is reconstructed for each specific wavelength component and pixel data corresponding to all divided pixels is acquired.

6. The method according to claim 5, characterized in that The generating of the sampling data includes performing sampling n×m times, wherein positional relationships of divided pixels of a specific pixel are different from each other in each of the n×m times of sampling.

7. The method according to claim 5 or 6, characterized in that: Generating the sampling data includes projecting the previous sampling data onto the sampling data and generating first projection data; The acquiring pixel data includes reconstructing the first projection data for each specific wavelength component and acquiring pixel data corresponding to all divided pixels.

8. The method according to claim 5 or 6, characterized in that: Generating the sampling data includes generating second projection data based on previous sampling data, the sampling data and a motion vector, wherein the motion vector is obtained based on the following data: the sampling data and pixel data obtained based on the previous sampling data; The acquiring pixel data includes reconstructing the second projection data for each specific wavelength component and acquiring pixel data corresponding to all divided pixels.

9. A computer program, characterized in that The computer of the imaging device performs the following operations: receiving light through a plurality of pixels, wherein the plurality of pixels include color filters for light having a specific wavelength component, each pixel being composed of n×m segmented pixels receiving light having the same wavelength, n≥2, m≥2; sampling light having a specific wavelength component corresponding to each pixel by one divided pixel and generating sampling data, wherein positional relationships of the divided pixels for each pixel included in a pixel group used in one sampling are different from each other; The sampled data is reconstructed for each specific wavelength component and pixel data corresponding to all divided pixels is acquired.

10. The computer program according to claim 9, characterized in that The generating of the sampling data includes performing sampling n×m times, wherein positional relationships of divided pixels of a specific pixel are different from each other in each of the n×m times of sampling.

11. The computer program according to claim 9 or 10, characterized in that Generating the sampling data includes projecting the previous sampling data onto the sampling data and generating first projection data; The acquiring pixel data includes reconstructing the first projection data for each specific wavelength component and acquiring pixel data corresponding to all divided pixels.

12. The computer program according to claim 9 or 10, characterized in that Generating the sampling data includes generating second projection data based on previous sampling data, the sampling data and a motion vector, wherein the motion vector is obtained based on the following data: the sampling data and pixel data obtained based on the previous sampling data; The acquiring pixel data includes reconstructing the second projection data for each specific wavelength component and acquiring pixel data corresponding to all divided pixels.

Images