Imaging device, imaging method, and program

Abstract

The present technology pertains to an imaging device, an imaging method, and a program capable of improving image quality. An imaging device is provided with: a pixel array unit in which pixels, which generate charges corresponding to the amount of received light and output signals corresponding to the charges, are arranged in a matrix in a row direction and a column direction; a reading unit that divides the pixel array unit into a plurality of blocks and reads signals from pixels at the same position in the blocks from each of the plurality of blocks; and a rearrangement unit performing rearrangement so that the arrangement of the signals read by the reading unit is equivalent to the arrangement of the pixels in the pixel array unit.

Description

Technical Field

[0001] The present technology relates to an imaging device, an imaging method, and a program, and for example, to an imaging device, an imaging method, and a program that can apply existing processing even when the method of reading from pixels is changed. Background Art

[0002] A general imaging element has a Bayer array color filter. In the Bayer array, green (G) is arranged in a checkered pattern, and red (R) and blue (B) are arranged in line order. Patent document 1 proposes a technology that enables an existing processing unit for the Bayer array to be used for signal processing of an imaging element including a white color filter array.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2014-161022 Summary of the invention

[0006] Technical issues to be solved by the invention

[0007] A method called raster scanning is used as a method of reading pixel signals from imaging elements in the Bayer array. A processing unit that processes the pixel signals read by raster scanning is generally used.

[0008] On the other hand, when reading is performed by raster scanning, a time difference occurs between a line read first and a line read last, and distortion may occur in the image.

[0009] Even when a reading method for preventing distortion from occurring in an image is applied, it is desirable to be able to apply existing processing.

[0010] The present technology has been developed in view of such circumstances, and aims to improve image quality.

[0011] Technical solutions for solving technical problems

[0012] An imaging device according to one aspect of the present technology is an imaging device comprising: a pixel array unit in which pixels are arranged in a matrix in row and column directions, the pixels generating charges corresponding to the amount of light received and outputting signals corresponding to the charges; a reading unit that divides the pixel array unit into a plurality of blocks and reads the signals from pixels at the same position in each of the plurality of blocks; and a rearrangement unit that rearranges so that the arrangement of the signals read by the reading unit is equivalent to the arrangement of the pixels in the pixel array unit.

[0013] An imaging method according to another aspect of the present technology is the following imaging method, wherein an imaging device has a pixel array unit, in which pixels are arranged in a matrix in row and column directions, and the pixels generate charges corresponding to the amount of light received and output signals corresponding to the charges, and the imaging method includes: dividing the pixel array unit into a plurality of blocks by the imaging device, and reading the signals from pixels at the same position in each of the plurality of blocks; and rearranging so that the arrangement of the read signals is equivalent to the arrangement of pixels in the pixel array unit.

[0014] A program according to another aspect of the present technology is a program that causes a computer controlling an imaging device to execute processing, wherein the imaging device has a pixel array unit, in which pixels are arranged in a matrix in row and column directions, and the pixels generate charges corresponding to the amount of light received and output signals corresponding to the charges, and the processing includes the following steps: dividing the pixel array unit into a plurality of blocks, and reading the signals from pixels at the same position in each of the plurality of blocks; and rearranging so that the arrangement of the read signals is equivalent to the arrangement of pixels in the pixel array unit.

[0015] In an imaging device, an imaging method, and a program according to multiple aspects of the present technology, a pixel array unit is provided, in which pixels are arranged in a matrix in row and column directions, the pixels generate charges corresponding to the amount of light received and output signals corresponding to the charges, the pixel array unit is divided into a plurality of blocks, the signals are read from pixels at the same position in each of the plurality of blocks, and rearranged so that the arrangement of the read signals is equivalent to the arrangement of pixels in the pixel array unit.

[0016] Note that the imaging device may be an independent device or an internal block constituting one device.

[0017] Note that the program can be provided by being transmitted via a transmission medium or being recorded on a recording medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A diagram showing the configuration of one embodiment of an imaging device to which the present technology is applied.

[0019] Figure 2 A diagram showing a configuration example of a pre-stage processing unit.

[0020] Figure 3 is a diagram for describing a pixel array in a pixel array unit.

[0021] Figure 4 A diagram for describing the reading order.

[0022] Figure 5 A diagram for describing the reading order.

[0023] Figure 6 102 is a diagram showing an example of the arrangement of pixel signals stored in the storage unit 102 .

[0024] Figure 7 A diagram for describing the processing of the horizontal rearrangement unit.

[0025] Figure 8 A diagram for describing a process of vertically rearranging a unit.

[0026] Fig. 9 A diagram for describing vertical rearrangement.

[0027] Fig.10 A diagram for describing compression processing.

[0028] Fig.11 is a diagram for describing the processing of the interpolation unit.

[0029] Fig.12 A diagram showing a configuration example of a PC.

[0030] Reference numerals

[0031] 1: imaging device; 11: pixel array unit; 12: block; 21: reading unit; 31: pre-processing unit; 41: post-processing unit; 101: input control unit; 102: storage unit; 103: horizontal rearrangement unit; 104: vertical rearrangement unit; 105: storage unit; 106: interpolation unit; 107: output control unit DETAILED DESCRIPTION

[0032] Hereinafter, a mode for implementing the present technology (hereinafter referred to as an embodiment) will be described.

[0033] <Configuration Example of Imaging Device>

[0034] Figure 1 1 is a diagram showing a configuration example of an embodiment of an imaging device to which the present technology is applied. The imaging device 1 includes a pixel array unit 11 , a reading unit 21 - 1 , a reading unit 21 - 2 , a pre-processing unit 31 , and a post-processing unit 41 .

[0035] The pixel array unit 11 has a configuration in which pixels are arranged in a matrix in the row direction and the column direction, and the pixels generate a charge corresponding to the amount of light received and output a signal corresponding to the charge. That is, the pixel array unit 11 has a plurality of pixels, and the plurality of pixels perform photoelectric conversion on the incident light and output a signal corresponding to the charge obtained as a result of the photoelectric conversion. Here, the row direction refers to the arrangement direction of the pixels in the horizontal direction, and the column direction refers to the arrangement direction of the pixels in the vertical direction. Note that the row direction is horizontal in the figure, and the column direction is vertical in the figure.

[0036] The pixel array unit 11 is divided into 12 blocks 12-1 to 12-12. In the following description, in the case where the blocks 12-1 to 12-12 do not need to be individually distinguished, these blocks are simply expressed as blocks 12. Other parts are expressed in a similar manner.

[0037] exist Figure 1 In the example shown, the pixel array unit 11 is divided into 12 blocks 12, but the number of divisions may be less than 12 or greater than 12. Other numerical values ​​are also numerical values ​​for explanation, rather than descriptions indicating limitations.

[0038] m×n pixels are arranged in one block 12. The imaging device 1 includes a reading unit 21-1 that reads signals from blocks 12-1 to 12-6 arranged on the left side of the pixel array unit 11 in the figure, and a reading unit 21-2 that reads signals from blocks 12-7 to 12-12 arranged on the right side of the pixel array unit 11 in the figure.

[0039] The reading unit 21-1 and the reading unit 21-2 each supply the read signal to the pre-processing unit 31. The details will be described later, for example, in the case where the reading unit 21-1 reads a signal from a pixel at a position of coordinates A (ma, na) arranged in the block 12-1, the reading unit 21-2 reads a signal from a pixel at a position of coordinates A (ma, na) arranged in position A in the block 12-7.

[0040] The coordinates A are coordinates indicating the coordinate position in each block 12 , and are coordinates obtained by setting the coordinates of the lower left pixel to (0, 0) for each block 12 , for example.

[0041] The blocks 12-1 to 12-6 to be read by the reading unit 21-1 are set as the block group A. The blocks 12-7 to 12-12 to be read by the reading unit 21-2 are set as the block group B. Pixels to be read at the same timing by the respective reading units 21 are in the blocks 12 arranged at the same position in the block group A and the block group B.

[0042] For example, in the case where the reading unit 21-1 reads a signal from a pixel in the block 12-1 located at the upper left in the block group A, the reading unit 21-2 reads a signal from a pixel in the block 12-7 located at the upper left in the block group B. In the case where the reading unit 21-1 reads a signal from a pixel arranged at a position of coordinates A (ma, na) in the block 12-1, the reading unit 21-2 reads a signal from a pixel arranged at a position of coordinates A (ma, na) in the block 12-7.

[0043] As described above, the reading unit 21-1 and the reading unit 21-2 are configured to read signals from pixels arranged at the same position in the same block in the block group at approximately the same timing. The reading unit 21-1 and the reading unit 21-2 read signals from predetermined pixels at approximately the same timing and supply the signals to the pre-processing unit 31, but some deviation may occur in the reading. As described later, the pre-processing unit 31 has a configuration that performs processing to absorb deviations when reading pixel signals.

[0044] Focusing on the reading unit 21-1, the reading unit 21-1 reads from the pixel arranged at the coordinates A (ma, mb) in the block 12-1, and then reads from the pixel arranged at the coordinates A (ma, mb) in the block 12-2 (or the block 12-4) adjacent to the block 12-1.

[0045] As described above, the reading unit 21 sequentially reads pixel signals from pixels arranged at the same coordinates within the block 12 .

[0046] The pre-stage processing unit 31 converts the arrangement of pixel signals read from the pixel array unit 11 into an arrangement equivalent to the case where pixel signals are read by raster scanning of a so-called Bayer array, and supplies the pixel signals to the post-stage processing unit 41 .

[0047] The post-processing unit 41 is a processing unit that processes a signal read from a pixel array unit of the Bayer array by raster scanning, and can be formed by a conventional LSI (Large-Scale Integration).

[0048] <Configuration of the pre-processing unit>

[0049] Figure 2 1 is a diagram showing a configuration example of the pre-processing unit 31. The pre-processing unit 31 includes an input control unit 101, a storage unit 102, a horizontal rearrangement unit 103, a vertical rearrangement unit 104, a storage unit 105, an interpolation unit 106, and an output control unit 107.

[0050] The input control unit 101 has a configuration including a physical layer (PHY) and a link layer (LINK), and supplies pixel signals supplied from the reading unit 21-1 and the reading unit 21-2 to the storage unit 102 for storage. The storage unit 102 includes, for example, a buffer, and sequentially stores pixel signals input under the control of the input control unit 101.

[0051] Since the pixel signals from the pixel array unit 11 are read by the reading unit 21-1 and the reading unit 21-2, the data strings (pixel signals) of the left and right systems are input to the storage unit 102. It is preferable that there is no deviation in the data strings of the two systems, but there may be a deviation of up to 1 line, for example. The process for absorbing the deviation is performed by performing synchronization processing with the buffering in the storage unit 102 (buffer).

[0052] When reading the pixel signal from the storage unit 102 , the horizontal rearrangement unit 103 rearranges the arrangement of pixels in the horizontal direction (row direction) and supplies the rearranged pixels to the vertical rearrangement unit 104 .

[0053] The pixel signals are read in an order different from the arrangement of the pixels in the pixel array unit 11, and are sequentially stored in the storage unit 102. The arrangement of the pixels stored in the storage unit 102 is different in both the horizontal direction and the vertical direction (row direction and column direction) compared to the arrangement of the pixels in the pixel array unit 11. The horizontal rearrangement unit 103 reads the pixel signals from the storage unit 102, while changing the arrangement of the pixels in the horizontal direction so that the pixels are arranged in the horizontal direction of the pixel array unit 11, and supplies the pixel signals to the vertical rearrangement unit 104.

[0054] When causing the subsequent storage unit 105 to store pixel signals, the vertical rearrangement unit 104 changes the arrangement of pixels in the vertical direction so that the pixels are arranged in the vertical direction of the pixel array unit 11 , and supplies the pixel signals to the storage unit and causes the storage unit 105 to store them.

[0055] The storage unit 105 is a frame memory, and includes, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The pixel signal stored in the storage unit 105 is output to the interpolation unit 106. As described later, when a pixel for phase difference detection is configured in the pixel array unit 11, the pixel signal is supplied to the interpolation unit 106 in a state where there is no pixel signal of a normal pixel corresponding to the position where the pixel for phase difference detection is configured. Therefore, the processing of interpolating the pixel signal of the normal pixel is performed by the interpolation unit 106.

[0056] The pixel signals stored in the storage unit 105 are supplied to the output control unit 107 by interpolating the pixels as needed. The pixel signals output from the output control unit 107 are supplied to the post-processing unit 41 ( Figure 1 As the post-processing unit 41, a processing unit of the related art that processes the signal from the pixel array unit 11 can be applied.

[0057] <About the order of reading from the pixel array unit>

[0058] Figure 3 1 is a diagram showing blocks 12 - 1 , 12 - 2 , 12 - 4 , and 12 - 5 arranged in the pixel array unit 11 , and showing the colors and read order of pixels arranged in each block 12 .

[0059] The block 12 includes a total of 48 pixels, including 8 pixels in the horizontal direction and 6 pixels in the vertical direction. Note that here, the case where a block 12 includes 48 pixels is used as an example for explanation, but when the present technology is applied, the number of pixels included in a block 12 is not limited.

[0060] exist Figure 3 In the figure, the numerical values ​​0 to 7 shown on the upper side of the block 12-1 represent the coordinates of the pixels in the horizontal direction (x-axis) of the block 12-1, and the numerical values ​​0 to 5 shown on the left side of the block 12-1 represent the coordinates of the pixels in the vertical direction (y-axis) of the block 12-1. The lower left coordinate of the block 12-1 is (0, 0).

[0061] exist Figure 3 In FIG. 1 , one rectangle represents one pixel. The letter R in one pixel represents a pixel provided with a red filter (hereinafter referred to as pixel G).

[0062] Similarly, G represents a pixel provided with a green filter (hereinafter referred to as pixel G), and B represents a pixel provided with a blue filter (hereinafter referred to as pixel B). P represents a pixel for phase difference detection (hereinafter referred to as pixel P).

[0063] The pixels arranged in the pixel array unit 11 are arranged in a Bayer array except for the portion where the pixel P is arranged. In the Bayer array, the pixel P is arranged at a position corresponding to the pixel B.

[0064] exist Figure 3 In , the numerical value recorded in one pixel (in one rectangle) indicates the reading order performed by the reading unit 21. Figure 3In the example shown, since 48 pixels are included in one block 12, numbers 0 to 47 are assigned to the pixels. Note that since the numbers are assigned starting from 0, the 0th reading means the first reading from the block 12, and the reading operation means reading in the 1st reading operation. In the following description, the nth reading means reading by the (n+1)th reading operation in the operation of reading the signals of the pixels in the block 12 by the reading unit 21.

[0065] exist Figure 3 In the example shown, for example, the pixel at coordinates (0, 0) is a red pixel R, indicating the pixel whose pixel signal is read out 20th. The pixel at coordinates (1, 0) is a green pixel G, indicating the pixel whose pixel signal is read out 22nd. In the following description, when it is desired to include the reading order, for example, the pixel R at coordinates (0, 0) is described as pixel R20. In such a case, the numerical value appended after the pixel is a numerical value indicating the reading order.

[0066] exist Figure 3 In the 0th row (coordinates (0, 0) to (7, 0)) of the block 12-1 shown, pixels R20, G22, R24, G26, R21, G23, R25, and G27 are arranged. In the 1st row (coordinates (0, 1) to (7, 1)) of the block 12-1, pixels G4, B6, G8, B10, G5, B7, G9, and B11 are arranged.

[0067] In the second row of block 12-1 (coordinates (0, 2) to (7, 2)), pixels R36, G38, R40, G42, R37, G39, R41, and G43 are arranged. In the third row of block 12-1 (coordinates (0, 3) to (7, 3)), pixels G28, B30, G32, B34, G29, B31, G33, and B35 are arranged.

[0068] In the 0th row (coordinates (0, 4) to (7, 4)) of block 12-4, pixels R12, G14, R16, G18, R13, G15, R17, and G19 are arranged. In the 5th row (coordinates (0, 5) to (7, 5)) of block 12-1, pixels G44, P0, G46, P2, G45, P1, G47, and P3 are arranged.

[0069] Here, the pixel arrangement in block 12-1 is used as an example for description, but the pixel arrangement in blocks 12 other than block 12-1 is also similar to the pixel arrangement in block 12-1 described here. That is, the pixel arrangement in block 12 is the same in all blocks 12, and the reading order is the same for each block 12.

[0070] As described above, the order of pixels in which pixel signals are read from the block 12 is different from the case of reading by ordinary raster scanning. The reading unit 21 reads pixel signals from the block 12 in such a reading order, and furthermore, Figure 4 As shown, the reading unit 21 reads the phase difference pixels and the normal pixels separately. The phase difference pixels are pixels P, and the normal pixels are pixels R, pixels G, and pixels B.

[0071] The reading of ordinary pixels is divided into three parts. Figure 4 As shown, when the reading unit 21 reads pixel signals from the predetermined block 12 of the pixel array unit 11, the reading unit 21 first reads pixel signals from the phase difference pixels, and then reads pixel signals from the normal pixels in three parts. Since the normal pixels are pixels for generating an image, one frame is read in three parts in this case.

[0072] The reading unit 21 reads pixel signals from the phase difference pixels P arranged in the 5th row in the block 12. Next, the reading unit 21 reads pixel signals from the normal pixels arranged in the 1st and 4th rows in the block 12. Next, the reading unit 21 reads pixel signals from the normal pixels arranged in the 0th and 3rd rows in the block 12. Next, the reading unit 21 reads pixel signals from the normal pixels arranged in the 2nd and 5th rows in the block 12.

[0073] When the reading unit 21 reads the pixel signal from the block 12, the reading unit 21 performs the reading based on the reading order set in the block 12. Figure 5 Describes reading in the order in which it is read.

[0074] In the reading period T1, pixel signals are read from the phase difference pixels P. The pixels P are arranged in the order of pixel P0, pixel P2, pixel P1, and pixel P3 from the left in the block 12. When reading, since the reading is performed in ascending order of the read number, in this case, the reading is performed in the order of pixel P0, pixel P1, pixel P2, and pixel P3.

[0075] In the reading period T2, pixel signals are read from ordinary pixels. In the reading period T2, signals from pixels arranged in the 1st and 4th rows in the block 12 are read. Among the pixels arranged in the 1st and 4th rows in the block 12, the pixel with the smallest read number is the pixel G4 arranged at the coordinates (0, 1). Therefore, in the reading period T2, reading starts from the pixel G4.

[0076] exist Figure 5 In the example shown, the pixels G4, G5, B6, B7, G8, G9, B10, and B11 of row 1 are read in this order. Next, the pixels R12, R13, G14, G15, R16, R17, G18, and G19 are read in this order.

[0077] In the reading period T3, pixel signals are read from ordinary pixels. In the reading period T3, the signals of the pixels arranged in the 0th and 3rd rows in the block 12 are read. Among the pixels arranged in the 0th and 3rd rows in the block 12, the pixel with the smallest read number is the pixel R20 arranged at the coordinates (0, 0). Therefore, in the reading period T3, reading starts from the pixel R20.

[0078] exist Figure 5 In the example shown, the pixels R20, R21, G22, G23, R24, R25, G26, and G27 of row 0 are read in this order. Next, the pixels G28, G29, B30, B31, G32, G33, B34, and B35 of row 3 are read in this order.

[0079] In the reading period T4, pixel signals are read from ordinary pixels. In the reading period T4, the signals of the pixels arranged in the 2nd and 5th rows in the block 12 are read. Among the pixels arranged in the 2nd and 5th rows in the block 12, the pixel with the smallest read number is the pixel R36 arranged at the coordinates (0, 2). Therefore, in the reading period T4, the reading starts from the pixel R36.

[0080] exist Figure 5 In the example shown, the pixels R36, R37, G38, G39, R40, R41, G42, and G43 of the second row are read in this order. Next, the pixels G44, G45, G46, and G47 of the fifth row are read in this order. In the fifth row, since the phase difference pixel P is provided and has been read in the reading period T4, only the pixel signals of the four pixels G44 to G47 are read from the fifth row.

[0081] In the case where reading is performed in such an order and writing is performed to the storage unit 102 in the order in which it is read, as shown in FIG. Figure 6 The pixel signals are stored as shown. Figure 6 1 is a diagram showing an example of arrangement of pixel signals read from the block 12 stored in the storage unit 102 . Figure 6An example is conceptually shown in which pixel signals are sequentially stored from the upper left of the storage area of ​​the storage unit 102 in the order of reading from the reading unit 21 .

[0082] The address in the storage unit 102 is represented by a coordinate system with (0, 0) at the upper left in the figure, and the description is continued assuming that pixel signals are stored in sequence from the coordinate (0, 0). Pixel P0, pixel P1, pixel P2, and pixel P3 are stored in sequence from the left side (upper left in the figure) of the first row of the storage unit 102. This is the reading order in the reading period T1.

[0083] In the 1st and 2nd rows of the storage unit 102, pixel signals are written in the order of pixels read from the 1st and 4th rows in the block 12 in the reading period T2. In the 3rd and 4th rows of the storage unit 102, pixel signals are written in the order of pixels read from the 0th and 3rd rows in the block 12 in the reading period T3.

[0084] In the 5th and 6th rows of the storage unit 102 , pixel signals are written in the order of pixels read from the 2nd and 5th rows in the block 12 in the reading period T4 .

[0085] Although the reading from one block 12 is described here as an example, similar reading is performed in all blocks 12 arranged in the pixel array unit 11. Pixels with the same reading number in each block 12 are read at the same timing. For example, the pixel P0 read first in the block 12 is also read first in each block 12 from block 12-1 to 12-12, and they are read at the same time. After that, the pixel P1 to be read second is read simultaneously in each block 12 from block 12-1 to 12-12.

[0086] Figure 1 The imaging device 1 shown is provided with two systems of reading units 21, and the left and right halves of the pixel array unit 11 are read by the corresponding reading units 21. When the pixel P0 is read from the block 12-1, the pixel P0 is also read from the block 12-7. Thereafter, when the pixel P0 is read from the block 12-2, the pixel P0 is also read from the block 12-8. As described above, the left and right sides of the pixel array unit 11 are configured to read signals from pixels corresponding to the same position at substantially the same timing.

[0087] Pixels read at the same timing are preferably supplied and stored in the storage unit 102 at the same timing, but the storage timing may vary. When reading from the storage unit 102 by the horizontal rearrangement unit 103, the variation can be absorbed by adjusting the reading timing and performing reading with the variation absorbed.

[0088] When Figure 6When the pixel signal is buffered in the storage unit 102, the horizontal rearrangement unit 103 rearranges the pixels. Figure 7 The processing of the horizontal rearrangement unit 103 is described.

[0089] Figure 7 The left figure of FIG. 1 is a diagram showing the arrangement of pixel signals stored in the storage unit 102, and Figure 6 When reading pixels P0 to P3 arranged in row 0, the horizontal rearrangement unit 103 reads pixels P0, P2, P1, and P3 in sequence and outputs the pixels P0, P2, P1, and P3 to the subsequent vertical rearrangement unit 104. The reading order is Figure 3 The order in which the phase difference pixels P are arranged in the fifth row within the block 12 is shown.

[0090] As described above, when reading pixels from the storage unit 102 , the horizontal rearrangement unit 103 reads pixels from the storage unit 102 so that the pixels are arranged in the horizontal direction (x-axis direction) in the arrangement order of pixels arranged in the pixel array unit 11 .

[0091] The rows other than row 0 stored in the storage unit 102 are read similarly. That is, when reading the pixels arranged in the order of pixel G4, pixel G5, pixel B6, pixel B7, pixel G8, pixel G9, pixel B10, and pixel B11 arranged in row 1, the horizontal rearrangement unit 103 reads pixel G4, pixel B6, pixel G8, pixel B10, pixel G5, pixel B7, pixel G9, and pixel B11 in sequence. This reading order is Figure 3 The order in which pixels G and pixels B are alternately arranged in the first row within the block 12 is shown.

[0092] When reading the pixels arranged in the order of pixel R12, pixel R13, pixel G14, pixel G15, pixel R16, pixel R17, pixel G18, and pixel G19 arranged in the second row, the horizontal rearrangement unit 103 reads the pixel R12, pixel G14, pixel R16, pixel G18, pixel R13, pixel G15, pixel R17, and pixel G1 in sequence. The reading order is Figure 3 The order in which pixels R and pixels G are alternately arranged in the fourth row within the block 12 is shown.

[0093] When reading the pixels arranged in the order of pixel R20, pixel R21, pixel G22, pixel G23, pixel R24, pixel R25, pixel G26, and pixel G27 arranged in the third row, the horizontal rearrangement unit 103 reads the pixel R20, pixel G22, pixel R24, pixel G26, pixel R21, pixel G23, pixel R25, and pixel G27 in sequence. The reading order is Figure 3The order in which pixels R and pixels G are alternately arranged in row 0 within block 12 is shown.

[0094] When reading the pixels arranged in the order of pixel G28, pixel G29, pixel B30, pixel B31, pixel G32, pixel G33, pixel B34 and pixel B35 arranged in the 4th row, the horizontal rearrangement unit 103 reads the pixel G28, pixel B30, pixel G32, pixel B34, pixel G29, pixel B31, pixel G33 and pixel B35 in sequence. The reading order is Figure 3 The order in which pixels G and pixels B are alternately arranged in the third row within the block 12 is shown.

[0095] When reading the pixels arranged in the order of pixel R36, pixel R37, pixel G38, pixel G39, pixel R40, pixel R41, pixel G42, and pixel G43 arranged in the 5th row, the horizontal rearrangement unit 103 reads the pixel R36, pixel G38, pixel R40, pixel G42, pixel R37, pixel G39, pixel R41, and pixel G43 in sequence. The reading order is Figure 3 The order in which pixels R and pixels G are alternately arranged in the second row within the block 12 is shown.

[0096] When reading the pixels arranged in the order of pixel G44, pixel G45, pixel G46, and pixel G47 arranged in the 6th row, the horizontal rearrangement unit 103 reads pixel G44, pixel G46, pixel G45, and pixel G47 in sequence. The reading order is Figure 3 The order in which the pixels G of the fifth row are arranged in the block 12 is shown. Since the phase difference pixels P are read at different timings, only the pixels G are read here.

[0097] As described above, the horizontal rearrangement unit 103 rearranges in the horizontal direction so that the arrangement of pixels output from the horizontal rearrangement unit 103 is the same as the arrangement of the pixel array unit 11 in the horizontal direction. The output from the horizontal rearrangement unit 103 is supplied to the vertical rearrangement unit 104.

[0098] Reference Figure 8 and Fig. 9 The processing of the vertical rearrangement unit 104 is described. When the pixel G4 is input to the vertical rearrangement unit 104, the vertical rearrangement unit 104 writes the pixel signal of the pixel G4 to the address corresponding to the coordinates (0, 1) in the block 12 of the pixel array unit 11 in the storage unit 105. Next, when the pixel B6 is input to the vertical rearrangement unit 104, the pixel signal of the pixel B6 is written to the address corresponding to the coordinates (1, 1) in the block 12 of the pixel array unit 11 in the storage unit 105.

[0099] Since the pixel columns that have been horizontally rearranged by the horizontal rearrangement unit 103 are input to the vertical rearrangement unit 104, the pixel signals of the pixels G4 to B11 are converted to the rows of the storage unit 105 corresponding to the rows arranged in the block 12 of the pixel array unit 11 and are sequentially written. Fig. 9 The conversion of the rows to be written will be described.

[0100] exist Fig. 9 In FIG. 1 , the left side shows the arrangement of the normal pixels stored in the first column of the storage unit 102 , and the right side shows the arrangement of the normal pixels stored in the first column of the storage unit 105 . Fig. 9 The numerical value shown on the left side of the rectangle representing the pixel shown on the left side indicates the row number of the storage unit 102 . Fig. 9 The numerical value shown on the right side of the rectangle representing the pixel shown on the right side indicates the row number of the storage unit 105 , and the numerical value shown in the bracket on the right side thereof indicates the row number corresponding to the row number in the block 12 of the pixel array unit 11 .

[0101] The pixel R4 stored in the first row of the storage unit 102 is written into the fifth row of the storage unit 105, which is the row corresponding to the first row in the block 12. The pixels B6, G8, B10, G5, B7, G9, and B11 ( Figure 8 ) are written sequentially to the 5th row of storage unit 105, which is the row corresponding to the 1st row in block 12.

[0102] The pixel R12 stored in the second row of the storage unit 102 is written into the second row of the storage unit 105, which is the row corresponding to the fourth row in the block 12. The pixels G14, R16, G18, R13, G15, R17, and G19 (pixels R19, R11, R12, R13, R15, R17, and R19) arranged in the row starting from the pixel R12 are written into the second row of the storage unit 105, which is the row corresponding to the fourth row in the block 12. Figure 8 ) are written sequentially into the second row of storage unit 105, which is the row corresponding to the fourth row in block 12.

[0103] The pixel R20 stored in the 3rd row of the storage unit 102 is written into the 6th row of the storage unit 105, which is the row corresponding to the 0th row in the block 12. The pixels G22, R24, G26, R21, G23, R25, and G27 (pixels R27, R28, R29, R30, R31, R32, R33, R34, R35, and R36) arranged in the row starting from the pixel R20 are written into the 6th row of the storage unit 105, which is the row corresponding to the 0th row in the block 12. Figure 8 ) are written sequentially into the 6th row of storage unit 105, which is the row corresponding to the 0th row in block 12.

[0104] The pixel G28 stored in the 4th row of the storage unit 102 is written into the 3rd row of the storage unit 105, which is the row corresponding to the 3rd row in the block 12. The pixels B30, G32, B34, G29, B31, G33, and B35 ( Figure 8 ) are written sequentially to the 3rd row of storage unit 105, which is the row corresponding to the 3rd row in block 12.

[0105] The pixel R36 stored in the fifth row of the storage unit 102 is written into the fourth row of the storage unit 105, which is the row corresponding to the second row in the block 12. The pixels G38, R40, G42, R37, G39, R41, and G43 ( Figure 8 ) are written sequentially into the 4th row of storage unit 105, which is the row corresponding to the 2nd row in block 12.

[0106] The pixel G44 stored in the 6th row of the storage unit 102 is written into the 1st row of the storage unit 105, which is the row corresponding to the 5th row in the block 12. The pixels G46, G45, and G47 arranged in the row starting from the pixel G44 are written into the 1st row of the storage unit 105, which is the row corresponding to the 5th row in the block 12, by skipping one pixel in sequence. Since the phase difference pixel P is arranged in the 5th row of the block 12 of the pixel array unit 11, the portion corresponding to the phase difference pixel P is left blank, and the pixels G44 to G47 are written.

[0107] As described above, the vertical rearrangement unit 104 transforms the arrangement position of the input pixels (pixel signals) in the vertical direction (y-axis) so that the arrangement is equivalent to the pixel arrangement within the block of the pixel array unit 11, and writes the input pixels (pixel signals) into the corresponding area of ​​the storage unit 105.

[0108] With the processing up to now, the pixels read from the pixel array unit 11 are rearranged into an arrangement within the block 12 of the pixel array unit 11. This arrangement is the same arrangement as when the pixel array unit 11 is raster scanned.

[0109] When the vertical rearrangement unit 104 writes the pixel signal to the storage unit 105, the pixel signal may be compressed before writing. When writing data to a memory such as the storage unit 105, in order to reduce the bandwidth of the memory capacity, the original data is compressed before writing. Such a compression technique may be applied to reduce the bandwidth of the storage unit 105.

[0110] Reference Fig.10 The compression process performed when the vertical rearrangement unit 104 writes the pixel signal to the storage unit 105 will be described. Fig.10A is a diagram for describing a compression process for pixels read from a row where no phase difference pixels P are arranged, Fig.10 B is a diagram for describing a compression process of pixels read from a row in which the phase difference pixels P are arranged.

[0111] For compression of raw data, a compression method using the correlation between adjacent pixels can be applied. When a compression method using the correlation between adjacent pixels is applied, in order to improve compression efficiency, horizontal reordering suitable for the raw data compression method can be performed together with vertical reordering.

[0112] exist Fig.10 In A of FIG. 1 , the compression processing when the pixels R36, G38, R40, G42, R37, G39, R41, and G43 are written to the storage unit 105 is described. When the pixels are compressed in the storage unit 105 in this arrangement, the compression efficiency may be reduced because the pixels R and the pixels G are alternately arranged. Therefore, the rearrangement in the horizontal direction is performed so that the pixels R and the pixels R are adjacent to each other and the pixels G and the pixels G are adjacent to each other.

[0113] exist Fig.10 In the example shown in A, during compression processing, pixel R40 is moved to behind pixel R36, pixel R37 is moved to behind pixel R40, and pixel R41 is moved to behind pixel R37. In addition, pixel G38 is moved to behind pixel R41, pixel G42 is moved to behind pixel G38, pixel G39 is moved to behind pixel G42, and pixel G43 is moved to behind pixel G39. In this way, by aggregating pixels of the same color, in other words, by aggregating pixels with high correlation, compression efficiency can be improved during compression processing.

[0114] As reference Figure 8 As described above, each compressed pixel signal is written into the area of ​​the corresponding arrangement position in the storage unit 105 so that the arrangement is equivalent to the pixel arrangement in the block 12 of the pixel array unit 11 .

[0115] This compression process is performed on the pixel columns arranged in the rows other than the row where the pixel R36 is arranged. In the row where the phase difference pixel P is arranged, the phase difference pixel P is read first, so the number of pixels in one row is small. Figure 8 In the illustrated example, the row (referred to as row B) where the phase difference pixels P are arranged corresponds to the fifth row of the storage unit 105 , but is arranged with only four pixels G. The other row (referred to as row A) is arranged with eight pixels.

[0116] Reference Fig.10The compression method of A includes rearrangement processing in which pixels of the same color are arranged adjacent to each other from a state in which pixels of different colors are arranged alternately as in row A. By making this method applicable to rows in which pixels of the same color are adjacent to each other and the number of pixels is smaller than that in row B, the same compression method can be applied to all rows.

[0117] Fig.10 FIG. B is a diagram for describing the compression process of a row configured with fewer pixels than row A, such as row B. The pixels of the row to be compressed are four pixels, namely, pixel G44, pixel G46, pixel G45, and pixel G47. Since there are only four pixels, four pixels are added to obtain eight pixels. Fig.10 As shown in the middle part of B, pixel a is inserted between pixel G44 and pixel G46, pixel b is inserted between pixel G46 and pixel G45, pixel c is inserted between pixel G45 and pixel G47, and pixel d is added after pixel G47.

[0118] Pixels a to d are pixels at positions where phase difference pixels P are arranged, and are pixels at positions where pixels B (blue pixels) are arranged in the case of a row where ordinary pixels other than phase difference pixels P are arranged. Pixels a to d may be pixels virtually allocated to correspond to pixel B, or may be pixels brought from other rows or other blocks 12.

[0119] Fig.10 The state shown in the middle of B is a state where one row includes 8 pixels and pixels G and pixels B (pixels a to d are inserted as equivalent to pixels B) are alternately arranged, which is similar to the state of row A. Therefore, similar to row A, it is possible to apply reference Fig.10 The compression method described in A is used for compression.

[0120] That is, Fig.10 As shown in the lower part of B, pixel G46 is moved to behind pixel G44, pixel G45 is moved to behind pixel G46, and pixel G47 is moved to behind pixel G45. In addition, pixel a is moved to behind pixel G47, pixel b is moved to behind pixel a, pixel c is moved to behind pixel b, and pixel d is moved to behind pixel c.

[0121] In this way, by aggregating pixels with high correlation, compression efficiency can be improved during the compression process.

[0122] By writing the pixel signal subjected to such compression processing into the storage unit 105 , the capacity of the storage unit 105 can be reduced.

[0123] The pixel signal stored in the storage unit 105 is supplied to the interpolation unit 106. Fig.10As shown in row B of FIG. 1 , the interpolation unit 106 performs pixel interpolation processing on the row with missing pixels because the row is a row where the phase difference pixel P is configured. Fig.11 This interpolation processing will be described.

[0124] Fig.11 The left picture and Figure 8 , and is a diagram showing a state in which pixel signals are stored in the storage unit 105 in the same pixel array as the pixel array unit 11. When the pixels G44, G46, G45, and G47 arranged in the fifth row are read from the storage unit 105, the interpolation unit 106 interpolates the pixels e, f, g, and h.

[0125] Pixel e is interpolated between pixel G44 and pixel G46, pixel f is interpolated between pixel G46 and pixel G45, pixel g is interpolated between pixel G45 and pixel G47, and pixel h is added after pixel G47. Since pixels e to h are interpolated at positions having the function of blue pixel B, pixels having values ​​corresponding to blue pixel B are generated and interpolated.

[0126] For example, the values ​​of a plurality of pixels B located around the position where the pixel e is configured can be read, the average value of these values ​​can be calculated, and the average value can be used as the value of the pixel signal of the pixel e for interpolation. As pixels f to h, the average value based on other pixels can also be generated. As the values ​​of pixels e to f, values ​​other than the average value can be generated and used.

[0127] As described above, in a row with a small number of pixels, pixels are interpolated and the same number of pixels as other rows are output. The interpolation unit 106 does not perform any processing on rows that do not require interpolation pixels (eg, rows corresponding to row A) and supplies the input pixels to the output control unit 107.

[0128] The pixels supplied to the output control unit 107 via the interpolation unit 106 are a pixel array equivalent to the pixels read by raster scanning when the pixel array unit 11 has a Bayer array. Figure 1 ) may be a processing unit that processes pixels read by raster scanning when the pixel array unit 11 has a Bayer array, and a processing unit (LSI, etc.) of related art can be used.

[0129] In the above embodiment, the case where the vertical arrangement is performed after the horizontal arrangement is performed is described as an example, but the horizontal arrangement may be performed after the vertical arrangement is performed.

[0130] In the above embodiment, the case where the pixel array unit 11 is provided with phase difference pixels is described as an example, but the present technology can also be applied when no phase difference pixels are provided. When the pixel array unit 11 is not provided with phase difference pixels, the processing of phase difference pixels can be omitted in the above description.

[0131] In the above embodiment, the case where the pixel array of the pixel array unit 11 is a Bayer array is described as an example, but the present technology can also be applied to arrays other than the Bayer array. For example, the present technology can also be applied to the case where the pixel array unit 11 is configured with red pixels R, green pixels G, blue pixels B, and white (transparent) pixels, or a case where pixels that receive infrared rays are configured.

[0132] According to the present technology, even when a method other than raster scanning is applied to reading pixel signals from the pixel array unit 11, the post-processing unit 41 that processes the pixel signals from the pixel array unit 11 can process in a manner similar to the case of reading by raster scanning. Therefore, the conventional LSI that processes on the premise of reading from the pixel array unit 11 of the general Bayer array by raster scanning can be used as the post-processing unit 41.

[0133] According to the present technology, a special readout format can be converted into a common Bayer array format. By enabling high-speed reading of the special readout format, simultaneous exposure and simultaneous reading, that is, a so-called global shutter, can be supported.

[0134] In the case of a circuit configuration for realizing simultaneous exposure of a global shutter, the above-mentioned read format of the pixel signal from the pixel array unit 11 can be applied. As its circuit configuration, as described above, the pixel array unit 11 can be divided into a plurality of blocks 12 and the same movement method (reading method) can be applied to all the blocks 12. By adopting such a general structure, it is possible to achieve miniaturization of the device and reduction of power consumption.

[0135] <Regarding Recording Media>

[0136] The above series of processing can be performed by hardware or software. In the case where a series of processing is performed by software, the program constituting the software is installed on a computer. Here, examples of computers include computers embedded in dedicated hardware, and general-purpose personal computers that can perform various functions by installing various programs, etc.

[0137] Fig.122004 is a block diagram showing a configuration example of hardware of a computer that executes the above-mentioned series of processes by a program. In the computer, a central processing unit (CPU) 2001, a read-only memory (ROM) 2002, and a random access memory (RAM) 2003 are connected to each other via a bus 2004. In addition, an input / output interface 2005 is connected to the bus 2004. An input unit 2006, an output unit 2007, a storage unit 2008, a communication unit 2009, and a drive 2010 are connected to the input / output interface 2005.

[0138] The input unit 2006 includes a keyboard, a mouse, a microphone, etc. The output unit 2007 includes a display, a speaker, etc. The storage unit 2008 includes a hard disk, a nonvolatile memory, etc. The communication unit 2009 includes a network interface, etc. The drive 2010 drives a removable medium 2011 such as a magnetic disk, an optical disk, an optical magnetic disk, or a semiconductor memory.

[0139] In the computer configured as described above, the CPU 2001 performs the above-described series of processing by, for example, loading a program stored in the storage unit 2008 into the RAM 2003 via the input / output interface 2005 and the bus 2004 and executing the program.

[0140] The program executed by the computer (CPU 2001) can be provided by being recorded in the removable medium 2011 as a package medium or the like, for example. Alternatively, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0141] In the computer, the program can be installed to the storage unit 2008 via the input / output interface 2005 by mounting the removable medium 2011 to the drive 2010. In addition, the program can be received by the communication unit 2009 via a wired or wireless transmission medium and installed in the storage unit 2008. In addition, the program can be pre-installed in the ROM 2002 or the storage unit 2008.

[0142] Note that the program executed by the computer may be a program that is processed in time series in the order described in this specification, or may be a program that is processed in parallel or at a necessary timing such as when a call is made.

[0143] In this specification, a system means an entire device including a plurality of devices.

[0144] Note that the effects described in this specification are merely illustrative and non-limiting, and other effects may be provided.

[0145] Note that the implementation of the present technology is not limited to the above-described embodiments, and various changes can be made without departing from the gist of the present technology.

[0146] Note that the present technology can also have the following configurations.

[0147] (1) An imaging device comprising:

[0148] A pixel array unit, in which pixels are arranged in a matrix in row and column directions, and the pixels generate charges corresponding to the amount of light received and output signals corresponding to the charges;

[0149] a reading unit that divides the pixel array unit into a plurality of blocks and reads the signal from pixels at the same position in each of the plurality of blocks; and

[0150] A rearrangement unit performs rearrangement so that the arrangement of the signals read by the reading unit is equivalent to the arrangement of pixels in the pixel array unit.

[0151] (2) The imaging device according to (1), wherein:

[0152] The rearrangement unit comprises:

[0153] a row direction rearrangement unit, which rearranges the arrangement of pixels read from the reading unit so that the arrangement in the row direction is equivalent to the arrangement of pixels in the pixel array unit in the row direction; and

[0154] The column direction rearrangement unit rearranges the arrangement of the pixels read from the reading unit so that the arrangement in the column direction is equivalent to the arrangement in the column direction of the pixels in the pixel array unit.

[0155] (3) The imaging device according to (2), wherein:

[0156] When reading the signal from the first storage unit storing the signal read from the pixel array unit, the row direction rearrangement unit rearranges so that the arrangement of pixels in the row direction is equivalent to the arrangement of pixels in the pixel array unit in the row direction.

[0157] (4) The imaging device according to (2) or (3), wherein:

[0158] When reading the signal from the first storage unit storing the signal read from the pixel array unit, the column direction rearrangement unit rearranges so that the arrangement of pixels in the column direction is equivalent to the arrangement of pixels in the pixel array unit in the column direction.

[0159] (5) The imaging device according to any one of (2) to (4), wherein:

[0160] When the signal is read from the first storage unit storing the signal read from the pixel array unit and output to the column direction rearrangement unit, the row direction rearrangement unit rearranges the pixels in the row direction so that the arrangement of the pixels is equivalent to the arrangement of the pixels in the row direction in the pixel array unit.

[0161] (6) The imaging device according to any one of (2) to (5), wherein:

[0162] The column-direction rearrangement unit writes the signal output from the row-direction rearrangement unit into a region of the second storage unit corresponding to a position of a pixel from which the signal is read in the pixel array unit.

[0163] (7) The imaging device according to (6), wherein:

[0164] The column direction rearrangement unit performs compression processing when writing the signal into the second storage unit.

[0165] (8) The imaging device according to (7), wherein:

[0166] In the case where the compression process is performed on a row having a smaller number of pixels than other rows, the pixels are interpolated and compressed in the same compression manner as the other rows.

[0167] (9) The imaging device according to any one of (1) to (8), further comprising:

[0168] The interpolation unit interpolates by generating a signal corresponding to the missing pixel when the signal output from the rearrangement unit is a signal from a row having a smaller number of pixels than other rows.

[0169] (10) The imaging device according to (9), wherein:

[0170] The arrangement of pixels output from the interpolation unit is an arrangement in the case where the signal is read from the pixel array unit using raster scanning.

[0171] (11) An imaging method, wherein:

[0172] The imaging device includes a pixel array unit in which pixels are arranged in a matrix in row and column directions, and the pixels generate charges corresponding to the amount of light received and output signals corresponding to the charges. The imaging method includes:

[0173] dividing the pixel array unit into a plurality of blocks, and reading the signals from pixels at the same position in each of the plurality of blocks; and

[0174] The rearrangement is performed so that the arrangement of the read signals is equivalent to the arrangement of pixels in the pixel array unit.

[0175] (12) A program for causing a computer controlling an imaging device to execute a process, wherein:

[0176] The imaging device includes a pixel array unit in which pixels are arranged in a matrix in row and column directions, and the pixels generate charges corresponding to the amount of light received and output signals corresponding to the charges. The processing includes the following steps:

[0177] dividing the pixel array unit into a plurality of blocks, and reading the signals from pixels at the same position in each of the plurality of blocks; and

[0178] The rearrangement is performed so that the arrangement of the read signals is equivalent to the arrangement of pixels in the pixel array unit.

Claims

1. An imaging device comprising: A pixel array unit, in which pixels are arranged in a matrix in row and column directions, and the pixels generate charges corresponding to the amount of light received and output signals corresponding to the charges; a reading unit, which divides the pixel array unit into a plurality of blocks and reads the signal from pixels at the same position in each of the plurality of blocks; as well as A rearrangement unit performs rearrangement so that the arrangement of the signals read by the reading unit is equivalent to the arrangement of pixels in the pixel array unit.

2. The imaging device according to claim 1, wherein: The rearrangement unit comprises: a row direction rearrangement unit, which rearranges the arrangement of pixels read from the reading unit so that the arrangement in the row direction is equivalent to the arrangement of pixels in the pixel array unit in the row direction; as well as The column direction rearrangement unit rearranges the arrangement of the pixels read from the reading unit so that the arrangement in the column direction is equivalent to the arrangement in the column direction of the pixels in the pixel array unit.

3. The imaging device according to claim 2, wherein: When reading the signal from the first storage unit storing the signal read from the pixel array unit, the row direction rearrangement unit rearranges so that the arrangement of pixels in the row direction is equivalent to the arrangement of pixels in the pixel array unit in the row direction.

4. The imaging device according to claim 2, wherein: When reading the signal from the first storage unit storing the signal read from the pixel array unit, the column direction rearrangement unit rearranges so that the arrangement of pixels in the column direction is equivalent to the arrangement of pixels in the pixel array unit in the column direction.

5. The imaging device according to claim 2, wherein: When the signal is read from the first storage unit storing the signal read from the pixel array unit and output to the column direction rearrangement unit, the row direction rearrangement unit rearranges the pixels in the row direction so that the arrangement of the pixels in the row direction is equivalent to the arrangement of the pixels in the pixel array unit in the row direction.

6. The imaging device according to claim 2, wherein: The column-direction rearrangement unit writes the signal output from the row-direction rearrangement unit into a region of the second storage unit corresponding to a position of a pixel from which the signal is read in the pixel array unit.

7. The imaging device according to claim 6, wherein: The column direction rearrangement unit performs compression processing when writing the signal into the second storage unit.

8. The imaging device according to claim 7, wherein: In the case where the compression process is performed on a row having a smaller number of pixels than other rows, the pixels are interpolated and compressed in the same compression manner as the other rows.

9. The imaging device according to claim 1, further comprising: The interpolation unit interpolates by generating a signal corresponding to the missing pixel when the signal output from the rearrangement unit is a signal from a row having a smaller number of pixels than other rows.

10. The imaging device according to claim 9, wherein: The arrangement of pixels output from the interpolation unit is an arrangement in the case where the signal is read from the pixel array unit using raster scanning.

11. An imaging method, wherein: The imaging device includes a pixel array unit in which pixels are arranged in a matrix in row and column directions, and the pixels generate charges corresponding to the amount of light received and output signals corresponding to the charges. The imaging method includes: dividing the pixel array unit into a plurality of blocks, and reading the signals from pixels at the same position in each of the plurality of blocks; and The rearrangement is performed so that the arrangement of the read signals is equivalent to the arrangement of pixels in the pixel array unit.

12. A program for causing a computer controlling an imaging device to execute a process, wherein: The imaging device includes a pixel array unit in which pixels are arranged in a matrix in row and column directions, and the pixels generate charges corresponding to the amount of light received and output signals corresponding to the charges. The processing includes the following steps: dividing the pixel array unit into a plurality of blocks, and reading the signals from pixels at the same position in each of the plurality of blocks; as well as The rearrangement is performed so that the arrangement of the read signals is equivalent to the arrangement of pixels in the pixel array unit.

Images