Imaging device and imaging method

Abstract

[PROBLEM] To provide an imaging device and an imaging method capable of suppressing a reduction in the speed of generating image data for display and capable of achieving high-resolution image data for display. [SOLUTION] The present disclosure provides an imaging device comprising: a first image generation unit that generates, on the basis of first image data generated by an imaging unit, second image data in which the arrangement of pixel data corresponding to each pixel is changed; a generation unit that divides the second image data into partial row data corresponding to a plurality of rows during image generation, that generates the partial row data as a plurality of pieces of data packet data in a predetermined data format, that generates at least a portion of the first image data corresponding to the partial row data as additional packet data in a predetermined data format, and that repeatedly outputs the plurality of pieces of data packet data and the corresponding additional packet data in this order; and a second image generation unit that uses the data packet data and the additional packet data to generate third image data to be used for display by a display unit.

Description

Imaging device and imaging method

[0001] The present disclosure relates to an imaging device and an imaging method.

[0002] It is generally known that in an imaging device, an image sensor that generates second image data by performing re-mosaic processing on first image data based on imaging light and a processor that generates image data for display are configured on different chips. In such cases, the second image data may be divided into multiple packet data that are more suitable for processing by the processor and then output.

[0003] JP 2019-161384 A JP 2020-43435 A

[0004] However, in order to speed up the generation of image data for display, the size of the frame data of the transferred second image data is reduced, and as a result, the first image data that can be used to generate high-resolution image data for display is not transferred and is not used to generate image data for display.

[0005] Therefore, the present disclosure provides an imaging device and an imaging method that can suppress a decrease in the generation speed of image data for display and can increase the resolution of image data for display.

[0006] In order to solve the above problems, according to the present disclosure, there is provided an imaging device comprising: an imaging unit in which pixel blocks each consisting of a plurality of pixels of the same color are arranged; a first image generation unit that generates second image data by changing the arrangement of pixel data corresponding to each pixel based on first image data generated by the imaging unit; a generation unit that divides the second image data into partial line data corresponding to a plurality of lines at the time of image generation, generates the partial line data as a plurality of data packet data in a predetermined data format, generates at least a portion of the first image data corresponding to the partial line data as additional packet data in the predetermined data format, and repeatedly outputs the plurality of data packet data and the corresponding additional packet data in that order; and a second image generation unit that uses the data packet data and the additional packet data to generate third image data to be used for display on a display unit.

[0007] The partial row data may be in a range corresponding to the number of rows of the pixel block, and the second image generation unit may convert the partial row data into the third image data based on at least a portion of the first image data and the second image data.

[0008] A shared on-chip lens may be formed in at least some of the pixel blocks to focus incident light onto pixels that generate image data for the pixel block.

[0009] The additional data included in the additional packet data may be at least a part of fourth image data generated by a pixel in which the shared on-chip lens is arranged, or phase difference data for controlling an AF function generated using the fourth image data.

[0010] The additional data may be at least a part of the phase difference data and the fourth image data, and may be generated within the same additional packet data.

[0011] The imaging device may further include an imaging system that focuses the incident light, and a focus processing unit that controls the focus of the imaging system based on the phase difference data.

[0012] The focusing processing unit may control a focus of the imaging system based on the phase difference data based on at least a part of the fourth image data.

[0013] The second image data may be re-mosaic image data in a Bayer array.

[0014] The generation unit may generate each of the image data for each row of the second image data as the data packet data in sequence, and may output the plurality of data packet data corresponding to a predetermined row of the second image data and the additional packet data in sequence to the second image generation unit.

[0015] The capacity of the frame data, which is the total data of the data packet data and the additional packet data, when the second image data is output to the second image generation unit may be fixed, and the generation unit may change the combination of data included in the additional packet data depending on the imaging mode.

[0016] The imaging device may have a plurality of imaging modes, the pixel blocks corresponding to red, blue, and green, respectively, the additional data being at least a portion of the first image data corresponding to red, blue, and green, respectively, and the selection of the first image data of red, blue, and green may differ depending on the imaging mode.

[0017] When the additional data includes phase difference data according to the imaging mode, the additional data may be at least a part of the first image data in the pixel block used to generate the phase difference data.

[0018] The pixel block in which the shared on-chip lens is arranged may correspond to the green color.

[0019] Depending on the imaging mode, one of a part of the first image data for red and blue, and a part of the first image data for green may be selected.

[0020] The additional data included in the additional packet data may be all of the first image data.

[0021] The pixels in the pixel block formed by the plurality of pixels of the same color may have different sensitivity characteristics, and the additional data included in the additional packet data may be at least one of the image data having different sensitivity characteristics.

[0022] The second image generating section may generate the third image data in which the resolution of a dark area is adjusted based on the additional data.

[0023] In order to solve the above problems, according to the present disclosure, there is provided an imaging method for an imaging device having an imaging section in which pixel blocks each composed of a plurality of pixels of the same color are arranged, the imaging method comprising: a first image generation step of generating second image data by changing the arrangement of pixel data corresponding to each pixel based on first image data generated by the imaging section; a generation step of dividing the second image data into partial line data corresponding to a plurality of lines at the time of image generation, generating the partial line data as a plurality of data packet data in a predetermined data format, generating at least a portion of the first image data corresponding to the partial line data as additional packet data in the predetermined data format, and repeatedly outputting the plurality of data packet data and the corresponding additional packet data in that order; and a second image generation step of generating third image data to be used for display on a display section using the data packet data and the additional packet data.

[0024] In the second image generating step, the first image data may be converted into the third image data based on at least a part of the first image data and the second image data.

[0025] A shared on-chip lens may be formed in at least some of the pixel blocks to focus incident light onto pixels that generate image data for the pixel block.

[0026] 1 is a diagram showing an example of a schematic configuration of an imaging device to which the present technology is applied. A diagram showing an example of the configuration of an image sensor. A diagram showing an example of the arrangement of phase difference detection pixels in a pixel array unit. A block diagram showing an example of the configuration of a captured image generation unit. A diagram conceptually illustrating conversion processing by a re-mosaic processing unit. A diagram showing the pixel relationship between first image data and second image data. A diagram schematically showing an example of generation of phase difference data in mode 1a. A diagram schematically showing an example of generation of phase difference data in mode 1b. A diagram schematically showing an example of generation of additional data in mode 2. A diagram schematically showing an example of generation of additional data in mode 4. A diagram showing packet data for one frame of data when transmitting second image data. A diagram showing packet data for one frame of data when transmitting second image data in mode 1b. A diagram showing packet data for one frame of data when transmitting second image data. A diagram showing packet data for one frame of data in mode 3a. A diagram showing packet data for one frame of data in mode 3b. A diagram showing packet data for one frame of data in mode 4. A block diagram showing an example of the configuration of a control processing unit. A diagram conceptually illustrating demosaic processing. A diagram conceptually illustrating demosaic processing in mode 2. A diagram schematically showing the Nyquist frequency in a developed image. A diagram showing an example of packet data generated by an encoding unit. A diagram schematically showing first image data, second image data, and third image data. A diagram schematically showing a first image, a second image, and additional data. A diagram schematically showing other additional data.

[0027] Hereinafter, embodiments of an imaging device and an imaging method will be described with reference to the drawings. The following description will focus on the main components of the imaging device and the imaging method, but the imaging device and the imaging method may include components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0028] First Embodiment Fig. 1 is a diagram showing a schematic configuration example of an imaging device to which the present technology is applied. The imaging device 1 shown in Fig. 1 includes an imaging unit 100, a control processing unit 200, a display unit 300, an operation input unit 400, and a storage unit 500. The imaging device 1 is installed in, for example, a smartphone.

[0029] The imaging unit 100 generates first image data based on imaging light passing through an imaging system made up of a lens, converts the first image data into second image data, and outputs the second image data to the control processing unit 200. Details of the imaging unit 100 will be described later.

[0030] The control processing unit 200 is, for example, an application processor, and converts the second image data into third image data used for display on the display unit 300, and outputs the third image data to the display unit 300. Details of the control processing unit 200 will also be described later.

[0031] The display unit 300 is, for example, a liquid crystal touch panel, and displays a third image based on the third image data. The operation input unit 400 inputs an operation signal from an operator to the imaging unit 100 and the control processing unit 200 via, for example, a touch panel or a physical shutter button. The storage unit 500 stores the third image data, etc.

[0032] The detailed configuration of the imaging unit 100 will be described with reference to Fig. 2 to Fig. 4. Fig. 2 is a diagram showing an example of the configuration of the image sensor 102. Fig. 3 is a diagram showing an example of the arrangement of phase difference detection pixels 21G in the pixel array unit 3. Fig. 4 is a block diagram showing an example of the configuration of the captured image generation unit 104. As shown in Figs. 2 to 4, the imaging unit 100 has the image sensor 102 and the captured image generation unit 104.

[0033] 2, the image sensor 102 is configured to include a pixel array section 3 in which pixels 21 are two-dimensionally arranged in a matrix on a semiconductor substrate 12 made of, for example, silicon (Si) as a semiconductor, and a peripheral circuit section surrounding the pixel array section 3. The peripheral circuit section includes a vertical drive circuit 4, a column signal processing circuit 5, a horizontal drive circuit 6, an output circuit 7, a control circuit 8, etc.

[0034] The pixels 21 convert imaging light received via the imaging system into an electrical signal. Each pixel 21 has a photodiode as a photoelectric conversion unit and multiple pixel transistors. The control circuit 8 generates clock signals and control signals that serve as references for the operations of the vertical drive circuit 4, column signal processing circuit 5, horizontal drive circuit 6, etc., based on a vertical synchronization signal, a horizontal synchronization signal, and a master clock. The control circuit 8 then outputs the generated clock signals and control signals to the vertical drive circuit 4, column signal processing circuit 5, horizontal drive circuit 6, etc.

[0035] The vertical drive circuit 4, which is configured, for example, by a shift register, selects a predetermined pixel drive wiring 10 and supplies pulses to the selected pixel drive wiring 10 to drive the pixels 21, thereby driving the pixels 21 row by row. The column signal processing circuit 5 is arranged for each column of the pixels 21 and performs signal processing such as noise reduction on signals output from each row of the pixels 21 for each pixel column. For example, the column signal processing circuit 5 performs signal processing such as CDS (Correlated Double Sampling) to remove fixed pattern noise specific to each pixel and AD conversion. The horizontal drive circuit 6, which is configured, for example, by a shift register, sequentially outputs horizontal scanning pulses to sequentially select each column signal processing circuit 5 and cause each column signal processing circuit 5 to output a pixel signal to a horizontal signal line 11. The output circuit 7 performs predetermined signal processing on the signals sequentially supplied from each column signal processing circuit 5 via the horizontal signal line 11, and outputs the signals to the captured image generation unit 104 as two-dimensional first image data for each pixel 21. The input / output terminal 13 exchanges signals with the outside. In addition, since the first image data is output row by row, the captured image generating unit 104 (described later) can process the first image data in the order of the output rows, thereby enabling faster processing.

[0036] 3 shows an example of the arrangement of the phase difference detection pixels 21G in the pixel array unit 3. The illustration shows the pixels 21 (=4×4) extracted from the pixel array unit 3. The pixels 21 of the pixel array unit 3 are arranged in a quad Bayer arrangement (Quad Bayer Coding) in which pixel blocks (=2×2) each composed of a plurality of identical color pixels 21R, 21G, and 21B are arranged.

[0037] 3, for the phase difference detection pixel 21G, four pixels having selective sensitivity to green (G) are covered by one shared on-chip lens L200. For the normal pixels 21R and 21B, each pixel has selective sensitivity to red (R) or blue (B), and is covered by an individual on-chip lens L202. Note that this example of the arrangement of the pixel array unit 3 is just an example, and is not limited to this. For example, the normal pixels 21R and 21B can also be covered by the shared on-chip lens L200 instead of the individual on-chip lens L202.

[0038] 4, the captured image generation unit 104 includes a remosaic processing unit 106, a phase difference processing unit 108, and an encoding unit 110. The remosaic processing unit 106 performs array conversion processing (remosaic processing (RMSC)) on first image data D20 in a Quad Bayer array to convert the first image data D20 into second image data D22 in a Bayer array (see, for example, Patent Document 2). The remosaic processing unit 106 according to this embodiment corresponds to the first image generation unit.

[0039] In this way, the re-mosaic processing unit 106 converts the first image data D20 having a pixel signal arrangement in a quad-Bayer array into a Bayer array and generates second image data D22 in a Bayer array. The Bayer array image data D22 includes two G pixels, one R pixel, and one B pixel, and these four pixels are arranged in a 2-pixel x 2-pixel grid so that the G pixels are not adjacent. In this embodiment, when an example of the arrangement of image data is schematically described as an image, the image data generated by pixel 21R may be referred to as an R pixel, the image data generated by pixel 21G may be referred to as a G pixel, and the image data generated by pixel 21B may be referred to as a B pixel.

[0040] Fig. 5 is a diagram schematically showing the conversion process of the re-mosaic processing unit 106. Fig. 6 is a diagram showing the pixel relationship between the first image data D20 and the second image data D22. As shown in Fig. 5, the re-mosaic processing unit 106 performs interpolation processing on the first image data D20 in the Quad Bayer array to generate G complement image data D20a. Next, smoothing (low-pass) filtering is performed on the G complement image data D20a at positions corresponding to the pixel positions of pixels other than G pixels (R pixels and B pixels) in the first image data D20 to generate G smoothed image data D20b.

[0041] The re-mosaic processing unit 106 also performs low-pass filtering on each pixel of R, G, and B of the first image data D20 to generate low-frequency component images using R pixels, B pixels, and G pixels, respectively. Then, as shown in Fig. 6, the re-mosaic processing unit 106 performs color conversion processing to generate pixel values ​​at the positions of R pixels, G pixels, and B pixels in the Bayer array using the G complement image data D20a, G smoothed image data D20b, and the low-frequency component images using R pixels, B pixels, and G pixels, respectively, to generate second image data D22 in the Bayer array.

[0042] The phase difference processing unit 108 generates phase difference data for controlling the AF function of the imaging unit 100, which is generated using image data generated by pixels in which the shared on-chip lens L200 is arranged. Details of the phase difference data will be described later.

[0043] The encoding unit 110 divides and generates frame data including at least the second image data D22 into packet data in a predetermined format, and outputs the packet data to the control processing unit 200. Note that the encoding unit 110 according to this embodiment corresponds to a generating unit. Details of encoding will be described later.

[0044] (Imaging Modes) The captured image generation unit 104 according to this embodiment has multiple imaging modes. Mode 1 is a mode in which phase difference data is generated as first additional data, and the resolution of the displayed image is at a first level that is approximately normal. In this embodiment, data different from the second image data D22 is referred to as additional data. Note that the additional data is included in additional packet data, which will be described later. Furthermore, the data size of the frame data is fixed regardless of the imaging mode. Mode 1 does not require the control processing unit 200 to calculate phase difference data, and is therefore more suitable for dynamic imaging, for example.

[0045] This mode 1 has a mode 1a in which phase difference data da is generated as the first additional data based on first image data generated by pixels in which different shared on-chip lenses L200 are arranged, and a mode 1b in which phase difference data db is generated as the first additional data based on first image data generated by pixels in which the same shared on-chip lens L200 is arranged.

[0046] In mode 2, a portion of the first image data generated by the pixel in which the shared on-chip lens L200 is disposed is generated as second additional data. Mode 2 is a mode in which the second additional data is transmitted instead of the phase difference data da or the phase difference data db. In the second mode, data is transmitted using additional packet data allocated to transmitting the phase difference data da or the phase difference data db. The resolution of the display image in mode 2 is a second level higher than the first level. In mode 2, the phase difference data can be generated on the control processing unit 200 side. Mode 2 is a mode in which high resolution can be achieved while maintaining the same data transfer capacity as mode 1. Mode 2 is more suitable for, for example, still image capture or medium-speed dynamic capture.

[0047] Mode 3 is a mode in which at least a portion of the single-color data (green color) of the first image data D20 is used as the second additional data, and the first additional data and the second additional data are used as the additional data. The resolution of the displayed image in Mode 3 is a second level higher than the first level. Mode 3a is a mode in which the phase difference data da and the second additional data are used as the additional data, and Mode 3b is a mode in which the phase difference data db and the second additional data are used as the additional data. Mode 3 is a mode in which high resolution can be achieved while maintaining the same data transfer capacity as Mode 1. Mode 2 is more suitable for, for example, still image capture or medium-speed dynamic capture in which the subject may move at high speed.

[0048] Mode 4 is a mode in which multiple color data acquired from R and B pixel blocks (see FIG. 3 ) are used as third additional data, and the first, second, and third additional data are used as additional data. The resolution of the displayed image is a third level, which is higher than the second level. The operator can select the desired imaging mode via the operation input unit. Mode 4 is a mode that allows for the highest resolution while maintaining the same data transfer capacity as Mode 1. Mode 4 is more suitable for, for example, still image capture or slow-speed dynamic imaging. Modes 2 and 4 allow for the capacity of the first image data D20 to be increased by an amount equivalent to the capacity without adding phase difference data while maintaining the same data transfer capacity as Mode 1, thereby enabling higher resolution. Furthermore, phase difference data can be generated on the captured image generation unit 104 side.

[0049] The phase difference processing unit 108 generates phase difference data according to the imaging mode as the first additional data. When the phase difference data is not added, the phase difference processing unit 108 generates the phase difference data based on the first image data generated by the pixel in which the shared on-chip lens L200 (see FIG. 3) is arranged.

[0050] (Additional Data in Mode 1a) Fig. 7 is a diagram schematically showing an example of generation of phase difference data in Mode 1a. In Mode 1a, the phase difference processing unit 108 determines phase difference data TL1 by subtracting the G pixel value of TRa from the G pixel value of TLa, and determines phase difference data TL2 by subtracting the G pixel value of TRb from the G pixel value of TLb. The value obtained by adding TL1 and TL2 is then determined as phase data TL. Meanwhile, the phase difference data TR1 is determined as phase difference data TR1 by subtracting the G pixel value of TLa from the G pixel value of TRa, and the phase difference data TR2 is then determined as phase difference data TR.

[0051] Similarly, the phase difference data obtained by subtracting the G pixel value of BRa from the G pixel value of BLa is defined as BL1, and the phase difference data obtained by subtracting the G pixel value of BRb from the G pixel value of BLb is defined as BL2. The value obtained by adding BL1 and BL2 is defined as phase data BL. On the other hand, the phase difference data obtained by subtracting the G pixel value of BLa from the G pixel value of BRa is defined as BR1, and the phase difference data obtained by subtracting the G pixel value of BLb from the G pixel value of BRb is defined as BR2. The value obtained by adding BR1 and BR2 is defined as phase data BR. These phase difference data 24a (TL, TR, BL, BR) are output as additional data, four per 8x8 pixels, and are used to control the focus of the imaging system. The phase difference data 24a is an example of first additional data.

[0052] (Additional Data in Mode 1b) Fig. 8 is a diagram schematically showing an example of generation of phase difference data in Mode 1b. In Mode 1b, the phase difference processing unit 108 determines L1 as phase difference data obtained by subtracting the value of the G pixel of Ra from the value of the G pixel of La, and determines L2 as phase difference data obtained by subtracting the value of the G pixel of Rb from the value of the G pixel of Lb. The value obtained by adding L1 and L2 is then determined as phase data L. Meanwhile, the phase difference data obtained by subtracting the value of the G pixel of La from the value of the G pixel of Ra is then determined as phase difference data R1, and the phase difference data obtained by subtracting the value of the G pixel of Lb from the value of the G pixel of Rb is then determined as phase difference data R.

[0053] Similarly, in mode 1b, the phase difference processing unit 108 determines L1 as phase difference data obtained by subtracting the G pixel value of Rc from the G pixel value of Lc, and L2 as phase difference data obtained by subtracting the G pixel value of Rd from the G pixel value of Ld. The value obtained by adding L1 and L2 is then determined as phase data L. Meanwhile, the phase difference data obtained by subtracting the G pixel value of Lc from the G pixel value of Rc is determined as phase difference data R1, and the phase difference data obtained by subtracting the G pixel value of Ld from the G pixel value of Rd is determined as phase difference data R. Four pieces of phase difference data 24b (L, R, L, R) are output as additional data for each 8×8 pixel, and are used to control the focus of the imaging system. The phase difference data 24b is an example of first additional data.

[0054] 9 is a diagram schematically illustrating an example of generation of additional data in mode 2. In mode 2, the phase difference processing unit 108 sets the value of the G pixel of La to G(La) of the additional data, the value of the G pixel of Rb to G(Rb) of the additional data, the value of the G pixel of Lc to G(Lc) of the additional data, and the value of the G pixel of Rs to G(Rd) of the additional data, thereby generating second additional data 26a.

[0055] 10 is a diagram schematically illustrating an example of generating additional data in mode 4. In mode 4, the phase difference processing unit 108 generates second additional data 26a and third additional data 26b. In mode 4, the phase difference processing unit 108 sets the value of the G pixel of TLc to G(TLa) of the additional data, the value of the B pixel of TLd to B(TLd) of the additional data, the value of the R pixel of BRc to R(BRc) of the additional data, and the value of the B pixel of BRd to B(BRd) of the additional data, thereby generating the third additional data 26b.

[0056] (Packet data in mode 1a) Fig. 11 is a diagram showing packet data for one frame of data when transmitting second image data D22. One frame of data is made up of data D100 to D126. As described above, in this embodiment, the data for one frame of data is fixed regardless of the imaging mode.

[0057] The encoding unit 110 generates each of the data D100 to D126 using the data generated by the remosaic processing unit 106 and the phase difference processing unit 108. In FIG. 11, packet data is generated and output in chronological order in the direction indicated by the arrow t. The encoding unit 110 generates data packet data for each row of the second image data D22. After generating four rows of data packet data, the encoding unit 110 generates phase difference data 24a (TL, TR, BL, BR) corresponding to the four rows of packet data as two rows of additional packet data. The second image data D22 is included in the data region D108 of the data packet data. Meanwhile, the phase difference data 24a (TL, TR, BL, BR) is included in the data regions D110 and D112 of the additional packet data. The data regions D110 and D112 of each row are half the size of the data region D108. Therefore, the additional packet data includes dummy data regions D114 and D116. The data packet data according to this embodiment and the additional packet data are packet data of the same format.

[0058] (Packet Data in Mode 1b) Fig. 12 is a diagram showing packet data for one frame of second image data D22 when transmitted in Mode 1b. One frame of data is made up of D100 to D130. As described above, in this embodiment, the data for one frame of data is fixed regardless of the imaging mode.

[0059] The encoding unit 110 generates each of D100 to D130 using the data generated by the re-mosaic processing unit 106 and the phase difference processing unit 108. In FIG. 12, packet data is generated and output in chronological order in the direction indicated by the arrow t. The encoding unit 110 generates data packet data for each row of data corresponding to the second image data D22. After generating four rows of data packet data, the encoding unit 110 generates phase difference data 24b (L, R, L, R) corresponding to one row of packet data as one row of additional packet data. The encoding unit 110 also generates one row of dummy packet data. The second image data D22 is included in the data region D108 of the data packet data. Meanwhile, the phase difference data 24a (TL, TR, BL, BR) is included in the data region D128 of the additional packet data. The dummy packet data includes dummy data. The data packet data and the dummy packet data according to this embodiment are packet data of the same format.

[0060] (Packet Data in Mode 2) Fig. 13 is a diagram showing packet data for one frame of data when transmitting second image data D22 in Mode 2. One frame of data is made up of D100 to D130. Mode 2 differs from the packet data in Mode 1b in that the encoding unit 110 generates second additional data 26a (see Fig. 9) as additional packet data for one row instead of phase difference data 24b (L, R, L, R) corresponding to packet data for one row.

[0061] As shown in FIG. 13 , the encoding unit 110 divides the second image data D22 into partial row data D108 corresponding to multiple rows during image generation, generates the partial row data D108 as four data packet data in a predetermined data format, and generates a portion of the first image data D20 corresponding to the partial row data D108 (second additional data 26a) as additional packet data in a data packet data format. The encoding unit 110 then repeatedly outputs the four data packet data and the corresponding additional packet data in this order. In this way, the encoding unit 110 sequentially outputs four rows of the second image data D22 and the corresponding second additional data 26a. This allows the demosaic processing unit 204 (described later) to perform demosaic processing using the four rows of the second image data D22 and the corresponding second additional data 26a without waiting for the input of subsequent data.

[0062] (Packet data in mode 3a) Figure 14 is a diagram showing packet data for one frame when transmitting second image data D22 in mode 3a. Data for one frame is made up of D100 to D126. Packet data in mode 3a differs from packet data in mode 1a in that the encoding unit 110 generates additional packet data that includes second additional data 26a (see Figure 9) instead of dummy data, in addition to packet data in mode 1a.

[0063] (Packet Data in Mode 3b) Fig. 15 is a diagram showing packet data for one frame when transmitting second image data D22 in Mode 3b. One frame of data consists of D100 to D130. The packet data differs from the packet data in Mode 1b in that the encoding unit 110 generates additional packet data by including second additional data 26a (see Fig. 9) instead of the dummy data in the dummy data packet in addition to the packet data in Mode 1b.

[0064] (Packet Data of Mode 4) Fig. 16 is a diagram showing packet data for one frame when transmitting second image data D22 in Mode 4. One frame of data is made up of D100 to D132. The packet data differs from the packet data of Mode 2 in that the encoding unit 110 generates additional packet data by including third additional data 26b (see Fig. 10) instead of the dummy data of the dummy data packet in addition to the packet data of Mode 1b.

[0065] Fig. 17 is a block diagram showing an example of the configuration of the control processing unit 200. As shown in Fig. 17, the control processing unit 200 has a decoding unit 202, a design processing unit 204, a focus processing unit 206, and a display control unit 208.

[0066] The decoding unit 202 sequentially extracts data contained in the input packet data and outputs it to the design processing unit 204. Based on the input packet data, the design processing unit 204 performs demosaic processing on the second image data D22, whose pixel arrangement has been converted to a Bayer array, to convert it into third image data (developed image data) in which each pixel has luminance information for R, G, and B. Details of the demosaic processing will be described later. The display control unit 208 displays the developed image on the display unit 300 using the third image data generated by the design processing unit 204.

[0067] The focusing processing unit 206 controls the focus of the imaging system 103 based on the phase difference data input from the decoding unit 202. In addition, in modes 2 and 4, the focusing processing unit 206 generates phase difference data based on the second additional data 26a (see FIG. 9 ) and controls the focus of the imaging system 103. Note that the imaging system 103 is composed of multiple lenses or a single lens.

[0068] 18 is a diagram conceptually illustrating demosaic processing in modes 1a and 1b. In modes 1a and 1b, the first image data D20 is not included as additional data, so third image data (developed image data) I100 is generated using only the second data D22. The data conversion processing in this case is schematically shown by a processing matrix M100. f indicates the Nyquist frequency.

[0069] FIG. 19 is a conceptual diagram illustrating demosaic processing in mode 2. In mode 2, additional data 26a (see FIG. 9) is included. Therefore, high-resolution image data I104 can be generated using the additional data 26a and second data D22. Two G pixels in region D104 of image data I104 correspond to G pixels in the first image data D20 included in the additional data 26a. The data conversion process for generating third image data (developed image data) I106 in this case is schematically shown by processing matrix M102. Furthermore, in modes 2 and 4, the focus processing unit 206 generates phase difference data using data of the G pixels in region D104. In this way, phase difference data can be generated even when phase difference data is not included.

[0070] 20 is a diagram showing the Nyquist frequency of G pixels (≈ luminance) in developed image data. m1 shows a region where G pixels are not adjacent, as in the second data D22, and m2 shows a region where G pixels are adjacent, as in the image data I104. Since m1 does not have adjacent G pixels, the Nyquist frequency of G (≈ luminance) in the developed image is f / 4. Since m2 has adjacent G pixels, the Nyquist frequency of G (≈ luminance) in the developed image is f / 2. In this way, when high-resolution image data I104 is generated using additional data 26a and the second data D22, the third image data (developed image data) I106 is made higher in resolution.

[0071] 13, four rows of second data D22 and data corresponding to G pixels of the first image data D20 corresponding to the four rows of second data D22 are transmitted in sequence, so that developed images are generated in the order of transmission. This prevents a decrease in display speed. This type of processing can also be performed in mode 2, mode 3a, mode 3b, and mode 4, which include additional data 26a. In this case, since phase difference data is also included in modes 3a and b, the processing for generating phase difference data is not required.

[0072] In mode 4, because additional data 26b (see FIG. 10) is also included, the Nyquist frequency of R and B (≈ luminance) in the developed image data is also greater than f / 4, resulting in high resolution. As a result, the developed image generated in mode 4 has the highest resolution. Furthermore, in mode 4, the color reproducibility of the developed image is further improved. As described above, in mode 4, as in mode 2, phase difference data can be generated using the first image data D20.

[0073] As described above, the re-mosaic processing unit 106 generates second image data D22 by re-mosaicing the layout of the first image data D20. The encoding unit 110 then divides the second image data D22 into partial line data D108 corresponding to multiple lines during image generation, generates the partial line data D108 as four data packets in a predetermined data format, and generates a portion of the first image data D20 corresponding to the partial line data D108 (second additional data 26a) as additional packet data in a data packet data format. The encoding unit 110 then repeatedly outputs the four data packets and the corresponding additional packet data in this order. In this way, the encoding unit 110 sequentially outputs four lines of the second image data D22 and the corresponding second additional data 26a. This allows the demosaic processing unit 204 to perform demosaic processing using the four lines of the second image data D22 and the corresponding second additional data 26a without waiting for the input of subsequent data. This allows for increased resolution while minimizing the reduction in processing time required to generate the third image data (developed image data).

[0074] (Variation 1 of First Embodiment) The imaging device 1 according to Variation 1 of the first embodiment differs from the imaging device 1 according to the first embodiment in that the imaging device 1 generates the first image data D20 and the second image data D22 as packet data. The differences from the imaging device 1 according to the first embodiment will be described below.

[0075] 21 is a diagram showing an example of packet data generated by the encoding unit 110 according to Modification 1 of the first embodiment. The encoding unit 110 according to Modification 1 of the first embodiment generates packet data by treating the first image data D20 and the second image data D22 as one frame each. That is, the encoding unit 110 outputs all pixels in a so-called DOL format. Note that when all pixels of the first image data D20 are output, the frame rate (fps: frames per second) drops to approximately 6 / 8 (6 rows / 8 rows) compared to the second mode.

[0076] 22 is a diagram schematically illustrating the first image data D20, the second image data D22, and the third image data D24. Examples of 2x2, 3x3, and 4x4 Quad Bayer arrays are shown from top to bottom. Furthermore, the regions Dr104, Dg104, and Db104 of the third image data D24 represent pixel values ​​of the first image data D20, the pixels indicated by 2 represent pixel values ​​of the second image data D22, and the remaining regions represent pixel values ​​estimated by demosaic processing. As can be seen from these figures, the resolution of the third image data D24 is improved, and the resolution and color reproducibility of the developed image displayed on the display unit 300 are also improved.

[0077] (Modification 2 of First Embodiment) The imaging device 1 according to Modification 2 of the first embodiment differs from the imaging device 1 according to the first embodiment in that the imaging device 1 according to Modification 2 has different accumulation times for adjacent pixels 21. The differences from the imaging device 1 according to the first embodiment will be described below.

[0078] Fig. 23 is a diagram schematically showing a first image D20, a second image D22, and additional data 260a. As shown in Fig. 23, in the imaging device 1 according to the second modification of the first embodiment, adjacent pixels in the first image D20 have different accumulation times. The imaging time of pixel L is a long accumulation time, and the imaging time of pixel S is a short accumulation time that is shorter than the long accumulation time. That is, the sensitivity of pixel L is different from that of pixel S. The additional data 260a has the pixel value of pixel L. The generation of the additional data 260a is similar to that of Fig. 9. This makes it possible to expand the dynamic range and improve the resolution of dark areas.

[0079] FIG. 24 is a diagram schematically illustrating a first image D20, a second image D22, and additional data 2600a. As shown in FIG. 24, the additional data 2600a differs from FIG. 23 in that it performs same-color addition of pixel L. This allows for an expanded dynamic range and improved color noise (S / N) in dark areas. Same-color addition may be performed depending on the packet output length. The re-mosaic second image D22 and first image D20 can be output and used for development in various pixel arrays and pixel drives, not just the 2x2 array, according to a request from the control processing unit 200.

[0080] The present technology can be configured as follows:

[0081] (1) An imaging device comprising: an imaging unit in which pixel blocks each made up of a plurality of pixels of the same color are arranged; a first image generation unit that generates second image data by changing the arrangement of pixel data corresponding to each pixel based on first image data generated by the imaging unit; a generation unit that divides the second image data into partial line data corresponding to a plurality of lines when generating an image, generates the partial line data as a plurality of data packet data in a predetermined data format, generates a portion of the first image data corresponding to the partial line data as additional packet data in the predetermined data format, and repeatedly outputs the plurality of data packet data and the corresponding additional packet data in that order; and a second image generation unit that uses the data packet data and the additional packet data to generate third image data to be used for display on a display unit.

[0082] (2) The imaging device according to (1), wherein the partial row data has a range corresponding to the number of rows of the pixel block, and the second image generation unit converts the first image data into the third image data based on at least a portion of the first image data and the second image data.

[0083] (3) The imaging device according to (2), wherein a shared on-chip lens is formed in at least some of the pixel blocks to focus incident light onto pixels that generate image data for the pixel blocks.

[0084] (4) The imaging device according to (3), wherein the additional data included in the additional packet data is at least a part of fourth image data generated by a pixel in which the shared on-chip lens is arranged, or phase difference data for controlling an AF function generated using the fourth image data.

[0085] (5) The imaging device according to (4), wherein the additional data is at least a part of the phase difference data and the fourth image data, and is generated within the same additional packet data.

[0086] (6) The imaging device according to (5), further comprising: an imaging system that focuses the incident light; and a focusing processing unit that controls a focus of the imaging system based on the phase difference data.

[0087] (7) The imaging device according to (6), wherein the focusing processing unit controls a focus of the imaging system based on the phase difference data based on at least a part of the fourth image data.

[0088] (8) The imaging device according to any one of (1) to (7), wherein the second image data is re-mosaic image data in a Bayer array.

[0089] (9) The imaging device according to any one of (1) to (8), wherein the generation unit sequentially generates the image data for each row of the second image data as the data packet data, and outputs the plurality of data packet data corresponding to a predetermined row of the second image data and the additional packet data to the second image generation unit in a repeated sequence.

[0090] (10) The imaging device according to (3), wherein a capacity of frame data, which is the total data of the data packet data and the additional packet data, is fixed when the second image data is output to the second image generation unit, and the generation unit changes a combination of data included in the additional packet data depending on an imaging mode.

[0091] (11) The imaging device according to (4), having a plurality of imaging modes, wherein the pixel blocks correspond to red, blue, and green, respectively, the additional data is at least a part of the first image data corresponding to red, blue, and green, respectively, and the selection of the first image data of red, blue, and green differs depending on the imaging mode.

[0092] (12) The imaging device according to (11), wherein, when the additional data includes phase difference data according to the imaging mode, the additional data is at least a part of the first image data in the pixel block used to generate the phase difference data.

[0093] (13) The imaging device according to (12), wherein the pixel block in which the shared on-chip lens is arranged corresponds to the green color.

[0094] (14) The imaging device according to (11), wherein either a part of the first image data for red and blue, or a part of the first image data for green is selected depending on the imaging mode.

[0095] (15) The imaging device according to (1), wherein the additional data included in the additional packet data is all of the first image data.

[0096] (16) The imaging device according to (1), wherein the pixels in the pixel block formed by the plurality of pixels of the same color have different sensitivity characteristics, and the additional data included in the additional packet data is at least one of the image data having different sensitivity characteristics.

[0097] (17) The imaging device according to (16), wherein the second image generation unit generates the third image data in which a resolution of a dark area is adjusted based on the additional data.

[0098] (18) An imaging method for an imaging device having an imaging section in which pixel blocks each composed of a plurality of pixels of the same color are arranged, the imaging method comprising: a first image generation step of generating second image data by changing the arrangement of pixel data corresponding to each pixel based on first image data generated by the imaging section; a generation step of dividing the second image data into partial line data corresponding to a plurality of lines at the time of image generation, generating the partial line data as a plurality of data packet data in a predetermined data format, generating a portion of the first image data corresponding to the partial line data as additional packet data in the predetermined data format, and repeatedly outputting the plurality of data packet data and the corresponding additional packet data in that order; and a second image generation step of generating third image data to be used for display on a display section using the data packet data and the additional packet data.

[0099] (19) The imaging method according to (18), wherein in the second image generating step, conversion into the third image data is performed based on at least a part of the first image data and the second image data.

[0100] (20) The imaging method according to (19), wherein a shared on-chip lens is formed in at least some of the pixel blocks to focus incident light onto pixels that generate image data for the pixel blocks.

[0101] The aspects of the present disclosure are not limited to the individual embodiments described above, but include various modifications that may be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-described contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present disclosure, which is derived from the contents defined in the claims and their equivalents.

[0102] 1: imaging device, 100: imaging unit, 103: imaging system, 106: re-mosaic processing unit, 110: encoding unit, 206: focusing processing unit, D20: first image data, D22: second image data, D24: third image data, da, db: phase difference data.

Claims

1. An imaging device comprising: an imaging unit in which pixel blocks each composed of a plurality of pixels of the same color are arranged; a first image generation unit that generates second image data by changing the arrangement of pixel data corresponding to each pixel based on first image data generated by the imaging unit; a generation unit that divides the second image data into partial line data corresponding to a plurality of lines at the time of image generation, generates the partial line data as a plurality of data packet data in a predetermined data format, generates at least a portion of the first image data corresponding to the partial line data as additional packet data in the predetermined data format, and repeatedly outputs the plurality of data packet data and the corresponding additional packet data in that order; and a second image generation unit that uses the data packet data and the additional packet data to generate third image data to be used for display on a display unit.

2. The imaging device described in claim 1, wherein the partial row data is in a range corresponding to the number of rows of the pixel block, and the second image generation unit converts the data into the third image data based on at least a portion of the first image data and the second image data.

3. The imaging device according to claim 2, wherein a shared on-chip lens is formed in at least some of the pixel blocks for focusing incident light onto pixels that generate image data for the pixel block.

4. The imaging device according to claim 3, wherein the additional data included in the additional packet data is either at least a portion of fourth image data generated by a pixel in which the shared on-chip lens is arranged, or phase difference data for controlling an AF function generated using the fourth image data.

5. The imaging device according to claim 4, wherein the additional data is at least a part of the phase difference data and the fourth image data, and is generated within the same additional packet data.

6. The imaging device according to claim 5, further comprising: an imaging system that focuses the incident light; and a focusing processing unit that controls the focus of the imaging system based on the phase difference data.

7. The imaging device according to claim 6, wherein the focus processing section controls the focus of the imaging system based on the phase difference data based on at least a part of the fourth image data.

8. The imaging device according to claim 1, wherein the second image data is re-mosaic image data of a Bayer array.

9. The imaging device of claim 1, wherein the generation unit sequentially generates each of the image data for each row of the second image data as the data packet data, and outputs a plurality of the data packet data corresponding to a predetermined row of the second image data and the additional packet data to the second image generation unit in a repeated sequence.

10. An imaging device as described in claim 3, wherein the capacity of the frame data, which is the total data of the data packet data and the additional packet data when the second image data is output to the second image generation unit, is fixed, and the generation unit changes the combination of data contained in the additional packet data depending on the imaging mode.

11. The imaging device of claim 4, having a plurality of imaging modes, the pixel blocks corresponding to red, blue, and green, respectively, the additional data being at least a portion of the first image data corresponding to red, blue, and green, respectively, and the selection of the first image data of red, blue, and green differs depending on the imaging mode.

12. The imaging device described in claim 11, wherein, when the additional data includes phase difference data depending on the imaging mode, the additional data is at least a portion of the first image data in the pixel block used to generate the phase difference data.

13. The imaging device according to claim 12, wherein the pixel block in which the shared on-chip lens is arranged corresponds to the color green.

14. The imaging device according to claim 11, wherein either a portion of the first image data for red and blue, or a portion of the first image data for green is selected depending on the imaging mode.

15. The imaging device according to claim 1, wherein the additional data included in the additional packet data is the entirety of the first image data.

16. The imaging device according to claim 1, wherein the pixels in the pixel block constituted by the plurality of pixels of the same color have different sensitivity characteristics, and the additional data included in the additional packet data is at least one of the image data having different sensitivity characteristics.

17. The imaging device according to claim 16, wherein the second image generating section generates the third image data in which the resolution of a dark area is adjusted based on the additional data.

18. An imaging method for an imaging device having an imaging section in which pixel blocks composed of a plurality of pixels of the same color are arranged, comprising: a first image generation step of generating second image data by changing the arrangement of pixel data corresponding to each pixel based on first image data generated by the imaging section; a generation step of dividing the second image data into partial line data corresponding to a plurality of lines at the time of image generation, generating the partial line data as a plurality of data packet data in a predetermined data format, generating at least a portion of the first image data corresponding to the partial line data as additional packet data in the predetermined data format, and repeatedly outputting the plurality of data packet data and the corresponding additional packet data in that order; and a second image generation step of generating third image data to be used for display on a display section using the data packet data and the additional packet data.

19. The imaging method according to claim 18, wherein in said second image generating step, conversion to said third image data is performed based on at least a part of said first image data and said second image data.

20. The imaging method according to claim 19, wherein at least some of said pixel blocks include a shared on-chip lens formed therein for focusing incident light onto pixels that generate image data for said pixel blocks.

Images