Imaging element, imaging device, information processing method, and program

Abstract

This imaging element comprises: a pixel array part in which pixels for outputting a light reception signal in accordance with a light reception amount are arranged in a row direction and a column direction; and a color filter provided for each pixel block composed of the plurality of pixels. The pixel array part outputs the light reception signal from the pixels arranged in a partial region of the pixel array part in accordance with a zooming operation. As output modes of the light reception signal in accordance with the zooming operation, provided are: a binning mode in which one light reception signal is output for each pixel block arranged in the partial region; and a non-binning mode in which one light reception signal is output for each pixel arranged in the partial region. The non-binning mode is set as a mode to be selected when a predetermined condition is satisfied.

Description

Image pickup element, image pickup device, information processing method, and program

[0001] The present technology relates to an image sensor, an image pickup device, an information processing method, and a program capable of pixel binning, which outputs one received light signal from a plurality of pixels.

[0002] Image sensors are becoming increasingly multi-pixelated. While this allows for the acquisition of high-resolution images, it can also cause the number of pixels to be too large for general use. To address this issue, a technology known as pixel binning (hereinafter simply referred to as "binning") is known, as described in Patent Document 1 below.

[0003] Special Publication No. 2022-535292

[0004] Binning is a so-called additive readout method in which the charges accumulated in multiple pixels are read out together and output as a single light-receiving signal. However, when binning is combined with the camera's zoom function, the image quality becomes excessively coarse.

[0005] Therefore, an object of the present disclosure is to optimize image quality during zooming by suitably combining the zoom function of a camera with binning.

[0006] The imaging element according to the present technology includes a pixel array section in which pixels are arranged in row and column directions, and each pixel array section outputs a light reception signal corresponding to an amount of received light. The pixel array section outputs the light reception signal from pixels arranged in a partial region of the pixel array section in response to a zooming operation. The pixel array section provides output modes for the light reception signal corresponding to the zooming operation, including a binning mode in which one light reception signal is output for each pixel block arranged in the partial region, and a non-binning mode in which one light reception signal is output for each pixel arranged in the partial region. The non-binning mode is a mode selected when a predetermined condition is met. When a zooming operation toward the telephoto side, i.e., a zoom-in operation, is continuously performed, the crop range of the pixel array section is narrowed as a zoom-in process. In the binning mode, image data of a predetermined size is generated and displayed while the crop range is narrowed while additive readout is continued. In the non-binning mode, additive readout is stopped, and image data of a predetermined size is generated and displayed while the light reception signal is output for each pixel. When switching between the binning mode and the non-binning mode in the zoom-in process, whether or not a predetermined condition is met is taken into consideration.

[0007] The imaging device according to the present technology includes a control unit that performs zooming processing in accordance with the detection result of a zooming operation, a pixel array unit in which pixels that output light reception signals in accordance with the amount of received light are arranged in row and column directions and cause the pixels arranged in a partial region to output the light reception signals in accordance with the zooming operation, and a color filter provided for each pixel block consisting of a plurality of the pixels, and the control unit switches between a binning mode in which one light reception signal is output for each pixel block arranged in the partial region and a non-binning mode in which one light reception signal is output for each pixel arranged in the partial region, depending on whether a predetermined condition is met.

[0008] In an information processing method according to the present technology, an information processing device performs a process of selecting, when a predetermined condition is met, a binning mode in which a light reception signal corresponding to the amount of received light is output for each pixel block, which is made up of a plurality of pixels and arranged in a partial region of a pixel array unit, in response to a zooming operation, and a non-binning mode in which a light reception signal corresponding to the amount of received light is output from each of a plurality of pixels arranged in the partial region in response to the zooming operation.

[0009] The program according to the present technology causes an information processing device to execute a process of selecting, when a predetermined condition is met, one of a binning mode in which a light reception signal corresponding to the amount of received light is output for each pixel block, consisting of a plurality of pixels, arranged in a partial region of a pixel array unit in response to a zooming operation, and a non-binning mode in which a light reception signal corresponding to the amount of received light is output from each of a plurality of pixels arranged in the partial region in response to the zooming operation.

[0010] 1 is a block diagram of an imaging device according to an embodiment of the present technology. FIG. 2 is a diagram illustrating an example of pixel arrangement and color filter arrangement in a pixel array unit according to an embodiment. FIG. 3 is an explanatory diagram of re-mosaic processing in the imaging device according to an embodiment. FIG. 4 is a schematic diagram of a display image obtained by zooming processing in a binning mode according to an embodiment. FIG. 5 is a schematic diagram of a display image obtained by zooming processing in a non-binning mode according to an embodiment. FIG. 6 is a flowchart illustrating an example of processing executed by a control unit according to an embodiment. FIG. 7 is a flowchart illustrating an example of condition satisfaction confirmation processing executed by a control unit according to an embodiment. FIG. 8 is an explanatory diagram illustrating changes in a readout range, a crop range, and a displayed image when a zoom magnification is changed in the imaging device according to an embodiment. FIG. 9 is a diagram illustrating an example in which a threshold value used in the condition satisfaction confirmation processing in the imaging device according to an embodiment is variable. FIG. 10 is a flowchart of a modified example of an embodiment.

[0011] Hereinafter, the embodiments will be described in the following order: <1. Configuration of imaging device> <2. Processing flow> <3. Relationship between zoom-in operation and each range> <4. Modification> <5. Summary> <6. Present technology>

[0012] In this disclosure, the term "image" is used to include both still images and moving images.

[0013] <1. Configuration of Imaging Device> As shown in FIG. 1 , the imaging device 1 includes an imaging element 2 of the present technology, an optical system 3 that causes light to be incident on the imaging element 2, a signal processing unit 4 that performs predetermined signal processing using a signal that corresponds to the amount of received light and is output from the imaging element 2, and a control unit 5 that performs overall control of the imaging device 1.

[0014] The imaging device 1 further includes a storage unit 6 for storing image data obtained based on the light receiving signal, a communication unit 7 for transmitting the image data to the outside, etc. The imaging device 1 may also be provided with a display unit used for checking the image after capture, etc.

[0015] The imaging element 2 is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, etc. The imaging element 2 is configured with a pixel array section 9 in which pixels 8 are arranged in row and column directions, and a readout circuit that reads out electrical signals, which are light-receiving signals obtained from each pixel 8 through photoelectric conversion.

[0016] In the readout circuit of the image sensor 2, for example, correlated double sampling (CDS) processing, automatic gain control (AGC) processing, etc. are performed on the electrical signals obtained by photoelectric conversion, and further, analog to digital (A / D) conversion processing is performed. Image data (RAW image data) is output as digital data from the image sensor 2.

[0017] Each pixel 8 in the pixel array section 9 is provided with a color filter 10 .

[0018] FIG. 2 shows an example of the arrangement of the pixels 8 and the color filters 10 in the pixel array section 9 of the image sensor 2.

[0019] Among the pixels 8, a pixel 8 that is provided with a color filter 10R that transmits R (red) light and performs photoelectric conversion on R light is referred to as pixel 8R. Among the pixels 8, a pixel 8 that is provided with a color filter 10G that transmits G (green) light and performs photoelectric conversion on G light is referred to as pixel 8G. Furthermore, among the pixels 8, a pixel 8 that is provided with a color filter 10B that transmits B (blue) light and performs photoelectric conversion on B light is referred to as pixel 8B.

[0020] In the pixel array unit 9 of the embodiment, color filters 10 of the same color are provided for one pixel block 11 consisting of four pixels 8. That is, the pixel array unit 9 is provided with the color filters 10 so as to form a Bayer array with the pixel block 11 as the reference.

[0021] 2, the color filter 10R is indicated by hatching with diagonal lines going up to the right, the color filter 10G is indicated by hatching with vertical lines, and the color filter 10B is indicated by hatching with diagonal lines going down to the right.

[0022] One color filter 10 may be provided for four pixels 8 included in a pixel block 11, or color filters 10 of the same color may be provided for each pixel 8.

[0023] The optical system 3 includes various lenses such as a cover lens, a zoom lens, and a focus lens, as well as an iris mechanism. The optical system 3 guides light from the subject (incident light) and focuses it on the light receiving surface of the pixel array unit 9 of the image sensor 2.

[0024] The signal processing unit 4 performs various necessary processes such as pre-processing, re-mosaic processing and demosaic processing (described later), YC generation processing, resolution conversion processing, and codec processing on the RAW image data supplied from the image sensor 2. The pre-processing includes clamping processing for clamping the R, G, and B black levels of the captured image signal to predetermined levels, and correction processing between the R, G, and B color channels.

[0025] The re-mosaic process is a process that performs interpolation or the like so that pixels 8 have color component data conforming to the Bayer array. Specifically, in the upper left pixel block 11 shown in Figure 2, all four pixels 8 are pixels 8R that output light reception signals for R light. The re-mosaic process performs interpolation so that, of these four pixels 8R, the upper left pixel 8R has pixel data of the R component, the lower right pixel 8R has pixel data of the B component, and the upper right and lower left pixels 8R have pixel data of the G component (see Figure 3). Note that the pixels 8 shown in parentheses in Figure 3 have color component data obtained by interpolation.

[0026] Demosaic processing is a process that uses color component data in a Bayer array to perform interpolation or the like so that image data for each pixel 8 has color component data for all R, G, and B. By performing re-mosaic processing before demosaic processing, it is possible to apply normal demosaic processing as is.

[0027] In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from the R, G, and B image data for each pixel 8 .

[0028] In the resolution conversion process, resolution conversion is performed on image data that has undergone various signal processes. In the codec process, the image data that has undergone the various processes described above is encoded and a file is generated for recording or communication, for example. In the codec process, moving image file formats such as MPEG-2 (Moving Picture Experts Group) and H.264 can be generated. It is also possible to generate still image files in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format).

[0029] The control unit 5 is configured with a microcomputer having, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The CPU executes various processes in accordance with programs stored in the ROM or programs loaded into the RAM, thereby realizing overall control of the imaging device 1 by the control unit 5.

[0030] The control unit 5 issues instructions to the driver unit 12 to drive the zoom lens, focus lens, diaphragm mechanism, etc., which constitute the optical system 3. The driver unit 12 supplies drive voltages to each unit, such as an actuator, based on instructions from the control unit 5, thereby realizing the movement of the focus lens and zoom lens, the opening and closing of the diaphragm blades of the diaphragm mechanism, etc.

[0031] The control unit 5 controls the writing and reading of various data to and from the storage unit 6 .

[0032] The control unit 5 transmits and receives information to and from an external information processing device via the communication unit 7. For example, the control unit 5 transmits image data obtained by the signal processing unit 4 or RAW image data obtained from the image sensor 2 that has been converted into a predetermined data format to the external information processing device.

[0033] The control unit 5 switches the output mode of the light reception signal from the image sensor 2 between a binning mode M1 and a non-binning mode M2.

[0034] The binning mode M1 is a mode in which so-called additive reading is performed, in which the charges accumulated in four pixels 8 are read out collectively and output as one light-receiving signal. In other words, the binning mode M1 is a mode in which additive reading is performed for each pixel block 11, and one light-receiving signal is output for each pixel block 11.

[0035] The non-binning mode M2 ​​is a mode in which a light-receiving signal is read out from each of the four pixels 8. That is, the non-binning mode M2 ​​is a mode in which four light-receiving signals are read out from one pixel block 11.

[0036] In binning mode M1, for example, 12M pixel RAW data is generated. In non-binning mode M2, for example, 48M pixel RAW data is generated. The RAW data with different resolutions obtained in binning mode M1 and non-binning mode M2 ​​is subjected to resolution conversion processing and converted into image data with a desired number of pixels. Therefore, even if the converted image data has the same number of pixels, the perceived resolution and image quality will be different between the data obtained in binning mode M1 and the data obtained in non-binning mode M2.

[0037] The results of the mode switching by the control unit 5 are supplied to the image sensor 2. This determines whether or not the image sensor 2 will perform additive readout.

[0038] The binning mode M1 and the non-binning mode M2 ​​are switched in response to, for example, a zooming operation by the user.

[0039] Here, we will explain the zooming operation to the telephoto side in binning mode M1, that is, the zooming in operation. Note that Fig. 4 is a diagram for explaining a display image obtained by zooming in binning mode M1, and Fig. 5 is a diagram for explaining a display image obtained by zooming in non-binning mode M2.

[0040] 4 and 5, the relationship between the display image and the crop area Ac is shown, and the display image and the number of light receiving signals used to generate the display image are shown. In addition, the rectangle indicated by the dashed line in each figure is an area cut out from the RAW image, and is a schematic representation of the cut-out area (crop area Ac).

[0041] When a zoom-in operation is performed, light reception signals are output from pixels 8 in a partial region 13 approximately in the center of the pixel array section 9. At this time, since the binning mode M1 is a mode in which light reception signals are output for each pixel block 11, when a zoom-in operation is performed, a low-resolution RAW image is obtained (see FIG. 4). Because the displayed image is obtained by performing resolution conversion by up-conversion on the low-resolution RAW image, the perceived resolution is lowered and the image quality is degraded.

[0042] Next, a zoom-in operation in non-binning mode M2 ​​will be described. When a zoom-in operation is performed, light reception signals are read out only from the pixels 8 in the partial region 13. Since non-binning mode M2 ​​is a mode in which light reception signals are output from each pixel 8 included in the partial region 13, when a zoom-in operation is performed, a RAW image with higher resolution than in binning mode M1 is obtained (see FIG. 5). The displayed image is obtained by performing a resolution conversion process on a RAW image with a relatively high resolution, resulting in a higher perceived resolution and higher image quality. Note that, if the number of pixels included in the crop area Ac in FIG. 5 matches the number of pixels in the displayed image, resolution conversion process does not need to be performed.

[0043] The control unit 5 receives a zoom-in operation by the user and determines whether to perform the zoom-in process in the binning mode M1 or the non-binning mode M2 ​​based on the zoom-in operation and various conditions. This determination process will be described in detail later.

[0044] The imaging device 1 is provided with an operation unit 14 for receiving zooming operations such as a zoom-in operation and a zoom-out operation performed by a user. The operation unit 14 for performing zooming operations may be provided as a physical operator such as a jog dial, a slide bar, or a zoom ring, or may be provided as an icon operator displayed on a display unit (not shown) provided as a touch panel.

[0045] 6 is a flowchart showing an example of processing executed by the control unit 5 of the imaging device 1. In step S101, the control unit 5 determines whether or not a zooming operation has been detected.

[0046] If it is determined that a zooming operation has not been detected, the control unit 5 performs the process of step S101 again. On the other hand, if it is determined that a zooming operation has been detected, the control unit 5 proceeds to step S102. Note that the detection of a zooming operation here may include not only an operation that directly instructs zooming, but also an operation that results in zooming.

[0047] In step S102, the control unit 5 performs a condition satisfaction confirmation process, which is a process for confirming whether predetermined conditions are met, and will be described in detail later.

[0048] If the control unit 5 determines in step S103 that the predetermined condition is met, it sets the non-binning mode M2 ​​to ON in step S104. That is, in frames after this setting, a light reception signal is output for each pixel 8 included in the partial region 13 as shown in FIG.

[0049] On the other hand, if it is determined in step S103 that the predetermined condition is not met, the control unit 5 sets the binning mode M1 to ON in step S105. That is, in frames after this setting, additive reading is performed on the pixel blocks 11 included in the effective pixel area, as shown in FIG. 4, and a light reception signal is output for each pixel block 11.

[0050] After executing the process of either step S104 or step S105, the control unit 5 sets the cropping range Ac in accordance with the zoom magnification in step S106.

[0051] The zoom magnification includes an optical zoom magnification, which is a zoom magnification that can be achieved by moving the zoom lens of the optical system 3, and a non-optical zoom magnification, which is a magnification for non-optical zoom that is achieved when a zoom-in operation is performed after the optical zoom magnification is set to maximum. The zoom magnification in step S106 refers to the non-optical zoom magnification. In other words, if the desired zoomed-in image can be obtained by changing the optical zoom magnification, the zoom magnification (non-optical zoom magnification) in step S106 is set to 1.0.

[0052] The control unit 5 acquires the cropped image in step S107. Note that the processes in steps S106 and S107 may be realized in cooperation with the signal processing unit 4 and the image sensor 2.

[0053] In step S108, the control unit 5 causes the signal processing unit 4 to perform enlargement or reduction processing according to the image size. This makes it possible to display an image that has been enlarged or reduced by resolution conversion processing in accordance with the non-optical zoom magnification based on the zooming operation on a display unit (not shown), or to generate an image file that has been enlarged or reduced in accordance with the non-optical zoom magnification. Note that enlargement and reduction of an image by resolution conversion processing is not essential, and is not necessary if the number of pixels in the readout range Ar set on the pixel array unit 9 matches the number of pixels of the image.

[0054] 7 shows an example of the process of confirming whether the condition is met in step S102 in FIG. 6. In step S201, the control unit 5 determines whether the continuous shooting mode is selected as the shooting mode. If the continuous shooting mode is selected, the control unit 5 proceeds to step S202 and determines that the predetermined condition is not met. That is, if the determination in step S202 is made, the control unit 5 sets the binning mode M1 to ON in step S105 in FIG. 5 and causes a signal to be output for each pixel block 11.

[0055] On the other hand, if it is determined in step S201 that the continuous shooting mode is not selected, the control unit 5 further determines in step S203 whether or not high frame rate shooting is being performed.

[0056] If it is determined that high frame rate shooting is being performed, the control unit 5 proceeds to the process of step S202.

[0057] In step S204, the control unit 5 determines whether the non-optical zoom magnification is less than a threshold. For example, if the threshold is set to "2x," the control unit 5 proceeds to step S202 if the non-optical zoom magnification is less than 2x. That is, a situation in which the determination in step S204 is "Yes" is considered to be a situation in which it can be determined that the resolution of the RAW image will not decrease to the extent that a light reception signal is output for each pixel 8.

[0058] In step S205, the control unit 5 determines whether the average luminance value for the predetermined region is less than a threshold value. The predetermined region is a region where a signal based on the amount of received light is output, and may be, for example, an effective pixel region or a central region of the angle of view. Alternatively, the predetermined region may be a region where an image of a subject is captured.

[0059] If it is determined that the average luminance value of the predetermined region is less than the threshold value, the control unit 5 proceeds to step S202.

[0060] In step S206, the control unit 5 determines whether the subject to be imaged is a moving subject. The subject to be imaged may be, for example, a person located near the center of the angle of view, a subject specified by the user, or a subject to be focused. The moving subject refers to, for example, a subject whose motion vector between frames is equal to or greater than a predetermined magnitude.

[0061] If the subject being imaged is a moving subject, binning mode M1, which requires a relatively short time for the reading process to suppress blur of the subject, is preferable, so the control unit 5 determines "Yes" in step S205 and proceeds to step S202.

[0062] Furthermore, in step S207, the control unit 5 determines whether the dynamic range of the subject to be imaged or the above-mentioned predetermined area is wider than the threshold. If it is determined that the dynamic range is wider than the threshold, the control unit 5 determines "Yes" in step S207 and proceeds to step S202, taking into consideration the need to perform HDR (High Dynamic Range) imaging. Note that the predetermined area here may be an area equivalent to the effective pixel range.

[0063] If all the conditions in steps S201, S203, S204, S205, S206, and S207 are not met and the determination is "No," the control unit 5 proceeds to step S208 and determines that the predetermined condition is met. As a result, the control unit 5 sets the non-binning mode M2 ​​to ON in step S104 of Fig. 6, and causes each pixel 8 to output a light reception signal in the subsequent frames.

[0064] 3. Relationship between Zoom-in Operation and Each Range When the user changes the zoom magnification by operating the operation unit 14, specifically when a zoom-in operation to the telephoto side is performed, an example of the change in the readout range Ar and the crop range Ac, and the change in the image displayed on the display unit, is shown in FIG. 8 .

[0065] The example shown in Figure 8 is a case where the user performs a zoom-in operation from time t1 to time t2. During the zoom-in operation from the wide-angle side to the telephoto side, there is a time (time t3) when the non-optical zoom magnification is set to 2.0x. This non-optical zoom magnification of 2.0x corresponds to the threshold value in step S204 in Figure 7.

[0066] 8 shows an example in which four frame images are output between time t1 and time t2, but this is for illustrative purposes only, and in reality, a zoom-in operation may be performed over a period of time in which many more frame images are output. The four frames in FIG. 8 are referred to in chronological order as a first frame fr1, a second frame fr2, a third frame fr3, and a fourth frame fr4.

[0067] In each frame, exposure of each line starts in sequence at approximately the same time as the frame start timing, and light reception signals according to the amount of received light are output in sequence from the line for which exposure has been completed.

[0068] In the first frame fr1, for example, the non-optical zoom magnification is set to 1.0, and the entire effective pixel range in the pixel array section 9 is set as the readout range Ar (the partial area 13 described above) and the crop range Ac.

[0069] Each of the four rectangles (hereinafter referred to as "rectangular areas") shown in the upper left of the image for explaining the readout range Ar and crop range Ac and in the upper left of the displayed image indicates a light-receiving area corresponding to one light-receiving signal. That is, in the first frame fr1, one rectangle corresponds to one pixel block 11.

[0070] Next, in the second frame fr2, which arrives when the non-optical zoom magnification is greater than 1.0x but less than 2.0x, the entire effective pixel range is set as the readout range Ar. The crop range Ac is set as an area inside the effective pixel range. At this time, the image displayed on the display unit is up-converted as a resolution conversion process, resulting in an image with lower resolution and image quality than the first frame fr1.

[0071] The rectangular area in the second frame fr2 corresponds to pixel block 11. As shown in the figure, the rectangular area on the display image is larger in the second frame fr2 than in the first frame fr1.

[0072] In the third frame fr3, which starts at time t3 when the non-optical zoom magnification reaches 2.0, signals are read out in the non-binning mode M2, where a quarter of the effective pixel range in the pixel array unit 9 (the central region) is set as the readout range Ar and the crop range Ac.

[0073] Furthermore, the rectangular area in the third frame fr3 corresponds to pixel 8. As shown in the figure, the rectangular areas on the display image are approximately the same size in the third frame fr3 and the first frame fr1. In other words, the degradation in resolution and image quality that occurred in the second frame fr2 has been eliminated in the third frame fr3. Note that in the third frame fr3, the number of pixels 8 included in the cropping range Ac is the same as the number of pixels in the display image, so resolution conversion processing does not need to be performed.

[0074] In the fourth frame fr4, which arrives when the non-optical zoom magnification exceeds 2.0, a quarter of the effective pixel range (the central region) is set as the readout range Ar, and the crop region Ac is set as the region inside the readout range Ar.

[0075] At this time, the image displayed on the display unit is subjected to up-conversion processing as a resolution conversion process, resulting in an image with lower resolution and image quality than the third frame fr3. As shown in the figure, the rectangular area on the displayed image is larger in the fourth frame fr4 than in the third frame fr3.

[0076] 4. Modifications In the above example, the various thresholds used in the condition satisfaction confirmation process shown in Fig. 7 are fixed thresholds. However, the present invention is not limited to this, and the various thresholds may be changed depending on the conditions.

[0077] An example is shown in Fig. 9. Fig. 9 shows an example in which the threshold is changed according to the average brightness value of a predetermined region, specifically, an example in which the threshold in step S205 is changed according to the average brightness value.

[0078] As shown in the figure, in a shooting environment where bright images can be acquired, the threshold for the non-optical zoom magnification is set to 2.0x, so that the mode switches from binning mode M1 to non-binning mode M2 ​​before the perceived resolution deteriorates too much.

[0079] In a shooting environment where only dark images can be captured, the threshold for the non-optical zoom magnification can be set higher than 2.0, such as to 3.0, to prevent the image from becoming too dark, while allowing for a decrease in resolution.

[0080] The slope of the line indicating the boundary between the binning mode M1 and the non-binning mode M2 ​​shown in Fig. 9 can be changed depending on the situation. For example, if emphasis is placed on perceived resolution and it is desired to minimize degradation of perceived resolution, the slope can be made larger to cause an earlier transition to the non-binning mode M2.

[0081] On the other hand, when emphasis is placed on the brightness of the generated image, the gradient is made smaller to delay the transition to the non-binning mode M2.

[0082] This allows transition to the non-binning mode M2 ​​at an appropriate timing in response to a request for an image to be generated or displayed.

[0083] 9 shows an example in which the threshold value compared with the average luminance value in step S205 is variable. Similarly, the threshold value used in determining whether or not the subject is a moving subject in step S206 and the threshold value used in determining whether or not the subject has a high dynamic range in step S207 can also be variable.

[0084] For example, in step S206, the vertical axis in Fig. 9 is replaced with the magnitude of the motion vector between frames. In this case, the higher the vertical axis of the graph, the larger the motion vector. This makes it less likely to transition to non-binning mode M2, even if the zoom magnification is increased, as the movement of the moving subject becomes more intense.

[0085] 9 is replaced with the width of the dynamic range. In this case, the lower the vertical axis of the graph, the narrower the dynamic range. This makes it more difficult to transition to non-binning mode M2 ​​even if the zoom magnification is increased as the dynamic range becomes wider.

[0086] Next, a modified example will be described in which the user specifies a combined zoom magnification that is the sum of the optical zoom magnification and the non-optical zoom magnification.

[0087] Here, the maximum optical zoom magnification is set to 3.0.

[0088] When the user specifies a combined zoom magnification of 1.0 to 3.0, the optical zoom is used, in which case the effective pixel range in the pixel array unit 9 is set as the read range Ar and the crop range Ac.

[0089] If the user specifies a combined zoom magnification of more than 3.0x, the optical zoom magnification is maximized and non-optical zoom is also used. In this case, in the example above, operation is in binning mode M1 until the non-optical zoom exceeds 2.0x, i.e., until the combined zoom magnification exceeds 6.0x. Then, once the non-optical zoom exceeds 2.0x, i.e., once the combined zoom magnification exceeds 6.0x, operation is in non-binning mode M2.

[0090] An example of processing executed by the control unit 5 in this modified example is shown in Fig. 10. Note that processing similar to that described in Fig. 6 is given the same step numbers, and description thereof will be omitted where appropriate.

[0091] If it is determined in step S101 that a zooming operation has been detected, the control unit 5 determines in step S121 whether the total zoom magnification specified by the user is equal to or less than the maximum optical zoom magnification. If it is determined that the total zoom magnification is equal to or less than the maximum optical zoom magnification, the control unit 5 sets the binning mode M1 to ON in step S105 without performing the condition satisfaction confirmation process in step S102.

[0092] That is, the confirmation process of step S102 is executed only when the set total zoom magnification is greater than the maximum optical zoom magnification, which reduces the frequency of the various determination processes shown in FIG. 7 and reduces the processing load on the control unit 5.

[0093] 5. Summary As described in the examples above, the image sensor 2 of the present technology includes a pixel array unit 9 in which pixels 8 are arranged in row and column directions, and each pixel 8 outputs a light reception signal corresponding to the amount of received light. The pixel array unit 9 outputs a light reception signal from pixels 8 arranged in a partial region 13 of the pixel array unit 9 in response to a zooming operation. The pixel array unit 9 provides two output modes for the light reception signal corresponding to the zooming operation: a binning mode M1 in which one light reception signal is output for each pixel block 11 arranged in the partial region 13, and a non-binning mode M2 ​​in which one light reception signal is output for each pixel 8 arranged in the partial region 13. The non-binning mode M2 ​​is a mode selected when a predetermined condition is met. When a zooming operation toward the telephoto side, i.e., a zoom-in operation, is continuously performed, a zoom-in process is performed in which the crop range Ac in the pixel array unit 9 is narrowed. In this case, in the binning mode M1, image data of a predetermined size is generated and displayed as appropriate while the crop range Ac is narrowed while continuing additive reading. On the other hand, in non-binning mode M2, additive readout is halted and image data of a predetermined size is generated and displayed while outputting a light-receiving signal for each pixel 8. When switching between binning mode M1 and non-binning mode M2 ​​during zoom-in processing, the fulfillment of certain conditions is taken into consideration. By considering the fulfillment of certain conditions, it is possible to avoid inappropriate zoom-in processing that would result in a desired image not being obtained. Examples of inappropriate zoom-in processing include processing that generates and displays an image that is too dark due to low exposure, or processing that generates and displays an image with a lot of noise as a result of performing exposure correction to optimize the exposure of an image that is too dark. Furthermore, image capture in non-binning mode M2 ​​requires longer readout processing than in binning mode M1. Therefore, inappropriate zoom-in processing also occurs when switching modes such that a high frame rate is required to obtain a suitable image.

[0094] As described with reference to Figures 4, 5, and 8, the partial region 13 of the image sensor 2 may be the central region of the pixel array unit 9. This allows the zoom-in process to be performed so that the subject captured at the center of the angle of view does not fall out of the frame, even when a zoom-in operation is performed with the subject captured at the center of the angle of view. Note that the zoom-in process may be performed so that the enlargement is centered on the designated region or pixel 8. In other words, the zoom-in process may be performed so that the designated region or pixel 8 approaches the center of the angle of view in response to the zoom-in operation. This allows the zoom-in process to be performed without the subject falling out of the frame, even when a zoom-in operation is performed with the specified subject positioned outside the center of the angle of view.

[0095] As described with reference to Fig. 8 and other figures, in the image sensor 2, the partial region 13 in the non-binning mode M2 ​​may be narrower than the partial region 13 in the binning mode M1. This makes it possible to reduce the number of light reception signals output compared to when light reception signals are output from all pixels 8 included in the effective pixel range in the non-binning mode M2. This makes it possible to reduce the burden of the light reception signal readout process and subsequent processing.

[0096] The imaging device 1 of the present technology includes a control unit 5 that performs zooming processing in response to a detection result of a zooming operation, a pixel array unit 9 in which pixels 8 are arranged in row and column directions and output light reception signals corresponding to the amount of received light from the pixels 8 arranged in a partial region 13 in response to a zooming operation, and a color filter 10 provided for each pixel block 11 consisting of a plurality of pixels 8. The control unit 5 switches between a binning mode M1 in which one light reception signal is output for each pixel block 11 arranged in the partial region 13 and a non-binning mode M2 ​​in which one light reception signal is output for each pixel 8 arranged in the partial region 13, depending on whether a predetermined condition is met. This allows the same effect as achieved by the imaging element 2 described above to be obtained. Note that some imaging elements 2 include an internal arithmetic processing unit equivalent to the control unit 5. Such an imaging element 2 can be considered as a device that captures images. In other words, the imaging element 2 arranged inside the camera housing of the present technology itself can be considered as one aspect of the imaging device 1.

[0097] 6 and 7, the control unit 5 in the imaging device 1 may determine that a predetermined condition is met and switch to non-binning mode M2 ​​when at least image capture in continuous shooting mode is not being performed. In non-binning mode M2, the image capture interval becomes longer, which is disadvantageous to continuous shooting mode. Therefore, by determining that a predetermined condition is met when at least continuous shooting mode is not being performed, in other words, by executing binning mode M1 during continuous shooting mode, it is possible to prevent a decrease in the continuous shooting speed in continuous shooting mode and an inability to capture the image intended by the user.

[0098] 6 and 7 , the control unit 5 in the imaging device 1 may determine that a predetermined condition is met and switch to the non-binning mode M2 ​​when at least the imaging frame rate is lower than a threshold. If the frame rate is equal to or higher than the threshold, for example, when high frame rate imaging is being performed, appropriate images (videos) may not be obtained in the non-binning mode M2, which has a long imaging interval. Therefore, by determining that a predetermined condition is met when at least the imaging frame rate setting is lower than the threshold, in other words, by executing the binning mode M1 when the frame rate is lower than the threshold, it is possible to prevent the intended imaging from being impossible during high frame rate imaging.

[0099] 6 and 7 , the control unit 5 in the imaging device 1 may determine that a predetermined condition is met and switch to non-binning mode M2 ​​when at least the zoom magnification (non-optical zoom magnification or combined zoom magnification) is equal to or greater than a threshold. The larger the zoom magnification, the smaller the crop range Ac cut out from the partial region 13 (readout range Ar) becomes, and the higher the non-optical zoom magnification becomes. Therefore, by determining that a predetermined condition is met and adopting non-binning mode M2 ​​when the zoom magnification is equal to or greater than a threshold, it is possible to suppress a decrease in resolution after digital zooming when high-magnification zooming is performed.

[0100] As described with reference to Figures 6 and 7, the control unit 5 in the imaging device 1 may determine that a predetermined condition is met and switch to non-binning mode M2 ​​when the average luminance value based on at least the light-receiving signals output from a predetermined region in the pixel array unit 9 is equal to or greater than a threshold. In non-binning mode M2, additive readout is not performed, which tends to result in a dark image. Therefore, by determining that a predetermined condition is met when at least the average luminance value is equal to or greater than the threshold, in other words, by executing binning mode M1 when capturing images in a dark environment where the average luminance value is less than the threshold, it is possible to avoid unduly increasing the degree of amplification due to gain adjustment in low-illuminance situations. This makes it possible to suppress the generation or increase of noise during gain adjustment when capturing images in low-illuminance situations.

[0101] 7 and other drawings, the predetermined area in the imaging device 1 may be an area in which an image of a target subject is captured. The target subject may be, for example, a subject to be focused on, a subject specified by a user, or a person. By executing the non-binning mode M2 ​​only when the average luminance value of the area in which the specific subject is captured is equal to or greater than a threshold, it is possible to reduce noise in the target subject.

[0102] 6 and 7 , the control unit 5 in the imaging device 1 may determine that a predetermined condition is met and switch to the non-binning mode M2 ​​when at least the subject to be imaged is not a moving subject. When imaging a moving subject, it is conceivable to reduce focal plane distortion by increasing the readout speed. Therefore, by determining that a predetermined condition is met when at least the subject to be imaged is not a moving subject, in other words, by executing the binning mode M1, which requires a relatively short time for readout processing, when imaging a moving subject, it is possible to suppress focal plane distortion when imaging a moving subject.

[0103] 7 and other figures, in the imaging device 1, it may be determined whether or not the subject to be imaged is a moving subject based on the magnitude of the motion vector between frames. This makes it possible to appropriately determine whether or not the subject to be imaged is a moving subject, and effectively suppress the occurrence and degree of focal plane distortion.

[0104] As described above, the control unit 5 in the imaging device 1 may determine that a predetermined condition is met when the dynamic range of at least the partial region 13 is narrower than a threshold value and switch to the non-binning mode M2. When the dynamic range is wide, HDR imaging may be performed. In HDR imaging, multiple images with different exposures are captured in a short period of time and then combined, enabling a richly expressed image to be obtained. It is desirable to capture multiple images in HDR imaging as quickly as possible. That is, by determining that a predetermined condition is met when the dynamic range of at least the partial region 13 (crop area Ac) is narrower than a threshold value, in other words, by executing the binning mode M1, which requires a relatively short readout process time, when performing HDR imaging, a clear composite image can be obtained during HDR imaging.

[0105] As described with reference to FIG. 10 and other figures, the control unit 5 in the imaging device 1 may determine whether a predetermined condition is met based on the optical zoom magnification of the optical system 3 provided upstream of the pixel array unit 9. For example, if the optical zoom magnification and the non-optical zoom magnification are combined to determine the combined zoom magnification, the comparison in step S204 in FIG. 7 may compare the combined zoom magnification with a threshold value. In this case, the threshold value compared with the combined zoom magnification may be determined based on the optical zoom magnification. For example, if the maximum optical zoom magnification is 3.0, the threshold value may be set to 6.0. As a result, if a further zoom-in operation is performed with the optical zoom magnification at its maximum value of 3.0, the combined zoom magnification becomes 6.0 when the non-optical zoom magnification reaches 2.0, and the binning mode M1 is switched to the non-binning mode M2. Therefore, the perceived resolution can be temporarily increased during the process of a decrease in perceived resolution due to a zoom-in operation.

[0106] 10 and other figures, the control unit 5 in the imaging device 1 may determine whether a predetermined condition is met when at least the specified zoom magnification (total zoom magnification) is equal to or greater than the maximum optical zoom magnification. This makes it possible to avoid the process of confirming whether the condition is met in step S102 when the total zoom magnification specified by the user can be accommodated by optical zoom, thereby reducing the processing load on the imaging device 1.

[0107] In the information processing method of the present technology, an information processing device performs a process of selecting non-binning mode M2 ​​when a predetermined condition is met, out of a binning mode M1 in which a light reception signal corresponding to the amount of received light is output for each pixel block 11 arranged in a partial region 13 in a pixel array section 9 out of pixel blocks 11 consisting of a plurality of pixels 8 in response to a zooming operation, and a non-binning mode M2 ​​in which a light reception signal corresponding to the amount of received light is output from each of a plurality of pixels 8 arranged in a partial region 13 in response to a zooming operation.

[0108] In addition, the program of the present technology causes an information processing device to execute a process of selecting, when a predetermined condition is met, a non-binning mode M2 ​​out of a binning mode M1 in which a light reception signal corresponding to the amount of light received is output for each pixel block 11 arranged in a partial region 13 in the pixel array section 9 out of a pixel block 11 consisting of a plurality of pixels 8 in response to a zooming operation, and a non-binning mode M2 ​​in which a light reception signal corresponding to the amount of light received is output from each of a plurality of pixels 8 arranged in a partial region 13 in response to a zooming operation.

[0109] The above-mentioned effects can also be obtained by such an information processing method and program.

[0110] The effects described in this specification are merely examples and are not limiting, and other effects may also be obtained. Furthermore, the examples described above can be combined in any way as long as the combination is not impossible.

[0111] 6. The Present Technology The present technology may also have the following configuration. (1) An imaging element including: a pixel array unit in which pixels are arranged in row and column directions, each pixel outputting a light reception signal according to an amount of received light; and a color filter provided for each pixel block consisting of a plurality of the pixels, wherein the pixel array unit outputs the light reception signals from the pixels arranged in a partial region of the pixel array unit in response to a zooming operation, and wherein output modes of the light reception signals in response to the zooming operation include a binning mode in which one light reception signal is output for each pixel block arranged in the partial region, and a non-binning mode in which one light reception signal is output for each pixel arranged in the partial region, and the non-binning mode is a mode selected when a predetermined condition is met. (2) The imaging element according to (1), wherein the partial region is a central region of the pixel array unit. (3) The imaging element according to any of (1) to (2), wherein the partial region in the non-binning mode is narrower than the partial region in the binning mode. (4) An imaging device comprising: a control unit that performs zooming processing in accordance with a detection result of a zooming operation; a pixel array unit in which pixels that output light reception signals in accordance with an amount of received light are arranged in row and column directions and cause the pixels arranged in a partial region to output the light reception signals in accordance with the zooming operation; and a color filter provided for each pixel block consisting of a plurality of the pixels, wherein the control unit switches between a binning mode in which one light reception signal is output for each pixel block arranged in the partial region and a non-binning mode in which one light reception signal is output for each pixel arranged in the partial region, depending on whether a predetermined condition is met. (5) The image sensor described in (4) above, in which the control unit determines that the predetermined condition is met and switches to the non-binning mode when imaging is not being performed in at least a continuous shooting mode. (6) The image sensor described in any of (4) to (5) above, in which the control unit determines that the predetermined condition is met and switches to the non-binning mode when at least an imaging frame rate is lower than a threshold.(7) The image sensor according to any one of (4) to (6), wherein the control unit determines that the predetermined condition is met and switches to the non-binning mode when at least the zoom magnification is equal to or greater than a threshold. (8) The image sensor according to any one of (4) to (7), wherein the control unit determines that the predetermined condition is met and switches to the non-binning mode when at least an average luminance value based on the light reception signals output from a predetermined region in the pixel array unit is equal to or greater than a threshold. (9) The image sensor according to (8), wherein the predetermined region is an area in which an image of a subject to be imaged is captured. (10) The image sensor according to any one of (4) to (9), wherein the control unit determines that the predetermined condition is met and switches to the non-binning mode when at least the subject to be imaged is not a moving subject. (11) The image sensor according to (10), wherein it is determined whether the subject to be imaged is a moving subject based on a magnitude of a motion vector between frames. (12) The image sensor according to any one of (4) to (11), wherein the control unit determines that the predetermined condition is met and switches to the non-binning mode when a dynamic range for at least the partial region is narrower than a threshold. (13) The image sensor according to any one of (4) to (12), wherein the control unit determines whether the predetermined condition is met according to an optical zoom magnification in an optical system provided upstream of the pixel array unit. (14) The image sensor according to (13), wherein the control unit determines whether the predetermined condition is met at least when a specified zoom magnification is equal to or greater than the maximum optical zoom magnification. (15) An information processing method, wherein an information processing device performs processing to select the non-binning mode when a predetermined condition is met, from among a binning mode in which a light reception signal according to an amount of received light is output for each pixel block, which is composed of a plurality of pixels and is arranged in a partial region of a pixel array unit, in response to a zooming operation, and a light reception signal according to an amount of received light is output from each of the plurality of pixels arranged in the partial region in response to the zooming operation.(16) A program that causes an information processing device to execute a process of selecting a non-binning mode when a predetermined condition is met, from a binning mode in which a light reception signal corresponding to the amount of light received is output for each pixel block, consisting of a plurality of pixels, arranged in a partial area of ​​a pixel array unit in response to a zooming operation, out of the pixel blocks, and a non-binning mode in which a light reception signal corresponding to the amount of light received is output from each of the plurality of pixels arranged in the partial area in response to the zooming operation.

[0112] DESCRIPTION OF SYMBOLS 1 Imaging device 2 Imaging element 3 Optical system 5 Control unit 8 Pixel 8B pixel 8G pixel 8R pixel 9 Pixel array unit 10 Color filter 10B color filter 10G color filter 10R color filter 11 Pixel block 13 Partial region M1 Binning mode M2 ​​Non-binning mode

Claims

1. An imaging element comprising: a pixel array section in which pixels are arranged in row and column directions, each pixel outputting a light reception signal according to an amount of received light; and a color filter provided for each pixel block consisting of a plurality of the pixels, wherein the pixel array section outputs the light reception signals from the pixels arranged in a partial area of ​​the pixel array section in response to a zooming operation, and wherein output modes for the light reception signals in response to the zooming operation include a binning mode in which one light reception signal is output for each pixel block arranged in the partial area, and a non-binning mode in which one light reception signal is output for each pixel arranged in the partial area, and wherein the non-binning mode is a mode selected when a predetermined condition is met.

2. The imaging element according to claim 1, wherein the partial region is a central region of the pixel array portion.

3. The image sensor according to claim 1, wherein the partial area in the non-binning mode is narrower than the partial area in the binning mode.

4. An imaging device comprising: a control unit that performs zooming processing in response to detection results of a zooming operation; a pixel array unit in which pixels that output light reception signals in response to the amount of received light are arranged in row and column directions and cause the pixels arranged in a partial region to output the light reception signals in response to the zooming operation; and a color filter provided for each pixel block consisting of a plurality of the pixels, wherein the control unit switches between a binning mode in which one light reception signal is output for each pixel block arranged in the partial region and a non-binning mode in which one light reception signal is output for each pixel arranged in the partial region in response to whether a predetermined condition is satisfied.

5. The imaging device according to claim 4, wherein the control unit determines that the predetermined condition is met and switches to the non-binning mode when imaging is not being performed at least in the continuous shooting mode.

6. The imaging device according to claim 4, wherein the control unit determines that the predetermined condition is met and switches to the non-binning mode when at least the shooting frame rate is lower than a threshold value.

7. The imaging device according to claim 4, wherein the control unit determines that the predetermined condition is met and switches to the non-binning mode when at least the zoom magnification is equal to or greater than a threshold value.

8. The imaging device according to claim 4, wherein the control unit determines that the specified condition is met and switches to the non-binning mode when an average luminance value based on the light receiving signal output from at least a specified region in the pixel array unit is equal to or greater than a threshold value.

9. The imaging device according to claim 8, wherein the predetermined area is an area in which an image of a subject to be imaged is captured.

10. The imaging device according to claim 4, wherein the control unit determines that the predetermined condition is met and switches to the non-binning mode when at least the subject being imaged is not a moving subject.

11. The imaging device according to claim 10, wherein whether or not the subject being imaged is a moving subject is determined based on the magnitude of a motion vector between frames.

12. The imaging device according to claim 4, wherein the control unit determines that the predetermined condition is met and switches to the non-binning mode when the dynamic range for at least the partial region is narrower than a threshold value.

13. The imaging device according to claim 4, wherein the control unit determines whether or not the predetermined condition is met depending on an optical zoom magnification in an optical system provided in front of the pixel array unit.

14. The imaging device according to claim 13, wherein the control unit determines whether or not the predetermined condition is met at least when the specified zoom magnification is equal to or greater than the maximum optical zoom magnification.

15. An information processing method in which an information processing device performs a process of selecting a non-binning mode when a predetermined condition is met, from a binning mode in which a light reception signal corresponding to the amount of light received is output for each pixel block consisting of a plurality of pixels arranged in a partial area of ​​a pixel array section in response to a zooming operation, and a non-binning mode in which a light reception signal corresponding to the amount of light received is output from each of a plurality of pixels arranged in the partial area in response to the zooming operation.

16. A program for causing an information processing device to execute a process for selecting a non-binning mode when a predetermined condition is met, out of a binning mode in which a light receiving signal corresponding to the amount of light received is output for each pixel block consisting of a plurality of pixels arranged in a partial area of ​​a pixel array unit in response to a zooming operation, and a non-binning mode in which a light receiving signal corresponding to the amount of light received is output from each of a plurality of pixels arranged in the partial area in response to the zooming operation.

Images