Image processing device, driving method, and recording medium

Abstract

The present technology relates to an image processing device, a driving method, and a recording medium with which it is possible to suppress any increase in power consumption while ensuring detection accuracy. The image processing device comprises: a plurality of image sensors; and a control unit that, on the basis of at least one of imaging conditions and information relating to a surrounding environment, determines a driving mode for the image sensor from among a plurality of driving modes, including a first mode for outputting an image and a second mode for performing only a process relating to detection for each image sensor. When driving in the second mode is performed, the image sensor stops the operation of an image pipeline composed of a processing block that performs a process for generating an image to be outputted. The present technology can be applied to mobile devices.

Description

Image processing device, driving method, and recording medium

[0001] The present technology relates to an image processing device, a driving method, and a recording medium, and more particularly to an image processing device, a driving method, and a recording medium that are capable of suppressing an increase in power consumption while ensuring detection accuracy.

[0002] Conventionally, devices with multiple cameras with different focal lengths are known. On such devices, by running other cameras (sub-cameras) in the background of the main camera, it is possible to seamlessly switch between cameras according to the subject, calibrate between multiple cameras, and increase the flexibility of work creation using parallax information.

[0003] Specifically, for example, if a sub-camera such as a telephoto camera is started in the background of a main camera such as a wide-angle camera, the processor can always grasp the phase difference information of the sub-camera. Therefore, the processor controls the AF (Auto Focus) driver based on the phase difference information of the sub-camera, making it possible to always perform AF on the sub-camera.

[0004] This allows the main camera to be seamlessly switched immediately after a camera switching command such as a change in shooting magnification is issued.

[0005] On the other hand, if multiple cameras are kept running at all times, the power required to operate the multiple cameras, i.e., the main camera and sub-camera, and the power required to process the information sent from the multiple cameras in the processor will increase.

[0006] Therefore, a technique has been proposed in which the increase in power consumption is suppressed by, for example, lowering the frame rate of data output from the sub-camera (see, for example, Patent Document 1).

[0007] US Patent Application Publication No. 2019 / 0320102

[0008] However, in the technology described in Patent Document 1, the number of samples of detection information in the time direction decreases by the amount of the frame rate of the sub-camera being lowered, which results in a decrease in detection accuracy of the sub-camera.

[0009] The present technology has been developed in light of these circumstances, and makes it possible to suppress an increase in power consumption while ensuring detection accuracy.

[0010] An image processing device according to one aspect of the present technology includes a plurality of image sensors and a control unit that determines, for each image sensor, a drive mode for the image sensor from among a plurality of drive modes including a first mode for outputting an image and a second mode for performing only detection-related processing, based on at least one of shooting conditions and information regarding the surrounding environment; and when the image sensor is driven in the second mode, it stops operation of an image pipeline consisting of a processing block that performs processing to generate the image to be output.

[0011] A driving method or recording medium according to one aspect of the present technology is a recording medium having a driving method or a program recorded thereon, in which an image processing device having a plurality of image sensors determines, for each image sensor, a driving mode for the image sensor from among a plurality of driving modes including a first mode for outputting an image and a second mode for performing only detection-related processing, based on at least one of shooting conditions and information related to the surrounding environment, and when the image sensor is driven in the second mode, the image sensor stops operation of an image pipeline consisting of processing blocks that perform processing to generate the image to be output.

[0012] In one aspect of the present technology, in an image processing device having a plurality of image sensors, a drive mode for each image sensor is determined from a plurality of drive modes including a first mode for outputting an image and a second mode for performing only detection-related processing based on at least one of shooting conditions and information on the surrounding environment, and when the image sensor is driven in the second mode, operation of an image pipeline consisting of a processing block that performs processing to generate the image to be output is stopped.

[0013] 1 is a diagram illustrating an example of the configuration of an image processing device. FIG. 1 is a diagram illustrating an example of the configuration of an image processing unit. FIG. 2 is a diagram illustrating an example of a pixel array. FIG. 3 is a diagram illustrating pausing an AD conversion unit. FIG. 4 is a diagram illustrating an example of a pixel array. FIG. 5 is a diagram illustrating an example of a phase difference pixel. FIG. 6 is a diagram illustrating an example of a phase difference pixel. FIG. 7 is a diagram illustrating an example of a phase difference pixel. FIG. 8 is a flowchart illustrating post-startup processing. FIG. 9 is a flowchart illustrating shooting magnification change processing. FIG. 10 is a flowchart illustrating angle of view change processing. FIG. 11 is a flowchart illustrating AF method change processing. FIG. 12 is a diagram illustrating an example of the configuration of an image processing device. FIG. 13 is a diagram illustrating pausing an AD conversion unit. FIG. 14 is a flowchart illustrating thinning rate change processing. FIG. 14 is a diagram illustrating an example of the configuration of an image processing device. FIG. 15 is a diagram illustrating an example of the configuration of an image processing device. FIG. 16 is a diagram illustrating an example of the configuration of an image processing device. FIG. 17 is a diagram illustrating monochrome readout and pausing an AD conversion unit. FIG. 18 is a diagram illustrating monochrome readout and pausing an AD conversion unit. FIG. 19 is a flowchart illustrating post-startup processing. FIG. 19 is a diagram illustrating an example of the configuration of an image processing device. FIG. 19 is a diagram illustrating an example of the configuration of an image processing device. FIG. 19 is a diagram illustrating an example of the configuration of an image processing device. It is a diagram for explaining the target range of AD conversion in the sub-camera.It is a diagram showing an example of the configuration of an image processing device.It is a diagram showing an example of driving the image processing device.It is a diagram showing an example of the configuration of a computer.

[0014] Hereinafter, embodiments to which the present technology is applied will be described with reference to the drawings.

[0015] First Embodiment Example of Configuration of Image Processing Device The present technology relates to a system having a block that can change the drive of an image sensor and output information to a processor based on a sensor mode signal generated according to shooting conditions and a shooting environment, and a block that realizes lens drive control based on a phase difference detection result.

[0016] FIG. 1 is a diagram showing an example of the configuration of an embodiment of an image processing apparatus to which the present technology is applied.

[0017] 1 is composed of a device such as a photographing device having a photographing function or a mobile device. Specifically, the image processing device 11 is, for example, a smartphone, a tablet, or an HMD (Head Mounted Display). Note that, although the image processing device 11 is described here as a single device, the image processing device 11 may also be a single system composed of multiple devices.

[0018] The image processing device 11 includes a camera 21-1, a camera 21-2, an AP (Application Processor) 22, and a display .

[0019] The cameras 21-1 and 21-2 have different focal lengths, ie, different suitable magnifications.

[0020] For example, in this example, camera 21-1 is a wide-angle camera with a shorter (smaller) focal length than camera 21-2, and camera 21-2 is a telephoto camera with a longer (larger) focal length than camera 21-1. Hereinafter, camera 21-1 will also be referred to as wide-angle camera 21-1, and camera 21-2 will also be referred to as telephoto camera 21-2.

[0021] Camera 21-1 and camera 21-2 are driven under the control of AP 22 to capture images of surrounding subjects. Camera 21-1 and camera 21-2 also output images (image data) obtained by capturing images and detection information indicating detection results to AP 22. Note that hereinafter, when there is no need to particularly distinguish between camera 21-1 and camera 21-2, they will also be simply referred to as camera 21.

[0022] For example, the image processing device 11 can be used as a compound eye by simultaneously operating the camera 21-1 and the camera 21-2. When using a compound eye, the AP 22 supplies each camera 21 with a sensor mode signal indicating the drive mode of the camera 21.

[0023] For example, the driving modes include a Viewing mode in which an image is output (captured) to be presented to the user or recorded, and a Phase Difference Detection mode in which only processing related to phase difference detection is performed without outputting (capturing) an image to be presented to the user or recorded.

[0024] Basically, the main camera is driven in a viewing mode, and the sub-camera is driven in a phase difference detection mode. The main camera is a camera that operates in the foreground as an object of user operation, and the sub-camera is a camera that operates in the background and is not an object of user operation. Note that when there are three or more cameras 21, for example, one camera 21 is designated as the main camera, and the remaining cameras 21 are all designated as sub-cameras.

[0025] 1, the cameras 21-1 and 21-2 basically have the same configuration except for the difference in focal length. Note that, although an example in which two cameras 21 are provided in the image processing device 11 will be described here, the present invention is not limited to this, and three or more cameras 21 may be provided.

[0026] The camera 21-1 has an AF driver 31-1 and an image sensor 32-1, and the image sensor 32-1 has a pixel section 41-1, a control section 42-1, and an image processing section 43-1.

[0027] The control unit 42-1 has an MCCU (Multi Camera Control Unit) 51-1, and the image processing unit 43-1 has an image pipeline 52-1, a detection pipeline 53-1, and a detection value processing unit 54-1.

[0028] Similarly, the camera 21-2 has an AF driver 31-2 and an image sensor 32-2, and the image sensor 32-2 has a pixel unit 41-2, a control unit 42-2, and an image processing unit 43-2. ​​The control unit 42-2 has an MCCU 51-2, and the image processing unit 43-2 has an image pipeline 52-2, a detection pipeline 53-2, and a detection value processing unit 54-2.

[0029] Hereinafter, when there is no need to particularly distinguish between the AF drivers 31-1 and 31-2, and the image sensors 32-1 and 32-2, they will also be simply referred to as the AF driver 31 and the image sensor 32.

[0030] Similarly, when there is no need to particularly distinguish between pixel unit 41-1 and pixel unit 41-2, control unit 42-1 and control unit 42-2, and image processing unit 43-1 and image processing unit 43-2, they will also be simply referred to as pixel unit 41, control unit 42, and image processing unit 43.

[0031] Furthermore, when there is no need to particularly distinguish between MCCU 51-1 and MCCU 51-2, image pipelines 52-1 and 52-2, detection pipelines 53-1 and 53-2, and detection value processing units 54-1 and 54-2, they are also simply referred to as MCCU 51, image pipeline 52, detection pipeline 53, and detection value processing unit 54.

[0032] The AF driver 31 realizes AF control (autofocus control) by moving the position of a lens (not shown) in the camera 21 under the control of the AP 22 .

[0033] The image sensor 32 is formed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and captures an image of a subject around the image processing device 11 .

[0034] The image sensor 32 is provided with a control unit 42 that controls the overall operation of the image sensor 32, a pixel unit 41 that takes images, and an image processing unit 43 that performs image processing on the images (image data) obtained by taking images.

[0035] The control unit 42 has an MCCU 51 , and the MCCU 51 controls the driving (operation) of the pixel unit 41 and the image processing unit 43 in response to a sensor mode signal supplied from the AP 22 .

[0036] For example, the MCCU 51 controls the driving of the pixel unit 41 by supplying the pixel unit 41 with a control signal Low_Power_en that indicates whether to drive the pixel unit 41 at normal power consumption or at low power consumption that is lower than the normal power consumption.

[0037] Furthermore, for example, the MCCU 51 controls the operation of the image processing unit 43 by supplying the image processing unit 43 with a control signal ipipe_en indicating whether or not to operate the image pipeline 52 .

[0038] The pixel unit 41 has a plurality of pixels arranged in a matrix and an AD (Analog to Digital) conversion unit that converts signals output from the pixels to generate pixel data, which is data (signals) of pixels of an image. The pixel unit 41 operates in response to a control signal Low_Power_en from the MCCU 51, generates image data of an image made up of pixel data of each of a plurality of pixels, and supplies (outputs) it to the image processing unit 43. For example, in the pixel unit 41, driving related to reading out signals from the pixels (readout driving) is changed in response to the control signal Low_Power_en.

[0039] The image processing unit 43 operates in response to a control signal ipipe_en from the MCCU 51, and appropriately changes the driving of the pipeline for processing pixel data. That is, based on the image data supplied from the pixel unit 41, the image processing unit 43 generates at least one of image data of an image to be presented (displayed) to a user or recorded (hereinafter also referred to as a display image or display image data) and detection information for controlling the camera 21, and outputs (supplies) it to the AP 22 in the subsequent stage.

[0040] Specifically, the image pipeline 52 performs various image processes on the image data (pixel data) supplied from the pixel unit 41 to generate image data for display, and supplies the image data to the AP 22 .

[0041] Furthermore, the detection pipeline 53 performs predetermined image processing on the image data (pixel data) supplied from the pixel unit 41, and supplies the resulting image data to the detection value processing unit 54. The detection value processing unit 54 generates detection information indicating the detection result based on the image data (pixel data) supplied from the detection pipeline 53, and supplies the detection information to the AP 22. For example, when the drive mode is the phase difference detection mode, phase difference information indicating the phase difference between the left and right is generated as detection information for phase difference detection. This phase difference information is used by the AP 22 for AF control of the camera 21.

[0042] The pixel unit 41 includes pixels (hereinafter also referred to as normal pixels) that output signals used to generate images for display, and pixels (hereinafter also referred to as phase difference pixels) that output signals used to generate phase difference information as detection information. Basically, the image pipeline 52 performs image processing on pixel data obtained based on the signals output from the normal pixels, and the detection pipeline 53 and the detection value processing unit 54 perform processing on pixel data obtained based on the signals output from the phase difference pixels.

[0043] In the following, images obtained by photographing on the wide-angle camera 21-1 side, such as display images output from the image pipeline 52-1, will be referred to specifically as wide-angle images, and images obtained by photographing on the telephoto camera 21-2 side, such as display images output from the image pipeline 52-2, will be referred to specifically as telephoto images.

[0044] The AP 22 is made up of a processor and controls the overall operation of the image processing device 11 .

[0045] The AP 22 includes a setting unit 61 , a multi-camera mode-generation unit (MCMU) 62 , a detection information processing unit 63 , and an image processing unit 64 .

[0046] The setting unit 61 sets the image shooting magnification (user-set magnification) and the like in response to user operations on the image processing device 11, and supplies the setting results to the MCMU 62. In addition, the setting unit 61 also supplies the results of various input operations, such as the user's operation to specify a subject of interest, to the MCMU 62 as appropriate.

[0047] The MCMU 62 determines the drive mode for each camera 21 (image sensor 32) based on information supplied from the setting unit 61, the detection information processing unit 63, and the image processing unit 64, generates a sensor mode signal indicating the result of the determination, and supplies (outputs) it to the MCCU 51 of each camera 21. In other words, by determining the drive mode and supplying the sensor mode signal to the camera 21, the MCMU 62 functions as a control unit that drives each camera 21 in the drive mode indicated by the sensor mode signal.

[0048] The detection information processing unit 63 performs AF control, that is, lens position control of the camera 21 , based on the detection information supplied from the detection value processing unit 54 of the camera 21 .

[0049] For example, the detection information processing unit 63 calculates the amount of movement of the lens of the camera 21 based on the detection information, and supplies lens control information indicating the calculation result to the AF driver 31, thereby controlling the movement of the lens position by the AF driver 31. In addition, for example, the detection information processing unit 63 supplies information related to AF control to the MCMU 62.

[0050] The image processing unit 64 performs various image processing such as demosaic processing, color space conversion processing, and WB (White Balance) adjustment processing on image data (display image data) supplied from the image pipeline 52 of the camera 21, and supplies the resulting final display image to be displayed on the display 23. The image processing unit 64 also supplies information related to the display image to the MCMU 62.

[0051] The display 23 displays various images supplied from the image processing unit 64. The display 23 displays, for example, a final display image obtained by shooting with the camera 21 designated as the main camera, and presents it to the user.

[0052] <Configuration Example of Image Processing Unit> More specifically, the image processing unit 43 provided in each camera 21 is configured as shown in Fig. 2. Note that in Fig. 2, parts corresponding to those in Fig. 1 are given the same reference numerals, and their explanation will be omitted as appropriate.

[0053] The image processing unit 43 shown in FIG. 2 includes a crop processing unit 91, a defect correction unit 92, an image pipeline 52, a detection pipeline 53, and a detection value processing unit .

[0054] The image pipeline 52 is made up of processing blocks that perform processing on a display image, more specifically, an image made up of pixel data of pixels that are not phase difference pixels, output from the pixel unit 41. In other words, the image pipeline 52 is made up of processing blocks that perform processing to generate a display image to be output to the image processing unit 64. Specifically, the image pipeline 52 has a gain correction unit 101, a clamp processing unit 102, and a signal processing unit 103.

[0055] The detection pipeline 53 is made up of processing blocks that perform processing for detection. Specifically, the detection pipeline 53 has a clamp processing unit 111.

[0056] The crop processing unit 91 performs crop processing on the image data (pixel data) supplied from the pixel unit 41 to cut out a part of the image, and supplies the resulting image data to the defect correction unit 92. The defect correction unit 92 performs defect correction processing on the image data supplied from the crop processing unit 91 to correct pixel data of defective pixels, and supplies the resulting image data to the gain correction unit 101 and the clamp processing unit 111.

[0057] The gain correction unit 101 performs gain correction on the image data supplied from the defect correction unit 92, and supplies the resulting image data to a clamp processing unit 102. The clamp processing unit 102 performs clamp processing, which is processing to correct the black level, on the image data supplied from the gain correction unit 101, and supplies the resulting image data to a signal processing unit 103.

[0058] The signal processing unit 103 performs various signal processing such as re-mosaic processing and NR (Noise Reduction) processing on the image data supplied from the clamp processing unit 102, and supplies the resulting image data (image data for display) to the image processing unit 64.

[0059] The clamp processing unit 111 performs clamp processing, which is processing for correcting the black level, on the image data supplied from the defect correction unit 92, and supplies the image data obtained as a result to the detection value processing unit .

[0060] The image processing unit 43 is provided with an image pipeline 52 as a pipeline (data path) for generating images to be displayed, and a detection pipeline 53 as a pipeline for generating detection information.

[0061] Comparing the image pipeline 52 and the detection pipeline 53, the image pipeline 52 performs a large amount of processing to generate image data for display, whereas the detection pipeline 53 only performs clamping processing. In other words, the detection pipeline 53 performs only the minimum amount of processing necessary to generate detection information such as phase difference information and luminance information, which will be described later. Therefore, the detection pipeline 53 achieves lower power consumption and faster operation than the image pipeline 52.

[0062] <Driving Example of Image Processing Device> An example of driving the image processing device 11 will be described with reference to Fig. 3. Note that in Fig. 3 and other figures describing the embodiments, parts corresponding to those in Fig. 1 are denoted by the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0063] For example, assume that the viewing mode is selected as the drive mode for the camera 21-1 and the phase difference detection mode is selected as the drive mode for the camera 21-2 in the MCMU 62. In particular, assume that the wide-angle camera 21-1 is the main camera and the telephoto camera 21-2 is the sub-camera.

[0064] In this case, the MCMU 62 supplies a sensor mode signal indicating the viewing mode to the MCCU 51-1, and supplies a sensor mode signal indicating the phase difference detection mode to the MCCU 51-2.

[0065] The MCCU 51-1 supplies a control signal Low_Power_en=Low to the pixel unit 41-1 and supplies a control signal ipipe_en=High to the image processing unit 43-1 in accordance with the supplied sensor mode signal, that is, the Viewing mode.

[0066] The control signal Low_Power_en=Low indicates driving with normal power consumption, that is, driving for reading out all pixels, and the pixel section 41-1 operates so that photoelectric conversion and signal reading are performed in all pixels.

[0067] For example, if the pixel section 41-1 has 3024 vertical pixels by 4032 horizontal pixels, image data of 3024 vertical pixels by 4032 horizontal pixels is output from the pixel section 41-1 to the image processing section 43-1.

[0068] In addition, the control signal ipipe_en = High indicates that the image pipeline 52 is to be operated, and the image processing unit 43-1 operates (drives) the image pipeline 52-1 to generate an image for display and also operates the detection pipeline 53-1.

[0069] As a result, the wide-angle camera 21-1, which is the main camera, outputs a 3024 × 4032 wide-angle image as an image to be displayed and phase difference information as detection information at a predetermined frame rate, such as 30 fps, to the AP 22. The wide-angle image captured by the main camera is displayed on the display 23 after being subjected to appropriate image processing.

[0070] In contrast, in the sub-camera camera 21-2, the MCCU 51-2 supplies a control signal Low_Power_en=High to the pixel unit 41-2 and a control signal ipipe_en=Low to the image processing unit 43-2 in accordance with the supplied sensor mode signal, i.e., the phase difference detection mode.

[0071] The control signal Low_Power_en=High indicates driving at low power consumption, and the pixel unit 41-2 operates so that only the outputs of some of the pixels in all pixels are AD converted. Specifically, the pixel unit 41-2 controls the operation of the AD conversion unit so that only the signals output from the phase difference pixels among the pixels provided in the pixel unit 41-2 are AD converted. In particular, the frame rate of the image captured by the pixel unit 41-2 is set to be the same as the frame rate of the pixel unit 41-1.

[0072] For example, when an AD conversion unit is provided for each pixel column consisting of pixels arranged in the column direction, the AD conversion unit connected to the pixel column having phase difference pixels performs AD conversion, and the AD conversion unit connected to the pixel column without phase difference pixels is in a stopped (paused) state. Therefore, image data consisting of pixel data of the phase difference pixels is output from the pixel unit 41-2.

[0073] For example, if the pixel unit 41-2 has 3024×4032 pixels, of which 378×504 pixels are phase difference pixels, image data of 378×504 pixels will be output from the pixel unit 41-2 to the image processing unit 43-2.

[0074] In a sub-camera driven in the phase difference detection mode, it is not necessary to output a display image, and only detection information is output, so only the output of the phase difference pixels necessary for generating the detection information needs to be AD converted, and it is possible to stop the operation of some of the AD conversion units. In this way, it is possible to reduce the power consumption of the image sensor 32-2.

[0075] Furthermore, the control signal ipipe_en=Low indicates that the operation of the image pipeline 52 is to be stopped, and the image processing unit 43-2 stops the operation of the image pipeline 52-2 and operates only the detection pipeline 53-2. For example, the image processing unit 43-2 stops the operation of the image pipeline 52-2 by stopping the clock (clock signal) supplied to the image pipeline 52-2, thereby reducing power consumption accordingly.

[0076] As a result, no display image is output from the telephoto camera 21-2, which is the sub-camera, and only phase difference information as detection information is output to the AP 22 at a predetermined frame rate, such as 30 fps. In particular, in this case, the image data and detection information output from the main camera 21-1 and the detection information output from the sub-camera camera 21-2 are data (information) with the same frame rate. In other words, the frame rate of the detection information (the number of samples in the time direction) is not changed, and therefore detection accuracy is maintained.

[0077] Furthermore, in the image processing device 11, a detection value processing unit 54 that generates detection information such as phase difference information is provided inside the image sensor 32. Therefore, compared to when detection information is generated by the detection information processing unit 63 that is located downstream of the image sensor 32, it is possible to compress the information output from the camera 21 to the AP 22. In other words, it is possible to reduce the amount of data input to the AP 22.

[0078] According to the image processing device 11 described above, it is possible to achieve low power consumption while maintaining the number of sampling times (frame rate) of detection information in the time direction in the camera 21 operating in phase difference detection mode at the same level as that in the camera 21 operating in viewing mode.

[0079] Specifically, in the camera 21, the operating power consumption of the image sensor 32 can be reduced by performing AD conversion only on the output of the phase difference pixels or by stopping the operation of the image pipeline 52 in accordance with the information output to the AP 22.

[0080] Furthermore, by limiting the information output from each image sensor 32 to the minimum necessary, the amount of data that needs to be processed on the AP 22 side can be reduced, thereby realizing reduced operating power and faster operation of the AP 22. For example, in the phase difference detection mode, by having the camera 21 output only detection information (phase difference information), it is possible to reduce power consumption in the AP 22 and realize faster operation.

[0081] <Driving Example of Image Processing Device> Here, an example of driving the pixel unit 41 in the phase difference detection mode will be described.

[0082] For example, as shown in Fig. 4, a plurality of pixels including pixel 131 and pixel 132 are arranged in a matrix in pixel section 41. In Fig. 4, one square represents one pixel, and in this example, a total of 64 pixels are arranged in an 8 x 8 array. Note that, in order to make the diagram easier to see, fewer pixels are drawn than in reality.

[0083] These 64 pixels include the normal pixels and phase difference pixels described above. For example, pixel 131 is a normal pixel, and pixel 132 is a phase difference pixel.

[0084] The letters "R," "G," and "B" written in the squares representing the pixels indicate the colors of the color filters provided in those pixels, and here the R pixels, G pixels, and B pixels are arranged in a Bayer array.

[0085] In the figure, one pixel column is made up of eight pixels lined up vertically, and each pixel in the pixel column is connected to the same vertical signal line, and each vertical signal line is connected to an AD conversion unit. Here, AD conversion units 133-1 to 133-8 are provided in the pixel section 41. For example, each pixel in a pixel column including pixel 131 and pixel 132 is connected to AD conversion unit 133-1 via the vertical signal line.

[0086] In the following description, when there is no need to particularly distinguish between the AD conversion units 133-1 to 133-8, they will also be simply referred to as AD conversion units 133.

[0087] Each pixel, such as pixel 131, photoelectrically converts light incident from the outside and holds the resulting signal. When a pixel is selected by a drive unit (not shown) provided in pixel unit 41, the selected pixel supplies the signal generated by photoelectric conversion and held therein to AD conversion unit 133 via a vertical signal line. AD conversion unit 133 converts the signal supplied from the pixel via the vertical signal line into an AD signal and outputs the resulting digital signal as pixel data.

[0088] In the pixel unit 41 described above, when a control signal Low_Power_en=Low is supplied from the MCCU 51, for example, all-pixel readout driving is performed to read signals from all pixels and perform AD conversion. That is, all AD conversion units 133 operate.

[0089] On the other hand, for example, when the control signal Low_Power_en=High is supplied from the MCCU 51 to the pixel unit 41, only the signals output from the phase difference pixels are AD converted as shown in FIG.

[0090] In the example of FIG. 5 , among the pixels of the pixel unit 41, the pixels in the hatched areas are normal pixels, and the pixels in the non-hatched areas, such as pixel 132, are phase difference pixels.

[0091] In this example, only the AD conversion units 133-1 and 133-5, which are the AD conversion units 133 connected to the pixel column having the phase difference pixels, operate, and the remaining AD conversion units 133 do not operate and are in a paused state. That is, the AD conversion units 133 to which the signals output from the phase difference pixels are not input are in a paused state.

[0092] Therefore, compared to the all-pixel readout drive shown in Fig. 4, the number of readouts, i.e., the number of pixel data to be read out, is 1 / 16, and the number of AD conversion units 133 to be driven is 1 / 4, so that the power consumption for drive can be reduced. In other words, low power consumption can be achieved.

[0093] The pixel arrangement of the pixel section 41 is not limited to the Bayer arrangement shown in Fig. 4, and may be any other arrangement. Another example of the pixel arrangement is shown in Fig. 6.

[0094] In the example of FIG. 6, the pixels are arranged in an array called a quad array (Quad Bayer Coding), in which a plurality of pixel groups including the pixel group 161 are arranged in a Bayer array.

[0095] For example, each pixel group, such as pixel group 161, is made up of four pixels adjacent to each other, and the four pixels belonging to the pixel group are provided with color filters of the same color.

[0096] Each pixel group arranged in the column direction is connected to an AD conversion unit via one vertical signal line. In this example, eight AD conversion units, including an AD conversion unit 162 and an AD conversion unit 163, are provided in the pixel unit 41. Furthermore, a switch is provided between adjacent vertical signal lines, such as a switch 164 provided between the vertical signal line connected to the AD conversion unit 162 and the vertical signal line connected to the AD conversion unit 163.

[0097] Even in such a quad arrangement, normal pixels and phase difference pixels are provided in the pixel section 41, and are driven in accordance with the control signal Low_Power_en.

[0098] Incidentally, the phase difference pixels provided in the pixel section 41 may be any type of phase difference pixels.

[0099] For example, as shown in FIG. 7 , a right light-shielded pixel 191 in which the right half of the light receiving area of ​​the pixel is light-shielded, and a left light-shielded pixel 192 in which the left half of the light receiving area of ​​the pixel is light-shielded may be provided as phase difference pixels.

[0100] In this case, in the pixel unit 41, one on-chip lens 193 is provided on the right light-shielding pixel 191, and one on-chip lens 194 is provided on the left light-shielding pixel 192. The right light-shielding pixel 191 and the left light-shielding pixel 192 are used as a pair of phase-difference pixels, and phase-difference information is generated as detection information from signals output from the right light-shielding pixel 191 and the left light-shielding pixel 192.

[0101] The phase difference information may be any information that indicates the phase difference between a pair of phase difference pixels and is obtained from a signal output from the phase difference pixel.

[0102] For example, the pixel data obtained by AD converting the signal output from the right light-shielding pixel and the pixel data obtained by AD converting the signal output from the left light-shielding pixel may be used as phase difference information. In this case, more specifically, the image processing unit 43 performs cropping, defect correction, and clamping on each pixel data, and the data supplied to the detection value processing unit 54 is used as phase difference information as is.

[0103] Alternatively, the phase difference information may be, for example, luminance information obtained by integrating pixel data of a plurality of right light-shielding pixels for each region (block), i.e., a reduced image of an image made up of pixel data of the right light-shielding pixels, and a reduced image obtained by integrating pixel data of a plurality of left light-shielding pixels for each region.Furthermore, difference information between the reduced images of the right light-shielding pixels and the reduced images of the left light-shielding pixels, i.e., the left-right phase difference, may be generated as the phase difference information.

[0104] For example, when the detection value processing unit 54 generates the pixel data itself or a reduced image as phase difference information, the downstream detection information processing unit 63 calculates the difference information of the above-mentioned reduced image as the final phase difference information based on the phase difference information supplied from the detection value processing unit 54.

[0105] Furthermore, for example, as shown in FIG. 8, by providing one on-chip lens 223 for a pixel 221 and a pixel 222 adjacent to each other, these pixels 221 and 222 may function as phase difference pixels.

[0106] In this case, by devising the on-chip lens 223, it is possible to make the amount of light incident on the two phase difference pixels, pixel 221 and pixel 222, different, and thereby it is possible to obtain information on the phase difference between pixel 221 and pixel 222.

[0107] Although an example in which one on-chip lens is provided for two pixels has been described here, one on-chip lens may be provided for 2 pixels × 2 pixels, and these four pixels may function as phase difference pixels. That is, a plurality of pixels provided directly under one on-chip lens may function as phase difference pixels.

[0108] Alternatively, for example, as shown in FIG. 9 , one on-chip lens 252 may be provided for one pixel 251, and the light receiving region of the pixel 251 may be divided into two, so that the pixel 251 functions as a phase difference pixel in the same manner as in the example of FIG. 8 .

[0109] <Explanation of Post-Startup Processing> An example of the operation of the image processing device 11 will be described.

[0110] For example, if the drive mode of both the main camera and the sub-camera is set to the phase difference detection mode immediately after starting up the image processing device 11, it is possible to start up quickly while keeping power consumption low. In particular, in this case, it becomes possible for either camera 21 to immediately focus on the target subject.

[0111] Hereinafter, the post-startup process by the image processing device 11 will be described with reference to the flowchart of Fig. 10. This post-startup process is executed immediately after the image processing device 11 is started up, that is, immediately after an instruction to start up the image processing device 11 is given.

[0112] In step S11, the MCMU 62 determines the imaging magnification of the camera 21 (image) and the main camera.

[0113] For example, immediately after startup, the MCMU 62 may set the shooting magnification to a predetermined shooting magnification such as 1x, or may set the shooting magnification to the shooting magnification indicated by the setting result supplied from the setting unit 61, i.e., the shooting magnification set by the user.

[0114] Furthermore, the MCMU 62 designates either the camera 21-1 or the camera 21-2 as the main camera and the other as the sub-camera, depending on the determined imaging magnification.

[0115] Specifically, for example, if camera 21-1 is capable of taking pictures at a magnification of less than 3x and camera 21-2 is capable of taking pictures at a magnification of 3x or more, then camera 21-1 on the wide-angle side will be selected (determined) as the main camera when the magnification is set to 1x. Note that the following description will be given assuming that camera 21-1 is capable of taking pictures at a magnification of less than 3x and camera 21-2 is capable of taking pictures at a magnification of 3x or more.

[0116] In step S12, the MCMU 62 instructs the main camera and the sub-camera to operate in the phase difference detection mode.

[0117] That is, immediately after the image processing device 11 is started up, the MCMU 62 sets the drive mode of all cameras 21 (image sensors 32) to the phase difference detection mode. The MCMU 62 generates a sensor mode signal indicating the phase difference detection mode and supplies the sensor mode signal to the MCCU 51-1 and MCCU 51-2.

[0118] In step S13, the cameras 21-1 and 21-2 are driven in the phase difference detection mode.

[0119] Specifically, the MCCU 51 of each camera 21 generates a control signal Low_Power_en=High and supplies it to the pixel unit 41 in accordance with the sensor mode signal supplied from the MCMU 62, and also generates a control signal ipipe_en=Low and supplies it to the image processing unit 43.

[0120] The pixel unit 41 appropriately selects each pixel in response to a control signal Low_Power_en=High from the MCCU 51, and causes the pixel to output a signal obtained by photoelectric conversion. Furthermore, the pixel unit 41 drives only those AD conversion units to which signals from phase difference pixels are input, among the AD conversion units, to perform AD conversion, and does not drive those AD conversion units to which signals from phase difference pixels are not input, placing them in a paused state (a state in which driving is stopped). This causes the pixel unit 41 to supply (output) image data made up of pixel data of the phase difference pixels to the image processing unit 43.

[0121] In addition, in response to the control signal ipipe_en = Low from the MCCU 51, the image processing unit 43 stops the operation of the image pipeline 52 by stopping the clock supplied to the image pipeline 52, while allowing each part other than the image pipeline 52 to operate.

[0122] As a result, the image data output from the pixel unit 41 and consisting only of pixel data of the phase difference pixels is subjected to cropping processing by the cropping processing unit 91, and further subjected to defect correction processing by the defect correction unit 92, before being supplied to the clamp processing unit 111. The clamp processing unit 111 performs clamping processing on the image data supplied from the defect correction unit 92, and supplies the image data obtained as a result to the detection value processing unit 54.

[0123] Based on the image data supplied from the clamp processing unit 111, the detection value processing unit 54 generates phase difference information as detection information indicating the detection result in the phase difference detection mode, and supplies it to the detection information processing unit 63 of the AP 22.

[0124] The detection information processing unit 63 generates final phase difference information as appropriate based on the phase difference information as detection information supplied from the detection value processing unit 54, and calculates the lens movement amount of each camera 21 based on the phase difference information. The detection information processing unit 63 supplies lens control information indicating the lens movement amount to the AF driver 31, and causes the AF driver 31 to move the lens position of the camera 21. In other words, the detection information processing unit 63 causes the AF driver 31 to perform a focus operation.

[0125] In this way, immediately after startup, both the main camera and the sub camera are driven in the phase difference detection mode.

[0126] In step S14, the MCMU 62 determines whether the AF accuracy is sufficient, in other words, whether the subject is in focus to a certain extent.

[0127] For example, when operating in the phase difference detection mode immediately after startup, the detection information processing unit 63 determines whether the AF accuracy is sufficient based on the detection information for each frame (time) supplied from the detection value processing unit 54 at a predetermined frame rate, and supplies the determination result to the MCMU 62.

[0128] For example, if the variation in the time direction of the phase difference information as detection information is within a predetermined range (predetermined value), the AF accuracy is determined to be sufficient. This is because when the variation in the phase difference information is large, it is highly likely that the subject is not in focus (is not in focus), and sufficient AF accuracy is not achieved.

[0129] The detection information processing unit 63 may be configured to determine that the AF accuracy is sufficient when the variation in the phase difference information of at least the main camera is equal to or less than a predetermined value, or may be configured to determine that the AF accuracy is sufficient when the variation in the phase difference information of both the main camera and the sub-camera is equal to or less than a predetermined value.

[0130] The MCMU 62 determines that the AF accuracy is sufficient when the detection information processing unit 63 supplies the MCMU 62 with a determination result that sufficient AF accuracy has been obtained. In this case, it can be said that the MCMU 62 determines whether the AF accuracy is sufficient based on the detection information.

[0131] Note that, when determining whether the AF accuracy is sufficient, information indicating the brightness of the surroundings of the image processing device 11, which is supplied to the MCMU 62 from a sensor (not shown) or the like, may be used. For example, if the surroundings of the image processing device 11 are sufficiently bright, there is little variation in the phase difference information, and it is highly likely that sufficient AF accuracy has been obtained, so it is determined that the AF accuracy is sufficient. Alternatively, the phase difference information itself or a value indicating variation in the phase difference information may be supplied from the detection information processing unit 63 to the MCMU 62 as information regarding AF accuracy, and the determination in step S14 may be made.

[0132] If it is determined in step S14 that the AF accuracy is insufficient, the process then returns to step S12, and the above-described process continues. In this case, the lens is moved by the AF control by the detection information processing unit 63, and as the focus is adjusted to the subject, the variation in the phase difference information also becomes smaller.

[0133] On the other hand, if it is determined in step S14 that the AF accuracy is sufficient, the subject is in focus, and the process then proceeds to step S15.

[0134] In step S15, the MCMU 62 sets the driving mode of the main camera to the viewing mode and the driving mode of the sub camera to the phase difference detection mode.

[0135] That is, the MCMU 62 generates a sensor mode signal indicating the viewing mode and supplies the sensor mode signal to the MCCU 51 of the main camera, and also generates a sensor mode signal indicating the phase difference detection mode and supplies the sensor mode signal to the MCCU 51 of the sub camera. In this case, the sensor mode signal indicating the viewing mode may also include information indicating the shooting magnification determined in step S11.

[0136] In step S 16 , the camera 21 - 1 and the camera 21 - 2 perform processing according to the drive mode indicated by the sensor mode signal supplied from the MCMU 62 .

[0137] Specifically, for example, in a sub-camera set to the phase difference detection mode, the same processing as in step S13 described above is performed, and the phase difference information as detection information generated by the detection value processing unit 54 is supplied to the detection information processing unit 63. In this case, since the operation of the image pipeline 52 is stopped in the sub-camera, no display image data is output from the sub-camera.

[0138] On the other hand, the MCCU 51 of the camera 21 designated as the main camera generates a control signal Low_Power_en = Low and supplies it to the pixel unit 41 in accordance with the sensor mode signal supplied from the MCMU 62, and also generates a control signal ipipe_en = High and supplies it to the image processing unit 43.

[0139] The pixel unit 41 drives all pixels in response to a control signal Low_Power_en=Low from the MCCU 51. That is, all pixels are selected in order, as appropriate, for example, in pixel row units, and signals obtained by photoelectric conversion are output from these pixels.

[0140] In addition, all of the AD conversion units in the pixel unit 41 are driven and perform AD conversion in these AD conversion units, so that image data consisting of pixel data for all pixels is supplied (output) from the pixel unit 41 to the image processing unit 43.

[0141] In addition, the image processing unit 43 supplies a clock to the image pipeline 52 in response to a control signal ipipe_en=High from the MCCU 51, thereby operating not only the detection pipeline 53 but also the image pipeline 52.

[0142] In this case, the detection pipeline 53 performs the same processing as in step S13 described above, and the phase difference information as detection information generated by the detection value processing unit 54 is supplied to the detection information processing unit 63.

[0143] In the image pipeline 52, the image data output from the defect correction unit 92 is subjected to gain correction in a gain correction unit 101, clamp processing in a clamp processing unit 102, and re-mosaic processing and NR processing in a signal processing unit 103. The display image data obtained by the signal processing unit 103 is then supplied from the signal processing unit 103 to the image processing unit 64.

[0144] Note that the main camera 21 takes pictures at a specified magnification. In this case, the magnification is adjusted (changed) by, for example, controlling the position of the zoom lens by the MCCU 51 or the AF driver 31, performing digital zoom processing in the downstream image processing unit 64, or by combining control of the position of the zoom lens and digital zoom processing.

[0145] In step S16, the display image data and detection information from the main camera and the detection information from the sub camera are output at the same frame rate.

[0146] In step S17, the detection information processing unit 63 performs processing related to AF based on the phase difference information as detection information supplied from the main camera and sub camera detection value processing units 54. That is, similar to the case in step S13, the detection information processing unit 63 calculates the lens movement amount of each camera 21 based on the phase difference information, and supplies lens control information indicating the calculation result to the AF driver 31 to control the movement of the lens position by the AF driver 31.

[0147] In step S18, the image processing unit 64 performs various image processing such as demosaic processing, color space conversion processing, and WB adjustment processing on the display image data supplied from the main camera signal processing unit 103, and supplies the resulting final display image to be displayed on the display 23. Therefore, for example, when wide-angle camera 21-1 is used as the main camera and shooting is performed at a shooting magnification of 1, the wide-angle image shot at the shooting magnification of "1" is displayed on the display 23 as the display image.

[0148] When the driving mode of the main camera is set to the Viewing mode and the image capturing for display starts, the post-startup process ends, and then the image capturing continues.

[0149] In this way, immediately after startup, the image processing device 11 drives both the main camera and the sub-camera in phase difference detection mode, and when sufficient AF accuracy is achieved, switches the drive mode of the main camera to Viewing mode and begins capturing an image for display.

[0150] This reduces power consumption and enables fast startup. Additionally, in the phase difference detection mode, the frame rate of the detection information is maintained at the same rate as when driving in the Viewing mode, and driving is performed such that only the signals of the phase difference pixels are subjected to AD conversion and the operation of the image pipeline 52 is stopped. This makes it possible to suppress an increase in power consumption while ensuring detection accuracy (without degrading detection accuracy).

[0151] <Explanation of Photographing Magnification Change Processing> After the post-startup processing described with reference to FIG. 10, while a photographing operation is being performed, a change in the photographing magnification may be instructed, for example, by a user operating the image processing device 11.

[0152] The image processing device 11 switches the drive mode of each camera 21 appropriately according to the shooting magnification by performing the shooting magnification change process shown in Fig. 11. Hereinafter, the shooting magnification change process performed by the image processing device 11 will be described with reference to the flowchart in Fig. 11.

[0153] In step S41, the MCMU 62 determines whether the changed imaging magnification is equal to or greater than a predetermined magnification, based on information indicating the setting result of the imaging magnification supplied from the setting unit 61 in response to a user operation or the like.

[0154] For example, assume that camera 21-1 can take pictures at a magnification of less than 3x, and camera 21-2 can take pictures at a magnification of 3x or more. In such a case, the MCMU 62 determines whether the magnification specified (set) by the user, i.e., the changed magnification, is 3x or more.

[0155] If it is determined in step S41 that the magnification is equal to or greater than the predetermined magnification, then in step S42 the MCMU 62 sets the telephoto camera 21-2 as the main camera and the wide-angle camera 21-1 as the sub-camera.

[0156] Therefore, for example, when photography is being performed with the wide-angle camera 21-1 as the main camera at a photography magnification of less than 3x, the main camera is switched from the wide-angle camera 21-1 to the telephoto camera 21-2 in step S42, thereby realizing a change in the photography magnification.

[0157] After the process of step S42 is performed, the process proceeds to step S44.

[0158] On the other hand, if it is determined in step S41 that the magnification is less than the predetermined magnification, in step S43 the MCMU 62 sets the wide-angle camera 21-1 as the main camera and the telephoto camera 21-2 as the sub-camera, and then the processing proceeds to step S44.

[0159] Therefore, for example, when photography is being performed with a magnification of less than 3 times and the wide-angle camera 21-1 as the main camera, the main camera is not switched in step S43.

[0160] Once the processing in step S42 or step S43 is performed and the main camera and sub camera are determined, the processing in step S44 is then performed.

[0161] In step S44, the MCMU 62 sets the drive mode of the main camera to the viewing mode and the drive mode of the sub-camera to the phase difference detection mode, generates a sensor mode signal, and supplies the sensor mode signal to the MCCU 51 of each camera 21. In step S44, the same process as in step S15 in Fig. 10 is performed. In this case, the sensor mode signal indicating the viewing mode may also include information indicating the changed shooting magnification.

[0162] After the processing of step S44 is performed, the processing of step S45 is performed and the photographing magnification change processing ends. However, since the processing of step S45 is the same as the processing of step S16 in FIG. 10, a description thereof will be omitted.

[0163] That is, the main camera is driven in the viewing mode, and the sub-camera is driven in the phase difference detection mode. The detection information processing unit 63 performs the same process as in step S17 of Fig. 10, and the image processing unit 64 performs the same process as in step S18 of Fig. 10, and the operation at the time of shooting continues.

[0164] In this way, the image processing device 11 changes the shooting magnification in response to user operations, etc., and appropriately changes the drive mode of each camera 21 depending on the result of the change. That is, switching between the main camera and the sub camera is performed. In this way, even when the shooting magnification is changed, it is possible to suppress an increase in power consumption while ensuring detection accuracy (without degrading detection accuracy).

[0165] <Explanation of angle of view change processing> When the display image includes a subject of interest (hereinafter also referred to as the subject of interest), the amount of data to be processed can be reduced and power consumption can be reduced by changing the angle of view of the image for phase difference detection using crop processing.

[0166] In the following description, it is assumed that the phase difference detection mode as a driving mode includes a full angle of view phase difference detection mode and a crop angle of view phase difference detection mode.

[0167] The full-angle phase difference detection mode is a mode in which phase difference information is generated for the entire image captured by the pixel unit 41, more specifically, for the entire image made up of pixel data from all phase difference pixels.

[0168] In contrast, the phase difference detection mode with a cropped angle of view is a mode in which phase difference information is generated for the entire image captured by the pixel unit 41, more specifically, for a portion of the entire image consisting of pixel data of all phase difference pixels, i.e., for an area that includes a subject of interest.

[0169] When changing the angle of view in phase difference detection depending on whether or not a target subject is present, the image processing device 11 performs, for example, the angle of view change process shown in Fig. 12. The angle of view change process performed by the image processing device 11 will be described below with reference to the flowchart in Fig. 12. This angle of view change process is performed during the shooting operation.

[0170] In step S71, the MCMU 62 determines whether or not a subject of interest is present in the display image captured by the main camera.

[0171] For example, a case where a human face is the target subject will be described as an example.

[0172] In this case, the image processing unit 64 performs image recognition or the like on the display image supplied from the main camera image pipeline 52 (signal processing unit 103) to detect the area of ​​the human face that is the target subject from the display image, and supplies the detection result to the MCMU 62. At this time, not only the position of the target subject but also the size of the target subject, i.e., the range of the area in the display image where the target subject is located, etc. are notified to the MCMU 62 as appropriate.

[0173] Based on the detection result supplied from the image processing unit 64, the MCMU 62 determines whether or not a target subject is present in the image for display.

[0174] The detection of the area of ​​the target subject may be performed based on an input operation by the user, that is, an operation by the user to specify the target subject.

[0175] If it is determined in step S71 that there is a subject of interest, in step S72 the MCMU 62 sets the drive mode of the main camera to Viewing mode and the drive mode of the sub-camera to Phase Difference Detection Mode with a cropped angle of view, generates a sensor mode signal, and supplies the sensor mode signal to the MCCU 51 of each camera 21.

[0176] In this case, the sensor mode signal indicating the phase difference detection mode of the crop angle of view may include information indicating the area where the target subject is located, etc. Also, the sensor mode signal indicating the Viewing mode may include information indicating the shooting magnification.

[0177] Once the sensor mode signal is generated, the process then proceeds to step S74.

[0178] On the other hand, if it is determined in step S71 that there is no subject of interest, in step S73 the MCMU 62 sets the drive mode of the main camera to Viewing mode and the drive mode of the sub-camera to full-angle phase difference detection mode, generates a sensor mode signal, and supplies the sensor mode signal to the MCCU 51 of each camera 21.

[0179] In this case, the sensor mode signal indicating the Viewing mode may also include information indicating the imaging magnification. After the sensor mode signal is generated, the process then proceeds to step S74.

[0180] After the process of step S72 or step S73 is performed, the process of step S74 is performed.

[0181] In step S74, the camera 21-1 and the camera 21-2 perform processing according to the drive mode indicated by the sensor mode signal supplied from the MCMU 62, and the view angle change processing ends.

[0182] For example, when the telephoto camera 21-2 is set as the main camera, the camera 21-2 operates in a viewing mode at a specified shooting magnification, such as 3x, and outputs detection information (phase difference information) and a telephoto image as a display image. At this time, when a phase difference detection mode with a crop angle of view is specified for the sub-camera, for example, the main camera may also generate phase difference information with a crop angle of view, as in the case of the sub-camera.

[0183] Furthermore, for example, when the wide-angle camera 21-1 is set as a sub-camera and a sensor mode signal indicating a full-angle phase difference detection mode is supplied to the camera 21-1, the camera 21-1 performs processing similar to that in step S13 of FIG. 10, and generates detection information (phase difference information).

[0184] In contrast, for example, when the wide-angle camera 21-1 is used as a sub-camera and a sensor mode signal indicating a phase difference detection mode for a crop angle of view is supplied to the camera 21-1, the camera 21-1 performs processing similar to that in step S13 of FIG. 10 on the area of ​​the subject of interest.

[0185] Specifically, for example, the MCCU 51-1 supplies the control signal ipipe_en=Low and information indicating the area of ​​the target subject to the image processing unit 43-1.

[0186] The image processing unit 43-1 stops the operation of the image pipeline 52-1 and, based on the information indicating the area of ​​the subject of interest, specifies the area to be cropped to the crop processing unit 91. The crop processing unit 91 crops the area including the subject of interest from the image supplied from the pixel unit 41-1 in accordance with the instruction from the image processing unit 43-1, and supplies the resulting image data consisting only of pixel data of phase difference pixels in the area including the subject of interest to the clamp processing unit 111 via the defect correction unit 92. As a result, the detection value processing unit 54 at the subsequent stage generates detection information (phase difference information) for only the area including the subject of interest within the entire image.

[0187] After the processing of step S74 is performed, the detection information processing unit 63 performs processing similar to that in step S17 of FIG. 10, and the image processing unit 64 performs processing similar to that in step S18 of FIG. 10, and the operation at the time of shooting continues.

[0188] In this way, the image processing device 11 changes the crop angle of view, i.e., the area for generating detection information, depending on whether or not the image for display includes a target subject. In this way, processing of pixel data of unnecessary phase difference pixels can be omitted, and power consumption can be reduced.

[0189] <Explanation of AF Method Changing Process> Furthermore, when the AF accuracy is insufficient during image capture, the AF method may be switched to a method with higher accuracy as appropriate by performing the process shown in FIG. 13, for example.

[0190] The AF method change processing by the image processing device 11 will be described below with reference to the flowchart in Fig. 13. Note that the description here is based on the assumption that the AF method change processing is performed while AF is being performed using a phase-difference AF method that uses phase difference information, such as an image plane phase-difference method.

[0191] In step S101, the MCMU 62 determines whether the AF accuracy is sufficient. For example, in step S101, the same process as in step S14 in FIG.

[0192] In addition, when determining whether the AF accuracy is sufficient, in addition to the phase difference information as detection information, information regarding the brightness around the image processing device 11 obtained by a sensor (not shown), an image for display of the main camera obtained by the image processing unit 64, etc. may also be used.

[0193] If it is determined in step S101 that the AF accuracy is sufficient, there is no need to change the AF method, so in step S102 the MCMU 62 sets the drive mode of the main camera to the viewing mode and the drive mode of the sub-camera to the phase difference detection mode.

[0194] The MCMU 62 then generates a sensor mode signal indicating the viewing mode and supplies the sensor mode signal to the MCCU 51 of the main camera, and also generates a sensor mode signal indicating the phase difference detection mode and supplies the sensor mode signal to the MCCU 51 of the sub camera.

[0195] After the process of step S102 is performed, the process proceeds to step S105, where the AF method remains the phase-difference AF method that has been used up until now.

[0196] On the other hand, if it is determined in step S101 that the AF accuracy is insufficient, the MCMU 62 sets the drive mode of the main camera and sub-camera to the viewing mode in step S103 in order to change the AF method. That is, the MCMU 62 sets the drive mode of all cameras 21 (image sensors 32) to the viewing mode.

[0197] The MCMU 62 generates a sensor mode signal indicating the viewing mode, and supplies the sensor mode signal to the MCCU 51 of the main camera and the MCCU 51 of the sub camera.

[0198] In step S104, the MCMU 62 changes the AF method and supplies (notifies) the detection information processing unit 63 with information indicating the changed AF method.

[0199] Here, the AF method is changed from the phase difference AF method that has been used up until now to an AF method that uses an image for display (an AF method based on an image for display), such as a contrast AF method. After the processing of step S104 is performed, the processing proceeds to step S105.

[0200] If sufficient accuracy cannot be achieved with the phase difference AF method, the drive mode of the main camera and sub camera can be set to Viewing mode, which allows each camera 21 to obtain a display image, and then AF can be performed using a more accurate AF method using these display images.

[0201] After the process of step S102 or step S104 has been performed, the cameras 21-1 and 21-2 perform processing according to the drive mode indicated by the sensor mode signal supplied from the MCMU 62 in step S105.

[0202] Furthermore, the detection information processing section 63 performs processing according to the selected AF method, and the image processing section 64 performs processing similar to that in step S18 of FIG.

[0203] For example, if the processing of step S102 has been performed, the same processing as in step S16 of Fig. 10 is performed in step S105. Furthermore, the detection information processing unit 63 controls the movement of the lens position by the AF driver 31 using the phase difference AF method based on the phase difference information as the detection information. That is, the detection information processing unit 63 performs the same processing as in step S17 of Fig. 10.

[0204] On the other hand, if the processing of step S104 is performed, for example, both the main camera and the sub camera are driven in the Viewing mode in step S105. However, in this case, since the phase difference information is not used for AF, the MCCU 51 may stop the clock supplied to the detection pipeline 53, thereby stopping the operation of the detection pipeline 53.

[0205] When the main camera and the sub camera are driven in the Viewing mode, the image processing unit 64 is supplied with display images from the image pipeline 52 of the main camera and the image pipeline 52 of the sub camera.

[0206] Here, the image processing unit 64 performs image processing such as demosaic on each display image, supplies each of the final display images to the detection information processing unit 63, and also supplies the final display image of the main camera to the display 23 for display.

[0207] The detection information processing unit 63 performs AF of the main camera and the sub camera using the AF method specified by the MCMU 62, specifically, for example, the contrast AF method, based on the display images of the main camera and the sub camera supplied from the image processing unit 64. That is, the detection information processing unit 63 calculates the lens movement amount of each camera 21 based on the display images, and supplies lens control information indicating the calculation results to the AF driver 31 to control the movement of the lens position.

[0208] Once the processing according to the selected AF method has been performed, the AF method change processing ends, and then the operation at the time of shooting continues.

[0209] In this way, the image processing device 11 appropriately switches the AF method depending on the result of the determination of whether the AF accuracy is sufficient. In this way, it is possible to focus on the subject more quickly and accurately, thereby obtaining a higher quality image.

[0210] Note that the drive mode of each camera 21 (image sensor 32) may be determined based on not only the shooting conditions such as the shooting magnification and the detection results, i.e., the AF accuracy determination results based on the detection information, but also environmental information related to the environment around the image processing device 11. In other words, the drive mode may be determined based on at least one of the shooting conditions, the detection results, and the environmental information.

[0211] As an example, the drive mode may be determined based on luminance information, which is environmental information, indicating the brightness (luminance) of the surroundings of the image processing device 11. Such luminance information may be obtained, for example, from a display image or from the output of a sensor or the like provided in the image processing device 11.

[0212] In this case, it is conceivable to perform the processing of step S101 in Fig. 13 based on the luminance information. Specifically, it is conceivable to determine that the AF accuracy is sufficient when the brightness indicated by the luminance information is within a predetermined range, and to determine that the AF accuracy is insufficient when the brightness is outside the predetermined range. This is because when the surroundings are sufficiently bright, there is little variation in the phase difference information in the time direction, and it is highly likely that sufficient accuracy can be obtained with the phase difference AF method.

[0213] <First Modification of the First Embodiment> <Regarding Driving in Phase Difference Detection Mode> When driving the camera 21 (image sensor 32) in the phase difference detection mode, pixel data of phase difference pixels used for phase difference detection, i.e., phase difference pixels used for generating phase difference information, may be thinned out depending on the AF accuracy, the surrounding environment, etc. By doing so, it is possible to further reduce power consumption related to reading out pixel data and generating phase difference information.

[0214] When thinning out phase difference pixels in the phase difference detection mode, the image processing device 11 drives the camera 21 as shown in FIG. 14, for example.

[0215] In the example shown in FIG. 14, similarly to the case described with reference to FIG. 3, the wide-angle camera 21-1 is set as the main camera, and the main camera is driven in the Viewing mode.

[0216] The telephoto camera 21-2 is a sub-camera, and is driven in a phase difference detection mode.

[0217] In this case, however, the MCMU 62 determines the thinning rate of the phase difference pixels in the phase difference detection mode based on the determination result of whether or not the detection accuracy is sufficient, which is supplied from the detection information processing unit 63, and on luminance information indicating the ambient brightness obtained from the display image, etc., which is supplied from the image processing unit 64, etc. The MCMU 62 generates a sensor mode signal indicating the phase difference detection mode, including information indicating the determined thinning rate, and supplies this sensor mode signal to the MCCU 51 of the sub-camera.

[0218] In response to the supplied sensor mode signal, the MCCU 51 of the sub camera supplies the control signal thinning[1:0] indicating the thinning rate specified by the MCMU 62 and the control signal Low_Power_en=High to the pixel unit 41. The MCCU 51 of the sub camera also supplies the control signal ipipe_en=Low to the image processing unit 43 to stop the operation of the image pipeline 52.

[0219] For example, the thinning rate of the phase difference pixels can be set in multiple stages.

[0220] As an example, the control signal thinning[1:0] = b00 can be set to no thinning, i.e., a thinning rate of 0, the control signal thinning[1:0] = b01 can be set to 1 / 4 thinning, the control signal thinning[1:0] = b10 can be set to 1 / 2 thinning, and the control signal thinning[1:0] = b11 can be set to 3 / 4 thinning, etc.

[0221] For example, in 1 / 2 thinning, pixel data is thinned out so that the number of pixels constituting the output image data is half the total number of pixels. Note that the thinning rate may indicate the proportion of pixels to be thinned out to all pixels of the pixel unit 41, or may indicate the proportion of pixels to be thinned out to all phase difference pixels, but here, the thinning rate will be described as indicating the proportion of pixels to be thinned out to all pixels.

[0222] Furthermore, when determining the thinning rate, for example, the higher the AF accuracy, that is, the smaller the variation in the detection information (phase difference information), the more pixels can be thinned out.

[0223] For example, when phase difference detection is performed using all phase difference pixels without thinning out, if the detection result is stable, it means that the subject is in focus and there is little change in the subject, so it is conceivable to change the thinning rate to 3 / 4 and further reduce power consumption.

[0224] 14, the control signal Low_Power_en=High and the control signal thinning[1:0]=b11 are supplied to the pixel unit 41-2. Therefore, the pixel unit 41-2 thins out 3 / 4 of the total pixels, and puts some of the AD conversion units into a pause state, as in the case of FIG. 5, so that the number of pixel data of phase difference pixels constituting the output image data becomes 1 / 4 of the total number of pixels, and outputs image data with the number of pixels corresponding to the thinning rate.

[0225] As an example, assume that a pixel section 41 has a total of 64 pixels, 8×8, and the thinning rate is 3 / 4, as shown in Fig. 15. Note that in Fig. 15, parts corresponding to those in Fig. 4 are given the same reference numerals, and their explanation will be omitted where appropriate.

[0226] In this example, pixels in the hatched portions are thinned out in accordance with a thinning rate of 3 / 4, and some of the AD conversion units 133 are put into a paused state so that only pixel data of phase difference pixels in the non-hatched portions is output.

[0227] In this example, only the AD conversion units 133-1, 133-3, 133-5, and 133-7 are driven, and the remaining AD conversion units 133-2, 133-4, 133-6, and 133-8 are inactive and do not operate.

[0228] Furthermore, adjacent vertical signal lines can be connected to each other as appropriate via switches (not shown) or the like. Therefore, a signal output from a phase difference pixel connected to an AD conversion unit 133 in a dormant state is supplied to an AD conversion unit 133 that is not in a dormant state via a vertical signal line adjacent to the vertical signal line to which the phase difference pixel is connected, and is AD converted. In this way, thinning out at a thinning rate of 3 / 4 is realized.

[0229] In the detection pipeline 53 and detection value processing section 54 subsequent to the pixel section 41, the same processing as that described with reference to FIG. 3 is performed, and phase difference information is generated as detection information.

[0230] Note that the example described here is one in which the control signal thinning[1:0] indicating the thinning rate is supplied to the pixel unit 41, and thinning is achieved by driving the pixel unit 41. However, the present invention is not limited to this, and the control signal thinning[1:0] may be supplied to the image processing unit 43, and the image processing unit 43 may thin out the pixel data.

[0231] In such a case, for example, in the pixel unit 41, some AD conversion units are appropriately put into a paused state so that image data made up of pixel data of all phase difference pixels is output, and in the image processing unit 43, pixels are thinned out, for example, in the crop processing unit 91 or the clamp processing unit 111. That is, of the pixel data of each pixel constituting the input image data, only pixel data of pixels determined by the thinning rate is cut out by cropping or is made the target of processing, and image data made up of only the processed pixel data is output.

[0232] <Description of Thinning-Out Rate Changing Process> When changing the thinning-out rate in the pixel section 41, for example, the image processing device 11 performs the thinning-out rate changing process shown in FIG. 16 during the image capturing operation.

[0233] The thinning rate changing process performed by the image processing device 11 will be described below with reference to the flowchart of FIG.

[0234] In step S131, the MCMU 62 determines whether the detection accuracy is sufficient.

[0235] For example, similar to the determination of whether the AF accuracy is sufficient, the detection information processing unit 63 determines whether the detection accuracy is sufficient based on the variation in the time direction of the phase difference information as detection information, and supplies the determination result to the MCMU 62. The MCMU 62 determines whether the detection accuracy is sufficient based on the determination result supplied from the detection information processing unit 63.

[0236] If the thinning rate can be set in multiple stages, the thinning rate may be determined according to the degree of variation in the detected wave information.

[0237] If it is determined in step S131 that the detection accuracy is sufficient, there is a high possibility that sufficient detection accuracy can be maintained even if the phase difference pixels are thinned out, and the process then proceeds to step S132.

[0238] In step S132, the MCMU 62 sets the driving mode of the main camera to the viewing mode, and the driving mode of the sub camera to the 3 / 4 thinned-out phase difference detection mode.

[0239] The MCMU 62 then generates a sensor mode signal indicating the Viewing mode and supplies the sensor mode signal to the MCCU 51 of the main camera, and also generates a sensor mode signal indicating a 3 / 4 thinning-out phase difference detection mode and supplies the sensor mode signal to the MCCU 51 of the sub camera. In step S132, the thinning-out rate is set to 3 / 4, and a sensor mode signal indicating the phase difference detection mode and including information indicating the thinning-out rate is generated.

[0240] After the process of step S132 is performed, the process proceeds to step S134.

[0241] On the other hand, if it is determined in step S131 that the detection accuracy is not sufficient, in step S133 the MCMU 62 sets the drive mode of the main camera to the viewing mode and the drive mode of the sub-camera to the phase difference detection mode without thinning.

[0242] The MCMU 62 then generates a sensor mode signal in the same manner as in step S132 and supplies it to the MCCU 51 of each camera 21. In this case, the sensor mode signal supplied to the sub-camera is a signal indicating a phase difference detection mode without thinning. After the processing of step S133 is performed, the processing proceeds to step S134.

[0243] In the above steps S132 and S133, it can be said that the thinning rate in the camera 21 (image sensor 32) operated in the phase difference detection mode is determined based on the determination result of whether the detection accuracy is sufficient, in other words, based on the detection information (phase difference information).

[0244] After the processing of step S132 or step S133 has been performed, in step S134, cameras 21-1 and 21-2 perform processing according to the drive mode indicated by the sensor mode signal supplied from MCMU 62, and the thinning rate change processing ends.

[0245] For example, when the processing of step S132 is performed, as described with reference to Fig. 14, in the camera 21 driven in the phase difference detection mode, 3 / 4 of the pixels are thinned out, and some of the AD conversion units are put into a suspended state so that the number of pixels of the image data output from the pixel unit 41 becomes 1 / 4 of the total number of pixels. As a result, image data consisting of pixel data of the number of phase difference pixels determined by the thinning rate (the number of pixels after thinning) is generated.

[0246] In this way, the image processing device 11 changes the thinning rate depending on the detection accuracy, thereby achieving a further reduction in power consumption while performing detection with sufficient accuracy.

[0247] Furthermore, as in the first embodiment, low power consumption can be achieved while maintaining the number of time-wise samplings of detection information in the sub-camera equivalent to that of the main camera, and by limiting the information output from each image sensor 32 to the bare minimum necessary, it is also possible to reduce the operating power of the AP 22 and speed up its operation.

[0248] 14, each camera 21 supplies detection information (phase difference information) to the detection information processing unit 63 of the AP 22, and receives lens position control instructions from the AP 22 to the AF driver 31. This is because the focal length of each camera 21 changes slightly due to temperature, deterioration over time, etc., and calibration for these changes due to temperature, etc. is performed in the detection information processing unit 63.

[0249] However, the present invention is not limited to this, and one of the two cameras 21 may be used as a master camera, and the other camera 21 may be used as a follower camera, with calibration and the like being performed by the master camera.

[0250] In such a case, the detection information (phase difference information) of each follower camera is supplied to the master camera, and the master camera performs calibration based on the detection information of the master camera and the detection information of the follower camera.

[0251] Then, for example, a lens position correction value for correcting a deviation in focal length obtained as a result of the calibration is output to the follower camera. In the AF driver 31 of the follower camera, the lens position obtained by AF is corrected using the correction value, and the lens is moved based on the corrected lens position. In such a case, feedback regarding AF to the AP 22 side is not required.

[0252] The correction of the deviation in focal length may be performed by both the master camera and the follower camera, or may be performed by only one of the master camera and the follower camera, with the other being used as a reference.

[0253] Second Embodiment Example of Configuration of Image Processing Device In the example of FIG. 3, an example in which the MCMU 62 is provided on the AP 22 side has been described, but the MCMU may also be provided inside the image sensor 32.

[0254] In such a case, the image processing device 11 is configured as shown in FIG.

[0255] The configuration of the image processing device 11 shown in Fig. 17 differs from the configuration of the image processing device 11 shown in Fig. 3 only in that the MCMU 62 is not provided in the AP 22, and an MCMU 301-1 is provided in the image sensor 32-1, and an MCMU 301-2 is provided in the image sensor 32-2. These MCMUs 301-1 and 301-2 function as control units that determine the drive mode.

[0256] In the example of FIG. 17, the setting result of the imaging magnification (user-set magnification) set by the user or the like is supplied from the setting unit 61 to the MCMU 301-1 and MCMU 301-2.

[0257] The MCMU 301-1 and MCMU 301-2 determine the drive mode of the camera 21 (image sensor 32) that they own, based on the setting results (photography magnification) and the like supplied from the setting unit 61. Then, the MCMU 301-1 and MCMU 301-2 generate sensor mode signals indicating the determined drive mode and supply them to the MCCU 51-1 and MCCU 51-2.

[0258] In the following description, when there is no need to particularly distinguish between the MCMU 301-1 and the MCMU 301-2, they will also be simply referred to as the MCMU 301.

[0259] The MCMU 301 basically operates in the same way as the MCMU 62. For example, the MCMU 301 can change the drive mode depending on the AF accuracy and detection accuracy upon receiving detection information from the detection value processing unit 54, or can change the drive mode depending on whether or not the target subject is present upon receiving information about the target subject from the image processing unit 64. The MCMU 301 may also set a thinning rate depending on the detection accuracy.

[0260] 17, for example, when the wide-angle camera 21-1 is set to the Viewing mode, the pixel unit 41-1 is supplied with a control signal Low_Power_en=Low, and all pixels are read out. Also, the image processing unit 43-1 is supplied with a control signal ipipe_en=High, and a display image and detection information are generated.

[0261] On the other hand, when the telephoto camera 21-2 is set to the phase difference detection mode, the pixel unit 41-2 is supplied with a control signal Low_Power_en=High, and only the signals of the phase difference pixels are AD converted and read out. Also, the image processing unit 43-2 is supplied with a control signal ipipe_en=Low, and the operation of the image pipeline 52-2 is stopped, and only the generation of detection information is performed.

[0262] In the example of Figure 17, as in the case of Figure 1, a detection value processing unit 54 that generates detection information such as phase difference information is provided inside the image sensor 32, so that the information output from the camera 21 to the AP 22 can be compressed.

[0263] Also, as shown in FIG. 18, for example, a master camera and a follower camera may be defined, and the MCMU 301 may also perform processing related to AF.

[0264] The configuration of the image processing device 11 shown in FIG. 18 differs from the configuration of the image processing device 11 shown in FIG. 17 only in that the AP 22 does not include the detection information processing unit 63 .

[0265] In the example of FIG. 18, the wide-angle camera 21-1 is the master camera, and the telephoto camera 21-2 is the follower camera.

[0266] The master camera performs calibration to correct deviations in focal length of each camera 21 that change due to temperature, aging, etc., i.e., calibration for AF control. In particular, the following description will be given assuming that deviations in focal length of the follower cameras are corrected using the master camera as a reference.

[0267] In camera 21-2, which is the follower camera, the detection information (phase difference information) obtained by the detection value processing unit 54-2 is supplied to MCMU 301-2 and MCMU 301-1. In addition, in camera 21-1, which is the master camera, the detection information (phase difference information) obtained by the detection value processing unit 54-1 is supplied to MCMU 301-1.

[0268] The MCMU 301-1 on the master camera side performs calibration based on the detection information supplied from the detection value processing unit 54-1 and the detection information supplied from the detection value processing unit 54-2 of the other camera 21 (follower camera). During calibration, the original focal length relationship between the master camera and follower camera is used as a reference, and a correction amount (correction value) for the lens position is calculated to correct for any deviation from this. The MCMU 301-1 supplies the correction value obtained by calibration to the MCMU 301-2 on the follower camera side.

[0269] Furthermore, on the master camera side, the MCMU 301-1 calculates the amount of movement of the lens of the camera 21-1 based on the detection information supplied from the detection value processing unit 54-1, and supplies lens control information indicating the calculation result to the AF driver 31-1, thereby controlling the movement of the lens position by the AF driver 31-1. In other words, AF control is performed by the MCMU 301-1.

[0270] On the follower camera side, the MCMU 301-2 calculates the amount of lens movement of the camera 21-2 based on the detection information supplied from the detection value processing unit 54-2, and corrects the calculation result based on the correction value supplied from the MCMU 301-1. For example, the correction is performed by adding the correction value to the amount of lens movement. The MCMU 301-2 supplies lens control information indicating the amount of lens movement after correction to the AF driver 31-2, thereby controlling the movement of the lens position by the AF driver 31-2.

[0271] In the example of FIG. 18, all processing for AF control is performed within the image sensor 32, so feedback regarding AF control to the AP 22 side is not required.

[0272] 17 and 18, the image processing device 11 has the MCMU 301 inside the image sensor 32, so that the processing for reducing power consumption without reducing the detection accuracy of the present technology can be completed inside the image sensor 32 without going through the AP 22. This enables high-speed feedback to, for example, the AF driver 31. As a result, high-speed AF can be performed immediately after the camera 21 is started up, for example.

[0273] Furthermore, as in the first embodiment, low power consumption can be achieved while maintaining the number of time-wise samplings of detection information in the sub-camera equivalent to that of the main camera, and by limiting the information output from each image sensor 32 to the bare minimum necessary, it is also possible to reduce the operating power of the AP 22 and speed up its operation.

[0274] In other embodiments described later, the MCMU 301 may also be provided inside the image sensor 32, as in the examples shown in FIGS.

[0275] Third Embodiment Driving Example of Image Processing Device A luminance detection mode for AE (Auto Exposure) control may be included as a driving mode of the camera 21 (image sensor 32). In the luminance detection mode, processing related to luminance detection for AE control is performed, and luminance information indicating the luminance (brightness) of each region in a captured image, i.e., each region around the image processing device 11, is generated as detection information.

[0276] As an example, immediately after starting up the image processing device 11, it is possible to drive all of the cameras 21, i.e., both the main camera and the sub-camera, in the brightness detection mode, as shown in Fig. 19. This is because it is difficult to perform appropriate AF control unless the exposure state is appropriate, and in order to achieve fast camera startup, it is important to first achieve an appropriate exposure state.

[0277] In the example shown in FIG. 19, the MCMU 62 supplies a sensor mode signal indicating the luminance detection mode to the MCCU 51-1 and MCCU 51-2.

[0278] Then, each MCCU 51 performs driving in accordance with the supplied sensor mode signal. That is, the MCCU 51 supplies a control signal Low_Power_en=High to the pixel unit 41 and supplies a control signal ipipe_en=Low to the image processing unit 43 in accordance with the supplied sensor mode signal, i.e., the luminance detection mode.

[0279] When the control signal Low_Power_en=High is supplied in the luminance detection mode, the pixel unit 41 performs binning readout, which is a drive for adding and reading out signals output from multiple pixels, including pixels of different colors. In this example, monochrome readout by four-color addition (2×2 binning) is performed, in which signals from four adjacent normal pixels are added and read out as a monochrome (luminance) signal.

[0280] As a result, for example, image data having a number of pixels less than the total number of pixels, such as 1512×2016 pixels, is output (supplied) from the pixel unit 41 to the image processing unit 43 .

[0281] In addition, in response to the control signal ipipe_en=Low from the MCCU 51, the image processing unit 43 stops the operation of the image pipeline 52 by stopping the supply of the clock signal, and operates only the detection pipeline 53.

[0282] The detection pipeline 53 performs clamping processing on the image data supplied from the pixel unit 41 via the crop processing unit 91 and the defect correction unit 92, and supplies the resulting image data to the detection value processing unit 54. The detection value processing unit 54 generates luminance information as detection information based on the image data supplied from the detection pipeline 53, and supplies the luminance information to the detection information processing unit 63. In this example, the detection value processing unit 54 of each camera 21 outputs detection information at the same frame rate (e.g., 30 fps) as the frame rate of the image during shooting.

[0283] The detection information processing unit 63 performs processing for AE control based on the luminance information as detection information supplied from the detection value processing unit 54, and supplies the processing results to the MCMU 62. In the AE control processing, for example, the exposure time and gain are determined based on the luminance information. The MCMU 62 controls the operation of the control unit 42 (MCCU 51) of each camera 21 so that shooting is performed with the determined exposure time and gain processing is performed with the determined gain.

[0284] When capturing an image, the main camera may be driven in a viewing mode and the sub-camera may be driven in a brightness detection mode, as appropriate.

[0285] Here, an example of monochrome reading and generation of luminance information (detection information) during driving in the luminance detection mode will be described.

[0286] For example, as shown in Fig. 20 , a pixel unit 41 has a plurality of pixels including pixel 341, pixel 342, pixel 343, and pixel 344 arranged in a matrix. Note that in Fig. 20 , parts corresponding to those in Fig. 4 are given the same reference numerals, and their description will be omitted. In addition, for ease of understanding, it is assumed here that all of the pixels depicted in Fig. 20 are normal pixels (pixels that are not phase difference pixels).

[0287] 20 , each pixel constituting a pixel column is connected to a vertical signal line, and a switch 351 is provided between adjacent vertical signal lines as needed. That is, the switch 351 makes it possible to electrically connect or disconnect adjacent vertical signal lines. Hereinafter, when there is no need to particularly distinguish between switches connected to vertical signal lines, they will be referred to as switches 351.

[0288] In the example shown in FIG. 20, when the device is driven in the viewing mode, the switch 351 is turned off, and the signals from all the pixels in the pixel section 41 are AD converted by the AD conversion section 133 .

[0289] On the other hand, when driven in the luminance detection mode, for example, as shown in FIG. 21, the switch 351 is turned on and monochrome reading by adding four colors is performed.

[0290] In the example of Figure 21, signals output from each of the 2 x 2 pixels within an area surrounded by a solid line frame, such as area R31, are added (source follower addition) on the vertical signal line before being input to the AD conversion unit 133.

[0291] For example, in region R31, the signal from pixel 341, which is an R pixel, the signal from pixels 342 and 343, which are G pixels, and the signal from pixel 344, which is a B pixel, are added together on the vertical signal line, and the signal after addition is input to AD conversion unit 133-1.

[0292] By adding the outputs of one R pixel, two G pixels, and one B pixel, a signal indicating luminance, i.e., a pixel signal of a monochrome image (luminance image), is obtained as a signal after addition. For example, if the pixel unit 41 has 3024 × 4032 normal pixels, then in the example of Figure 21, image data of 1512 × 2016 pixels will be output from the pixel unit 41.

[0293] In this example, since AD ​​conversion is not required in the AD conversion units 133-2, 133-4, 133-6, and 133-8, these AD conversion units 133 are put into a dormant state, thereby reducing power consumption. In particular, in this example, half of the AD conversion units 133 can be put into a dormant state.

[0294] Note that while Figure 21 shows an example in which signals from 2 x 2 pixels are added together via vertical signal lines, this is not limiting, and signals from 4 x 4 pixels or signals from 8 x 8 pixels may be added together, depending on the wiring arrangement, etc.

[0295] In the subsequent detection value processing unit 54, luminance information is generated as detection information based on the supplied image data of the luminance image.

[0296] In this case, information indicating luminance is obtained for each block consisting of a plurality of adjacent pixels, such as region R41 in FIG. 21, and the information indicating the luminance for each block is used as luminance information (detection information).

[0297] In this example, one block is an area consisting of 4 vertical pixels and 6 horizontal pixels on the pixel unit 41. This block is an area of ​​2 × 3 adjacent pixels on the image supplied to the detection value processing unit 54, so the detection value processing unit 54 calculates the average value of the pixel data of each of the 2 × 3 pixels in the block as information indicating the luminance of that block.

[0298] The blocks to be used for calculating the arithmetic average when generating luminance information as detection information are not limited to the example shown in FIG. 21 , and the entire image may be divided into 16 vertical by 9 horizontal blocks, and the arithmetic average of pixel data may be calculated for each of these blocks to be used as luminance information.

[0299] <Explanation of Post-Startup Processing> When the drive mode includes a luminance detection mode, the processing shown in Fig. 22 may be performed as post-startup processing that is performed immediately after startup of the image processing device 11. Hereinafter, the post-startup processing performed by the image processing device 11 will be described with reference to the flowchart of Fig. 22 .

[0300] In step S171, the MCMU 62 determines the main camera and the imaging magnification of the camera 21 (image). For example, in step S171, the same process as in step S11 in FIG.

[0301] In step S172, the MCMU 62 instructs the main camera and the sub camera to operate in the brightness detection mode.

[0302] That is, immediately after the image processing device 11 is started, the MCMU 62 sets the drive mode of all cameras 21 (image sensors 32) to the luminance detection mode. The MCMU 62 generates a sensor mode signal indicating the luminance detection mode and supplies the sensor mode signal to the MCCU 51-1 and MCCU 51-2.

[0303] In step S173, the cameras 21-1 and 21-2 are driven in the luminance detection mode.

[0304] Specifically, the MCCU 51 of each camera 21 generates a control signal Low_Power_en=High and supplies it to the pixel unit 41 in accordance with the sensor mode signal supplied from the MCMU 62, and also generates a control signal ipipe_en=Low and supplies it to the image processing unit 43.

[0305] The pixel unit 41 outputs signals obtained by photoelectric conversion from normal pixels that are not phase difference pixels, in response to a control signal Low_Power_en=High from the MCCU 51. Furthermore, the pixel unit 41 connects two adjacent vertical signal lines by turning on switches 351 provided between the vertical signal lines (conducting state), and drives only those of the AD conversion units 133 to which signals from normal pixels are input to perform AD conversion, while not driving the other AD conversion units 133 to which signals from normal pixels are not input, keeping them in a dormant state.

[0306] As a result, signals from multiple pixels, such as four pixels adjacent to each other on a vertical signal line, are added together and supplied to the AD conversion unit 133, which then performs AD conversion on the added signal. As a result, image data indicating the luminance of each region made up of multiple pixels is supplied (output) from the pixel unit 41 to the image processing unit 43.

[0307] In addition, in response to the control signal ipipe_en = Low from the MCCU 51, the image processing unit 43 stops the operation of the image pipeline 52 by stopping the clock supplied to the image pipeline 52, while allowing each part other than the image pipeline 52 to operate.

[0308] Furthermore, the image data supplied from the pixel unit 41 to the image processing unit 43 is subjected to cropping by the crop processing unit 91, defect correction by the defect correction unit 92, and clamping by the clamp processing unit 111, and the image data obtained as a result is supplied to the detection value processing unit 54. The detection value processing unit 54 calculates the arithmetic mean value of the pixel data for each block described above based on the image data supplied from the clamp processing unit 111, generates luminance information as detection information, and supplies it to the detection information processing unit 63.

[0309] The detection information processing unit 63 performs processing for AE control based on the luminance information as detection information supplied from the detection value processing unit 54, and supplies the processing results to the MCMU 62. The MCMU 62 controls the operation of the control unit 42 (MCCU 51) of each camera 21 based on the processing results for AE control supplied from the detection information processing unit 63, and changes the exposure time in the pixel unit 41 and the gain value inside the pixel unit 41 or in the image processing unit 43 as appropriate.

[0310] In this way, immediately after startup, both the main camera and the sub camera are driven in the brightness detection mode, so that appropriate exposure can be achieved more quickly.

[0311] In step S174, the MCMU 62 determines whether the AE accuracy is sufficient, in other words, whether the exposure is appropriate.

[0312] For example, when operating in the luminance detection mode immediately after startup, the detection information processing unit 63 determines whether the AE accuracy is sufficient based on the detection information for each frame (time) supplied from the detection value processing unit 54 at a predetermined frame rate, and supplies the determination result to the MCMU 62.

[0313] As a specific example, if the variation in the time direction of the luminance information as detection information is within a predetermined range (predetermined value), the AE accuracy is determined to be sufficient. This is because when the variation in the luminance information is small, there is a high possibility that the exposure is appropriate.

[0314] The detection information processing unit 63 may be configured to determine that the AE accuracy is sufficient when the variation in the brightness information of at least the main camera is below a predetermined value, or may be configured to determine that the AE accuracy is sufficient when the variation in the brightness information of both the main camera and the sub-camera is below a predetermined value.

[0315] The MCMU 62 determines that the AE accuracy is sufficient when the detection information processing unit 63 supplies the MCMU 62 with a determination result that sufficient AE accuracy has been obtained. In this case, it can be said that the MCMU 62 determines whether the AE accuracy is sufficient based on the detection information.

[0316] If it is determined in step S174 that the AE accuracy is insufficient, the process returns to step S172, and the above-described process is continued. In this case, the exposure time and gain value are changed appropriately, and brightness detection is performed.

[0317] On the other hand, if it is determined in step S174 that the AE accuracy is sufficient, the processes from step S175 to step S181 are performed, and the post-startup process ends. Note that the processes from step S175 to step S181 are similar to the processes from step S12 to step S18 in FIG. 10, and therefore a description thereof will be omitted.

[0318] In this way, the image processing device 11 drives both the main camera and the sub-camera in the brightness detection mode immediately after startup, thereby speeding up the startup of the camera 21 with high-speed appropriate exposure and reducing power consumption of the camera 21.

[0319] For example, during luminance detection, since problems such as a decrease in detection accuracy are unlikely to occur even if the number of signals added on the vertical signal lines of the pixel unit 41 is increased, the number of signals added on the vertical signal lines can be increased to, for example, 4 × 4 pixels to 64 × 64 pixels, etc. This makes it possible to achieve a high dynamic range of luminance information, high-speed luminance detection, and low power consumption.

[0320] In particular, by setting all cameras 21 to the brightness detection mode immediately after startup, appropriate exposure can be achieved quickly, resulting in faster startup of the cameras 21. In other words, the time required to reach a state where appropriate photography is possible can be shortened.

[0321] Furthermore, as in the first embodiment, low power consumption can be achieved while maintaining the number of time-wise samplings of detection information in the sub-camera equivalent to that of the main camera, and by limiting the information output from each image sensor 32 to the bare minimum necessary, it is also possible to reduce the operating power of the AP 22 and speed up its operation.

[0322] In particular, this embodiment differs from the first embodiment in that the number of AD conversions per frame can be reduced by performing signal binning on the vertical signal lines in accordance with the information output to the AP 22 and AD converting the signal after binning, thereby achieving low power consumption without sacrificing detection accuracy.

[0323] <Fourth embodiment> <Driving example of image processing device> When the dynamic range of the photographed scene is narrow, the power supply voltage of the analog circuit inside the image sensor 32 of the camera 21 driven in the phase difference detection mode may be reduced.

[0324] An example will be described in which the voltage (power supply voltage) supplied to the AD conversion unit 133 as an analog circuit inside the image sensor 32, that is, the voltage value used for AD conversion, is changed to a smaller value.

[0325] As an example, as shown in FIG. 23, it is assumed that the wide-angle camera 21-1 is the main camera and the telephoto camera 21-2 is the sub-camera.

[0326] In this example, the wide-angle camera 21-1, which is the main camera, is driven in the viewing mode, and the telephoto camera 21-2, which is the sub-camera, is driven in the phase difference detection mode. In particular, the driving in the viewing mode is the same as that described with reference to FIG.

[0327] The image processing unit 64 generates brightness information, which is information relating to the brightness of the display image, based on the image data of the display image supplied from the main camera image pipeline 52 - 1 , and supplies this to the MCMU 62 .

[0328] This luminance information may be information that can identify the dynamic range of luminance in the image for display, and may be the same as the luminance information obtained as detection information in the luminance detection mode, for example. Alternatively, the luminance information may be generated by the image processing unit 64 or the detection information processing unit 63 based on the image for display, etc., and supplied from the detection information processing unit 63 to the MCMU 62.

[0329] Based on the brightness information supplied from the image processing unit 64, the MCMU 62 determines whether the dynamic range of brightness in the display image is equal to or less than a predetermined value, i.e., whether the dynamic range is narrow, and supplies the sub-camera with a signal corresponding to the determination result and the drive mode.

[0330] For example, if the MCMU 62 determines that the dynamic range is narrow, it supplies a signal indicating that the dynamic range is narrow, together with a sensor mode signal indicating the phase difference detection mode, to the MCCU 51-2 of the sub-camera 21-2, as shown in Fig. 23. In other words, a sensor mode signal indicating that the dynamic range is narrow and that the mode is the phase difference detection mode is supplied to the MCCU 51-2.

[0331] In this case, in response to the sensor mode signal supplied from the MCMU 62, the MCCU 51-2 supplies a control signal Low_Power_en=High to the pixel unit 41-2, and supplies a control signal ipipe_en=Low to the image processing unit 43-2.

[0332] Here, the MCCU 51-2 supplies a control signal Low_Power_en=High to the pixel unit 41-2, thereby lowering (changing) the power supply voltage of the analog circuit in the pixel unit 41-2 so that the voltage supplied to the analog circuit in the pixel unit 41-2 becomes lower. Also, in the pixel unit 41-2, some of the AD conversion units 133, i.e., AD conversion units 133 to which signals from the phase difference pixels are not input, are put into a paused state, as in the example of FIG.

[0333] The analog circuit here refers to a circuit provided between the pixel provided in the pixel section 41 and the AD conversion section 133, such as a circuit within the pixel, the AD conversion section 133, or a load MOS (Metal Oxide Semiconductor) circuit that applies a load (voltage) to the vertical signal line.

[0334] For example, if the voltage supplied to the AD conversion unit 133, i.e., the voltage value used for AD conversion (voltage during AD conversion), is reduced, the dynamic range of the pixel data, i.e., the dynamic range of the phase difference information obtained as detection information, decreases. However, if the dynamic range of the display image is narrow, the detection accuracy is unlikely to decrease due to the reduction in the dynamic range of the phase difference information. Furthermore, by lowering the power supply voltage of the analog circuit in exchange for the reduction in the dynamic range of the phase difference information, it is possible to reduce the power consumption of the image sensor 32.

[0335] In this example, regardless of whether the power supply voltage of the analog circuit is changed, image data of, for example, 378 × 504 pixels is output from the pixel unit 41-2 of the camera 21-2, which is the sub-camera, in the same manner as in the example of Fig. 3. This image data is image data made up of pixel data of phase difference pixels.

[0336] 3, in the image processing unit 43-2, the operation of the image pipeline 52-2 is stopped based on the control signal IPipe_en=Low, and various processes are performed in the detection pipeline 53-2 and the detection value processing unit 54-2, and phase difference information is generated as detection information. In the example of Fig. 23, as in the example of Fig. 3, phase difference information as detection information is generated at the same frame rate as the display image and detection information obtained by the main camera, such as 30 fps, and is supplied from the detection value processing unit 54-2 to the detection information processing unit 63.

[0337] As described above, in the example of FIG. 23 , when driving in the phase difference detection mode, the power consumption can be reduced by appropriately lowering the voltage supplied to the analog circuits in the pixel unit 41, such as the pixel power supply, load MOS, and AD conversion unit 133, i.e., the power supply voltage of the analog circuits.

[0338] The power consumption when pixel data is read from the pixel unit 41 is determined by the voltage, the amount of pixel data read (the number of pixels in the image data), and the frequency when the pixel data is read. In other embodiments, power consumption is reduced by reducing the amount of read, but in this embodiment, power consumption can also be reduced by lowering the voltage.

[0339] Furthermore, as in the first embodiment, low power consumption can be achieved while maintaining the number of time-wise samplings of detection information in the sub-camera equivalent to that of the main camera, and by limiting the information output from each image sensor 32 to the bare minimum necessary, it is also possible to reduce the operating power of the AP 22 and speed up its operation.

[0340] Fifth Embodiment Example of Driving Image Processing Device When the shooting scene is sufficiently bright (the surroundings of the image processing device 11 are sufficiently bright), the number of bits during AD conversion may be lowered (reduced), that is, the number of bits of pixel data may be reduced, thereby realizing low power consumption.

[0341] An example of driving the image processing device 11 when the number of bits during AD conversion is reduced will be described with reference to FIG.

[0342] In the example of Figure 24, the wide-angle camera 21-1 is the main camera, and the telephoto camera 21-2 is the sub-camera. The wide-angle camera 21-1, which is the main camera, is driven in the viewing mode, and the telephoto camera 21-2, which is the sub-camera, is driven in the phase difference detection mode. Driving in the viewing mode is the same as that described with reference to Figure 3.

[0343] 23 , the image processing unit 64 generates luminance information, which is information relating to the luminance of the display image, based on image data of the display image from the main camera, and supplies the luminance information to the MCMU 62. Note that the luminance information may be generated by the image processing unit 64 or the detection information processing unit 63 based on the display image, etc., and supplied from the detection information processing unit 63 to the MCMU 62.

[0344] Based on the brightness information supplied from the image processing unit 64, the MCMU 62 determines whether the image to be displayed is a sufficiently bright image, i.e., whether the surroundings of the image processing device 11 are sufficiently bright, and supplies the sub-camera with a signal corresponding to the determination result and the drive mode.

[0345] The MCMU 62 determines that the image is sufficiently bright when the brightness of the surroundings of the image processing device 11 is equal to or greater than a predetermined value, such as when the average brightness of the display image indicated by the brightness information is equal to or greater than a predetermined threshold value. Note that the determination of whether the image is sufficiently bright or not is not limited to threshold processing related to the brightness of the display image, and any method may be used.

[0346] If the MCMU 62 determines that the image is sufficiently bright, it supplies a signal indicating that the image is sufficiently bright, together with a sensor mode signal indicating the phase difference detection mode, to the MCCU 51-2 of the sub-camera camera 21-2. In other words, a sensor mode signal indicating that the image is sufficiently bright and that the image is in the phase difference detection mode is supplied to the MCCU 51-2.

[0347] In this case, in response to the sensor mode signal supplied from the MCMU 62, the MCCU 51-2 supplies a control signal Low_Power_en=High to the pixel unit 41-2, and supplies a control signal ipipe_en=Low to the image processing unit 43-2.

[0348] Here, the MCCU 51-2 supplies a control signal Low_Power_en=High to the pixel unit 41-2, thereby instructing control of the operation of the AD conversion unit 133 so as to reduce the number of bits during AD conversion. In other words, the MCCU 51-2 changes the number of bits of pixel data during AD conversion so as to reduce the number of bits of pixel data obtained by AD conversion. Also, in the pixel unit 41-2, as in the example of FIG. 3, the AD conversion unit 133 to which no signal is input from the phase difference pixel is put into a pause state.

[0349] By performing such control, only the signals output from the phase difference pixels are AD converted, and image data consisting of pixel data for each phase difference pixel obtained as a result is obtained. That is, for example, image data of 378 × 504 pixels is output from the pixel unit 41-2, as in the example of FIG. 3. However, in this example, AD conversion is performed in the AD conversion unit 133 so that the number of bits of pixel data for one pixel is smaller than in the example of FIG. 3. In other words, the value of the pixel data is expressed with fewer bits than in the example of FIG. 3.

[0350] 3, in the image processing unit 43-2, the operation of the image pipeline 52-2 is stopped based on the control signal IPipe_en=Low, and various processes are performed in the detection pipeline 53-2 and the detection value processing unit 54-2, and phase difference information is generated as detection information. In the example of Fig. 24, as in the example of Fig. 3, phase difference information as detection information is generated at the same frame rate as the display image and detection information obtained by the main camera, such as 30 fps, and is supplied from the detection value processing unit 54-2 to the detection information processing unit 63.

[0351] As described above, in the example of Figure 24, when driving in the phase difference detection mode, the number of bits during AD conversion (reducing the number of bits of pixel data) can be appropriately reduced to reduce the number of bits (data volume) of image data. This makes it possible to reduce the amount of data, such as image data and detection information, handled by the image processing unit 43 and the like downstream of the pixel unit 41, thereby achieving lower power consumption and faster processing. Furthermore, by reducing the number of bits during AD conversion, the time required for AD conversion can be shortened, thereby achieving even lower power consumption.

[0352] Furthermore, as in the first embodiment, low power consumption can be achieved while maintaining the number of time-wise samplings of detection information in the sub-camera equivalent to that of the main camera, and by limiting the information output from each image sensor 32 to the bare minimum necessary, it is also possible to reduce the operating power of the AP 22 and speed up its operation.

[0353] Sixth Embodiment Example of Driving Image Processing Device When the shooting scene is very bright and there is a possibility of frequent whiteout, low power consumption may be achieved by attenuating the signal before AD conversion.

[0354] An example of driving the image processing device 11 when attenuating a signal before AD conversion will be described with reference to FIG.

[0355] In the example of Figure 25, the wide-angle camera 21-1 is the main camera, and the telephoto camera 21-2 is the sub-camera. The wide-angle camera 21-1, which is the main camera, is driven in the viewing mode, and the telephoto camera 21-2, which is the sub-camera, is driven in the phase difference detection mode. Driving in the viewing mode is the same as that described with reference to Figure 3.

[0356] 24 , the image processing unit 64 generates luminance information based on image data of the display image from the main camera and supplies it to the MCMU 62. Note that the luminance information may be generated by the image processing unit 64 or the detection information processing unit 63 based on the display image or the like, and supplied from the detection information processing unit 63 to the MCMU 62.

[0357] Based on the brightness information supplied from the image processing unit 64, the MCMU 62 determines whether the area around the image processing device 11 is sufficiently bright (very bright) in the same manner as in FIG. 24, and supplies a signal corresponding to the determination result and the drive mode to the sub-camera.

[0358] For example, when the brightness of the surroundings of the image processing device 11 is equal to or greater than a predetermined value, such as when the average brightness of the display image indicated by the brightness information is equal to or greater than a predetermined threshold, it is determined that the surroundings are sufficiently bright. In this case, for example, the threshold used for the determination is set to a value greater than that in the case of FIG.

[0359] If the MCMU 62 determines that the image is sufficiently bright, it generates a sensor mode signal indicating that the image is sufficiently bright and that the mode is phase difference detection mode, and supplies the signal to the MCCU 51-2. In this case, the MCCU 51-2 supplies a control signal Low_Power_en=High to the pixel unit 41-2 and a control signal ipipe_en=Low to the image processing unit 43-2 in response to the sensor mode signal supplied from the MCMU 62.

[0360] The MCCU 51-2 controls the operation of the pixel section 41-2 by supplying a control signal Low_Power_en=High to the pixel section 41-2 so that parasitic capacitance is added to the vertical signal line.

[0361] The pixel unit 41-2, under the control of the MCCU 51-2, turns on a switch (not shown) in the pixel unit 41-2 (to a conductive state) and adds parasitic capacitance to the vertical signal line provided between the pixel and the AD conversion unit 133 by electrically connecting a capacitor via the switch to the vertical signal line. Also, in the pixel unit 41-2, the AD conversion unit 133 to which no signal is input from the phase difference pixel is put into a dormant state, as in the example of FIG.

[0362] By performing such control, the signals output from the phase difference pixels are attenuated in the vertical signal lines by the added parasitic capacitance and input to the AD conversion unit 133. Furthermore, the AD conversion unit 133 performs AD conversion on only the signals output from the phase difference pixels, and as a result, image data made up of pixel data for each phase difference pixel is obtained. That is, for example, image data of 378 × 504 pixels is output from the pixel unit 41-2, as in the example of FIG. 3 .

[0363] In the image processing unit 43-2, as in the example of Fig. 3, the operation of the image pipeline 52-2 is stopped based on the control signal ipipe_en = Low, and various processes are performed in the detection pipeline 53-2 and the detection value processing unit 54-2, and phase difference information is generated as detection information. In the example of Fig. 25, as in the example of Fig. 3, phase difference information as detection information is generated at the same frame rate as the display image and detection information obtained by the main camera, such as 30 fps, and is supplied from the detection value processing unit 54-2 to the detection information processing unit 63.

[0364] As described above, in the example of FIG. 25 , by appropriately attenuating the signal before AD conversion when driving in the phase difference detection mode, it is possible to reduce the power consumption of the image sensor 32 while ensuring the dynamic range of the signal at the time when the signal from the phase difference pixel is input to the AD conversion unit 133.

[0365] Furthermore, as in the first embodiment, low power consumption can be achieved while maintaining the number of time-wise samplings of detection information in the sub-camera equivalent to that of the main camera, and by limiting the information output from each image sensor 32 to the bare minimum necessary, it is also possible to reduce the operating power of the AP 22 and speed up its operation.

[0366] Seventh Embodiment Example of Configuration of Image Processing Device When attenuating a signal before AD conversion depending on whether the surroundings of the image processing device 11 are sufficiently bright, information obtained by an external sensor provided outside the image sensor 32 may be used.

[0367] In such a case, the image processing device 11 is configured as shown in FIG. 26, for example.

[0368] The configuration of the image processing device 11 shown in FIG. 26 is the same as that of the image processing device 11 shown in FIG. 1, except that an external sensor 391 is newly provided.

[0369] The external sensor 391 is a sensor different from the image sensor 32, and is, for example, an ambient light sensor provided inside the image processing device 11 or attached to the outside of the image processing device 11. The external sensor 391 detects the brightness (luminance) of the surroundings of the image processing device 11, and supplies luminance information indicating the detection result to the MCMU 62.

[0370] Based on the brightness information supplied from the external sensor 391, the MCMU 62 determines whether the area around the image processing device 11 is sufficiently bright, in the same manner as in FIG. 25, and supplies a signal corresponding to the determination result and the drive mode to the sub-camera.

[0371] For example, if the MCMU 62 determines that the image is sufficiently bright, it generates a sensor mode signal indicating that the image is sufficiently bright and that the mode is the phase difference detection mode, and supplies this to the MCCU 51 of the sub-camera.

[0372] 26, the wide-angle camera 21-1 is the main camera, and the telephoto camera 21-2 is the sub-camera. The wide-angle camera 21-1, which is the main camera, is driven in the viewing mode, and the telephoto camera 21-2, which is the sub-camera, is driven in the phase difference detection mode.

[0373] The driving of the main camera in the viewing mode and the driving of the sub-camera in the phase difference detection mode are the same as in the example of FIG.

[0374] Therefore, the pixel unit 41-2 of the telephoto side camera 21-2, which is the sub-camera, operates so that parasitic capacitance is added to the vertical signal line, and the AD conversion unit 133, to which signals from the phase difference pixels are not input, is put into a dormant state. As a result, the signals output from the phase difference pixels are attenuated in the vertical signal line. Furthermore, image data made up of pixel data for each phase difference pixel is output from the pixel unit 41-2.

[0375] In the image processing unit 43-2, the operation of the image pipeline 52-2 is stopped, and the image processing unit 43-2 outputs phase difference information as detection information to the AP 22 at the same frame rate as the display image and detection information obtained by the main camera.

[0376] As described above, in the example of FIG. 26 , by appropriately attenuating the signal before AD conversion when driving in the phase difference detection mode, it is possible to reduce the power consumption of the image sensor 32 while ensuring the dynamic range of the signal at the time when the signal from the phase difference pixel is input to the AD conversion unit 133.

[0377] In particular, in this example, input from the external sensor 391 to the MCMU 62 can be asynchronous with the image sensor 32, so that the results of determining the drive method based on the luminance information can be reflected at high speed.

[0378] When determining the drive mode based on information obtained by the image sensors 32, it is assumed that the image sensors 32 are synchronized, so a delay of one frame or more occurs before the drive mode determination result is reflected. In contrast, if the external sensor 391 operates asynchronously with the image sensors 32, for example, and the external sensor 391 operates at a speed equal to or faster than the frame rate of the image sensors 32, the drive mode determination result can be reflected with a delay of one frame or less.

[0379] Also, as in the first embodiment, low power consumption can be achieved while maintaining the same number of sampling times of detection information in the sub-camera as in the main camera, and reducing the amount of information output from each image sensor 32 to the minimum necessary can also reduce the operating power consumption and increase the operating speed of the AP 22. Note that the output of the external sensor 391 can also be used to determine whether the dynamic range is narrow in the fourth embodiment described above, or whether the light is sufficiently bright in the fifth embodiment.

[0380] Eighth Embodiment Driving Example of Image Processing Device In the first embodiment, it has been described that the image processing device 11 changes the angle of view for phase difference detection in the sub-camera depending on whether or not a target subject is present.

[0381] In this embodiment, a more specific example of changing the angle of view will be described.

[0382] For example, as shown in FIG. 27, it is assumed that the wide-angle camera 21-1 is the main camera and the telephoto camera 21-2 is the sub-camera.

[0383] Also, assume that the wide-angle camera 21-1, which is the main camera, is driven in the viewing mode, and the telephoto camera 21-2, which is the sub-camera, is driven in the phase difference detection mode. Driving in the viewing mode and driving in the phase difference detection mode are basically the same as those described with reference to FIG.

[0384] The image processing device 11 performs the angle of view change process described with reference to Fig. 12. Here, the description of the parts that have already been described with reference to Fig. 12 will be omitted as appropriate.

[0385] In the angle of view change process, the image processing unit 64 performs image recognition and the like on the display image supplied from the main camera image pipeline 52 (signal processing unit 103) to detect the area of ​​the subject of interest from the display image.

[0386] The subject of interest may be a human face, a subject in focus, a subject designated by the user who is the photographer, etc. For example, when a subject of interest is designated by the user, the image processing unit 64 detects the area of ​​the subject of interest based on a signal supplied from the setting unit 61 in response to an input operation performed by the user to designate the subject of interest.

[0387] The image processing unit 64 generates coordinate information indicating the area (range) of the target subject in the display image, and supplies the coordinate information to the MCMU 62. In step S71 of Fig. 12, the MCMU 62 determines whether or not the target subject is present, based on the coordinate information supplied from the image processing unit 64.

[0388] Furthermore, if it is determined in step S71 that a subject of interest is present, the MCMU 62 sets the drive mode of the main camera to Viewing mode and the drive mode of the sub-camera to Phase Difference Detection Mode with a cropped angle of view, generates a sensor mode signal, and supplies the sensor mode signal to the MCCU 51 of each camera 21. In this case, the MCMU 62 causes the sub-camera driven in Phase Difference Detection Mode to generate detection information (phase difference information) targeting the area of ​​the subject of interest.

[0389] 27, the MCMU 62 supplies a signal indicating the position (area) of the target subject along with a sensor mode signal indicating the phase difference detection mode to the MCCU 51-2 of the telephoto camera 21-2, which is a sub-camera. In other words, a sensor mode signal indicating the phase difference detection mode and including information indicating the area of ​​the target subject is supplied to the MCCU 51-2.

[0390] In step S74 of FIG. 12, the MCCU 51-2 generates a control signal Low_Power_en=High in response to the supplied sensor mode signal and supplies it to the pixel unit 41-2, and also generates a control signal ipipe_en=Low and supplies it to the image processing unit 43-2.

[0391] At this time, based on the information indicating the area of ​​the target subject contained in the sensor mode signal, the MCCU 51-2 generates range information indicating the area of ​​the target subject in the pixel unit 41-2, more specifically, the area (range) of the target subject in the image captured by the pixel unit 41-2, and supplies this information to the image processing unit 43-2.

[0392] For example, the range information may be information indicating the coordinates of the top left vertex position of the area of ​​the target subject, and the width and height of the area of ​​the target subject. This range information can also be said to be information indicating the crop range.

[0393] In response to the control signal Low_Power_en=High, the pixel unit 41-2 places the AD conversion unit 133, to which signals from the phase difference pixels are not input, in a pause state, as in the example of FIG. 3, and outputs image data consisting only of pixel data of the phase difference pixels to the image processing unit 43-2.

[0394] In the image processing unit 43-2, as in the example of Figure 3, the operation of the image pipeline 52-2 is stopped based on the control signal ipipe_en = Low, and various processing is performed in the detection pipeline 53-2 and the detection value processing unit 54-2, and phase difference information is generated as detection information.

[0395] However, in this example, the crop processing unit 91 of the image processing unit 43-2 performs crop processing to cut out the range (area) indicated by the range information supplied from the MCCU 51-2 from the image based on the image data supplied from the pixel unit 41-2.

[0396] That is, as a process for generating phase difference information for the region of the target subject, a cropping process is performed to crop the region indicated by the range information, whereby image data consisting only of pixel data of phase difference pixels within the region of the target subject indicated by the range information is supplied from the cropping process unit 91 to the defect correction unit 92.

[0397] Therefore, in this example, the range to be cropped (cut out), that is, the range indicated by the range information, is adaptively changed according to the position of the target subject in the display image.

[0398] For example, as shown in FIG. 28, it is assumed that a display image P11 of a predetermined (n-1)th frame is captured by the main camera.

[0399] In this example, the image processing unit 64 detects a subject of interest in an area R81 of the display image P11, and coordinate information indicating the area R81 containing the subject of interest as a detection result is supplied to the MCMU 62. Therefore, the MCMU 62 supplies the MCCU 51 of the sub-camera with a sensor mode signal indicating the phase difference detection mode and including information indicating the area R81.

[0400] Therefore, in the next nth frame, when an image P12 consisting of pixel data of phase difference pixels is obtained by the pixel section 41 of the sub-camera, an area of ​​the image P12 corresponding to the area R81 is cropped, and detection information (phase difference information) is generated based on the resulting image P13.

[0401] For example, as shown in FIG. 29, it is assumed that the image output from the pixel unit 41 of the sub-camera and input to the crop processing unit 91 is an image of 378×504 pixels.

[0402] This 378×504 pixel image corresponds to image P12 in Fig. 28. Also, assume that the region of the target subject in the 378×504 pixel image, that is, the crop range indicated by the range information, is a region of 100×56 pixels.

[0403] 29, in the crop processing unit 91, a 100×56 pixel area including the subject of interest, which is indicated by the range information, is cropped from the input image of 378×504 pixels, and the cropped image of 100×56 pixels is supplied to the subsequent defect correction unit 92. This 100×56 pixel area corresponds to image P13 in FIG.

[0404] 27, in the telephoto camera 21-2, which is a sub-camera, the image (image data) obtained by the crop processing by the crop processing unit 91 is processed by the defect correction unit 92, the clamp processing unit 111, and the detection value processing unit 54-2. As a result, detection information (phase difference information) is obtained that targets only the area of ​​the target subject.

[0405] In the example shown in Figure 27 above, by cropping (cutting out) the area of ​​the subject of interest, the area processed by the sub-camera detection pipeline 53 and detection value processing unit 54 is limited to the crop range (the range of the subject of interest).

[0406] This reduces the amount of data to be processed within the image sensor 32, thereby enabling high-speed readout, i.e., faster generation of detection information, and lower power consumption. In particular, driving in such a phase difference detection mode with a cropped angle of view can be very effective when all pixels are operating as phase difference pixels.

[0407] Furthermore, as in the first embodiment, low power consumption can be achieved while maintaining the number of time-wise samplings of detection information in the sub-camera equivalent to that of the main camera, and by limiting the information output from each image sensor 32 to the bare minimum necessary, it is also possible to reduce the operating power of the AP 22 and speed up its operation.

[0408] Ninth Embodiment Example of Driving Image Processing Device When the range of phase difference detection in phase difference detection mode is set to the range of a target subject, the area to be subjected to AD conversion may be adaptively changed instead of cropping (digital cropping) the image output from the pixel unit 41.

[0409] An example of driving the image processing device 11 when adaptively changing the area to be subjected to AD conversion will be described with reference to FIG.

[0410] In the example of Figure 30, the wide-angle camera 21-1 is the main camera, and the telephoto camera 21-2 is the sub-camera. The wide-angle camera 21-1, which is the main camera, is driven in the viewing mode, and the telephoto camera 21-2, which is the sub-camera, is driven in the phase difference detection mode. Driving in the viewing mode is the same as that described with reference to Figure 3.

[0411] As in the example of Figure 27, the image processing unit 64 detects the area of ​​the subject of interest from the display image supplied from the main camera image pipeline 52, and supplies coordinate information indicating the detection result to the MCMU 62.

[0412] In addition, the MCMU 62 determines whether or not there is a subject of interest based on the coordinate information supplied from the image processing unit 64, determines the drive mode of each camera 21 depending on the determination result and the shooting magnification, and supplies a sensor mode signal to the MCCU 51 of each camera 21.

[0413] In the example of Fig. 30, a sensor mode signal indicating the Viewing mode is supplied to the MCCU 51-1 of the main camera, and a sensor mode signal indicating the phase difference detection mode of the crop angle of view is supplied to the MCCU 51-2 of the sub-camera, similar to the example of Fig. 27. In particular, the sensor mode signal indicating the phase difference detection mode includes information indicating the area of ​​the target subject.

[0414] The MCCU 51-2 of the sub camera generates a control signal Low_Power_en=High in response to the supplied sensor mode signal and supplies it to the pixel unit 41-2, and also generates a control signal ipipe_en=Low and supplies it to the image processing unit 43-2.

[0415] At this time, the MCCU 51-2 controls the operation of the pixel unit 41-2 based on information indicating the area of ​​the target subject included in the sensor mode signal, so that only signals output from phase difference pixels within the area of ​​the pixel unit 41-2 corresponding to the target subject are AD converted. In other words, the MCCU 51-2 specifies the area (range) corresponding to the target subject as the range to be subjected to AD conversion, and the pixel unit 41-2 performs AD conversion on the range specified by the MCCU 51-2. In this case, the AD conversion unit 133 to which signals from the phase difference pixels are not input is put into a sleep state.

[0416] The MCCU 51-2 specifies the area of ​​the target subject by supplying the pixel unit 41-2 with range information indicating the coordinates of the upper left vertex position of the area of ​​the target subject and the width and height of the area of ​​the target subject, as in the example of Figure 27.

[0417] Image data consisting only of pixel data of phase difference pixels within the area of ​​the target subject is supplied from the pixel section 41-2 to the image processing section 43-2.

[0418] For example, as shown in FIG. 31, assume that in a predetermined (n-1)th frame, a display image P21 is captured by the main camera, and an image P22 made up of pixel data of phase difference pixels is captured by the sub camera.

[0419] In this example, the image processing unit 64 detects the subject of interest in area R91 of the display image P21, and coordinate information indicating the area R91 containing the subject of interest as a detection result is supplied to the MCMU 62. Therefore, the MCMU 62 supplies the MCCU 51 of the sub-camera with a sensor mode signal indicating the phase difference detection mode and including information indicating area R91.

[0420] Therefore, in the next n-th frame, AD conversion is performed in the pixel section 41 of the sub-camera only on the region R92 of the target object corresponding to the region R91.

[0421] 31, an image P23 is an image obtained at the full angle of view, and is an image made up of pixel data of all phase-contrast images.

[0422] In the nth frame, the phase difference detection mode is set at a crop angle of view, so only the signals output from the phase difference pixels within the area of ​​the target subject are AD converted, and the image output from the pixel unit 41 is an image of the area R92 of the target subject in image P23.

[0423] Note that, here, an example will be described in which signals obtained by photoelectric conversion from all phase difference pixels are output in the pixel unit 41, and among these signals, only signals from phase difference pixels within the area of ​​the target subject are AD converted.

[0424] However, the present invention is not limited to this, and it is also possible that, among all the phase difference pixels, signals obtained by photoelectric conversion are output only from phase difference pixels that are within the region of the target subject, and these signals are subjected to AD conversion. Even in such a case, the output of the pixel unit 41 becomes an image made up of pixel data of the phase difference pixels that are within the region of the target subject.

[0425] In this way, by subjecting only the area of ​​the target subject to AD conversion and output of signals from pixels, the AD conversion unit 133, to which signals from the phase difference pixels are not input, can be put into a sleep state, thereby reducing power consumption.

[0426] Returning to the explanation of Fig. 30, in the image processing unit 43-2 of the sub-camera, the operation of the image pipeline 52-2 is stopped based on the control signal ipipe_en = Low, and various processes are performed in the detection pipeline 53-2 and the detection value processing unit 54-2, and phase difference information is generated as detection information. In this case, unlike the example of Fig. 27, the image input from the pixel unit 41-2 to the image processing unit 43-2 has already been cropped to the area of ​​the subject of interest, and therefore no particular cropping is performed in the crop processing unit 91.

[0427] It may be difficult to perform AD conversion on only the range of the target subject due to limitations of the image sensor 32. In such a case, it is possible to combine the cropping (digital cropping) of the range of the target subject described in the eighth embodiment with the AD conversion of the specified range performed in this embodiment.

[0428] As a specific example, there may be a case where the area to be subjected to AD conversion can only be specified in units of pixel rows in the pixel unit 41. In such a case, for example, the area of ​​the subject of interest, i.e., the range to be subjected to AD conversion, can be specified in units of pixel rows in the pixel unit 41, and the area to be finally cut out can be cropped as the area of ​​the subject of interest in the crop processing unit 91 at the subsequent stage.

[0429] As described above, in the example of FIG. 30, the range to be subjected to AD conversion is adaptively changed according to the position of the target subject in the display image.

[0430] By limiting the range (area) that is the target of AD conversion in the pixel unit 41 that is driven in the phase difference detection mode, the number of AD conversion units 133 that can be put into a sleep state can be increased, thereby achieving further reduction in power consumption compared to the example in FIG. 27 . Furthermore, since the amount of data to be processed is reduced, processing speed can also be increased. In particular, the technique of limiting the range that is the target of AD conversion can be significantly effective when all pixels are operating as phase difference pixels.

[0431] Furthermore, as in the first embodiment, low power consumption can be achieved while maintaining the number of time-wise samplings of detection information in the sub-camera equivalent to that of the main camera, and by limiting the information output from each image sensor 32 to the bare minimum necessary, it is also possible to reduce the operating power of the AP 22 and speed up its operation.

[0432] Tenth embodiment Example of configuration of image processing device When changing the angle of view of an image for phase difference detection in the sub-camera depending on whether or not a target subject is present, information obtained by an external sensor provided outside the image sensor 32 may be used.

[0433] In such a case, the image processing device 11 is configured as shown in FIG.

[0434] The configuration of the image processing device 11 shown in FIG. 32 is the same as that of the image processing device 11 shown in FIG. 1, except that an external sensor 441 is newly provided.

[0435] The external sensor 441 is a sensor different from the image sensor 32, and may be, for example, a distance measurement sensor or an EVS (Event-based Vision Sensor) that is provided inside the image processing device 11 or attached outside the image processing device 11.

[0436] The external sensor 441 detects a subject of interest around the image processing device 11, and supplies subject of interest information indicating the detection result to the MCMU 62. As an example, the external sensor 441 detects a moving subject as a subject of interest.

[0437] The MCMU 62 determines whether or not a target subject is present based on target subject information supplied from the external sensor 441 in the same manner as in FIG.

[0438] The MCMU 62 then determines the drive mode of each camera 21 according to the result of determining whether or not a target subject is present and the shooting magnification, and supplies a sensor mode signal to the MCCU 51 of each camera 21 .

[0439] In the example of Fig. 32, similarly to the example of Fig. 27, a sensor mode signal indicating the Viewing mode is supplied to the MCCU 51-1 of the main camera, and a sensor mode signal indicating the phase difference detection mode of the crop angle of view is supplied to the MCCU 51-2 of the sub-camera. In particular, the sensor mode signal indicating the phase difference detection mode includes information indicating the area of ​​the target subject generated based on the target subject information.

[0440] The driving of the main camera in the viewing mode and the driving of the sub-camera in the phase difference detection mode with a cropped angle of view are the same as in the example of FIG.

[0441] Therefore, image data made up of pixel data of all phase difference pixels is output from the pixel unit 41-2 of the telephoto side camera 21-2, which is the sub-camera. In this case, the AD conversion unit 133 to which signals from the phase difference pixels are not input is put into a dormant state.

[0442] Furthermore, in the image processing unit 43-2, the operation of the image pipeline 52-2 is stopped, and the crop processing unit 91 performs crop processing to cut out the range (area) indicated by the range information supplied from the MCCU 51-2 from the image based on the image data output from the pixel unit 41-2. As a result, the image processing unit 43-2 outputs detection information (phase difference information) targeting only the area of ​​the target subject.

[0443] 32, the presence or absence of a subject of interest and the area of ​​the subject of interest are detected using external sensor 441. Furthermore, by cropping the area of ​​the subject of interest, the area processed by detection pipeline 53 and detection value processing unit 54 of the sub-camera is limited to the crop range (the area of ​​the subject of interest).

[0444] In particular, in this example, by using the sensing results (detection results) of the external sensor 441, it is possible to input from the external sensor 441 to the MCMU 62 asynchronously with the image sensor 32. This allows the results of determining the drive method based on the presence or absence of a target subject to be reflected at high speed.

[0445] When determining the drive mode based on information obtained by the image sensors 32, it is assumed that the image sensors 32 are synchronized, so a delay of one frame or more occurs before the drive mode determination result is reflected. In contrast, if the external sensor 441 operates asynchronously with the image sensors 32, for example, and the external sensor 441 operates at a speed equal to or faster than the frame rate of the image sensors 32, the drive mode determination result can be reflected with a delay of one frame or less.

[0446] Furthermore, as in the first embodiment, low power consumption can be achieved while maintaining the number of time-wise samplings of detection information in the sub-camera equivalent to that of the main camera, and by limiting the information output from each image sensor 32 to the bare minimum necessary, it is also possible to reduce the operating power of the AP 22 and speed up its operation.

[0447] Even when only the area corresponding to the target subject is subjected to AD conversion as in the ninth embodiment, the target subject may be detected by the external sensor 441 .

[0448] Eleventh embodiment Driving example of image processing device The driving mode of the camera 21 (image sensor 32) may include a color detection mode in which color detection processing for AWB (Auto White Balance) control (white balance control) is performed. In the color detection mode, information about each color of an image captured by the camera 21 is generated as detection information.

[0449] An example of driving in the color detection mode will be described with reference to FIG.

[0450] In the example of FIG. 33, the wide-angle camera 21-1 is the main camera, and the telephoto camera 21-2 is the sub-camera.

[0451] The wide-angle camera 21-1, which is the main camera, is driven in the viewing mode, and the telephoto camera 21-2, which is the sub-camera, is driven in the color detection mode. Note that the driving in the viewing mode is the same as that described with reference to FIG. 3.

[0452] In this case, the MCMU 62 supplies a sensor mode signal indicating the color detection mode to the MCCU 51-2 of the telephoto camera 21-2, which is a sub-camera.

[0453] In response to the sensor mode signal supplied from the MCMU 62, the MCCU 51-2 supplies a control signal Low_Power_en=High to the pixel unit 41-2 to instruct it to add up pixels of the same color, and also supplies a control signal ipipe_en=Low and a control signal AWB_en=High to the image processing unit 43-2. ​​The control signal AWB_en=High is a signal that instructs the generation of information about each color of the image obtained by capturing with the image sensor 32 (hereinafter also referred to as color detection information) as detection information in color detection mode.

[0454] The pixel unit 41-2 performs binning readout, which is a drive that adds and reads out signals output from multiple adjacent normal pixels of the same color, in accordance with the control signal Low_Power_en=High and an instruction to add pixels of the same color from the MCCU 51-2.

[0455] In this example, same-color pixel 2×2 binning is performed in which signals from four adjacent normal pixels of the same color (pixels provided with color filters of the same color) are added together and read out as a signal for each color.

[0456] Note that reading of signals from the phase difference pixels may or may not be performed depending on the type of the phase difference pixels.

[0457] Addition of signals from pixels of the same color can be achieved by, for example, providing a switch between vertical signal lines as appropriate, as in the case described with reference to Fig. 21. In this case, if there is an AD conversion unit 133 to which no signal is input from a pixel, that AD conversion unit 133 is put into a dormant state.

[0458] For example, image data having a number of pixels less than the total number of pixels, such as 1512×2016 pixels, is output (supplied) from the pixel unit 41-2 to the image processing unit 43-2.

[0459] In addition, the image processing unit 43-2 stops the operation of the image pipeline 52 by stopping the supply of the clock signal in response to the control signal ipipe_en=Low from the MCCU 51-2, and operates only the detection pipeline 53-2.

[0460] Furthermore, the detection value processing unit 54-2 generates color detection information as detection information based on the image data supplied from the detection pipeline 53-2 (clamp processing unit 111) in response to the control signal AWB_en = High supplied from the MCCU 51-2, and supplies it to the detection information processing unit 63.

[0461] As an example, the detection value processing unit 54-2 extracts pixel data of pixels of the same color from the image data supplied from the detection pipeline 53-2, and sets the color image of each color consisting of pixel data of pixels of the same color as color detection information.

[0462] Specifically, for example, if the image data is composed of pixel data of R pixels, pixel data of G pixels, and pixel data of B pixels, the R image (R Plane), G image (G Plane), and B image (B Plane) are used as color detection information. For example, the R image is an image made up of R pixels.

[0463] The color detection information is not limited to color images and may be any information relating to each color.

[0464] For example, the integral value of the pixel values ​​(pixel data values) of each color may be calculated for each predetermined region (color detection frame) in the image, and the integral value for each color of each color detection frame may be used as the color detection information, or the ratio of the integral values ​​for each color, i.e., R / G and B / G, may be used as the color detection information. R / G is the integral value of the R pixel divided by the integral value of the G pixel, and B / G is the integral value of the B pixel divided by the integral value of the G pixel.

[0465] In the example of Figure 33, color detection information is generated as detection information at the same frame rate as the display image and detection information obtained by the main camera, for example 30 fps, and is supplied from the detection value processing unit 54-2 to the detection information processing unit 63.

[0466] The detection information processing unit 63 calculates color information as appropriate based on the color detection information as detection information supplied from the detection value processing unit 54-2, and supplies the color information to the image processing unit 64. For example, the color information may be the color detection information itself, or may be an integral value for each color or a ratio of the integral values ​​for each color (R / G and B / G) calculated from a color image as the color detection information.

[0467] The image processing unit 64 performs white balance adjustment processing on the display image (wide-angle image) supplied from the wide-angle camera 21-1, which is the main camera, as appropriate, based on the color information supplied from the detection information processing unit 63. For example, the white balance adjustment processing also includes AWB calibration, which corrects discrepancies in white balance between the main camera and the sub-camera. The display image that has been subjected to various image processing such as white balance adjustment and demosaic processing is supplied to the display 23 and displayed.

[0468] As described above, in the example of FIG. 33, when driven in the color detection mode, addition of pixels of the same color in the pixel section 41-2 and operation of the image pipeline 52-2 are stopped.

[0469] In this color detection mode, color detection information can be obtained for color detection, and AWB calibration between the main camera and sub camera can be performed with low power consumption. For color detection purposes, accuracy does not decrease even if the number of pixels of the same color is increased to a certain extent, so power consumption can be further reduced by increasing the number of pixels to be added, for example, from 4x4 pixels to 64x64 pixels.

[0470] Furthermore, as in the first embodiment, low power consumption can be achieved while maintaining the number of time-wise samplings of detection information in the sub-camera equivalent to that of the main camera, and by limiting the information output from each image sensor 32 to the bare minimum necessary, it is also possible to reduce the operating power of the AP 22 and speed up its operation.

[0471] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0472] FIG. 34 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0473] In the computer, a CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .

[0474] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a recording unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0475] The input unit 506 includes a keyboard, a mouse, a microphone, an image sensor, etc. The output unit 507 includes a display, a speaker, etc. The recording unit 508 includes a hard disk, a non-volatile memory, etc. The communication unit 509 includes a network interface, etc. The drive 510 drives a removable recording medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0476] In a computer configured as described above, the CPU 501 loads a program recorded in the recording unit 508, for example, into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program, thereby performing the above-described series of processes.

[0477] The program executed by the computer (CPU 501) can be provided by being recorded on a removable recording medium 511 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0478] In a computer, a program can be installed in the recording unit 508 via the input / output interface 505 by inserting a removable recording medium 511 into the drive 510. The program can also be received by the communication unit 509 via a wired or wireless transmission medium and installed in the recording unit 508. Alternatively, the program can be installed in the ROM 502 or the recording unit 508 in advance.

[0479] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0480] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0481] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0482] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0483] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0484] Furthermore, the present technology can also be configured as follows.

[0485] (1) An image processing device comprising: a plurality of image sensors; and a control unit that determines, for each image sensor, a drive mode for the image sensor from a plurality of drive modes including a first mode for outputting an image and a second mode for performing only detection-related processing based on at least one of shooting conditions and information on the surrounding environment, wherein the image sensor stops operation of an image pipeline made up of processing blocks that perform processing for generating the image to be output when driven in the second mode. (2) The image processing device described in (1), wherein the image sensor has a detection value processing unit that generates detection information indicating a detection result based on pixel data of a plurality of pixels obtained by shooting. (3) The image processing device described in (2), wherein the image sensor outputs only the detection information when driven in the second mode. (4) The image processing device described in any one of (1) to (3), wherein the second mode is a phase difference detection mode that performs processing related to phase difference detection, and the image sensor performs AD conversion only on signals output from phase difference pixels when driven in the phase difference detection mode. (5) The image processing device according to (4), wherein, when driven in the phase difference detection mode, the image sensor puts an AD conversion unit to which no signal output from the phase difference pixel is input into a sleep state. (6) The image processing device according to (4) or (5), wherein the control unit determines a pixel thinning rate in the image sensor driven in the phase difference detection mode based on phase difference information indicating a result of phase difference detection, and when driven in the phase difference detection mode, the image sensor generates image data made up of pixel data of the number of phase difference pixels determined by the thinning rate. (7) The image processing device according to any one of (1) to (3), wherein the second mode is a luminance detection mode that performs processing related to luminance detection for AE control, and when driven in the luminance detection mode, the image sensor performs driving to add and read out signals output from a plurality of pixels including pixels of different colors.(8) The image processing device according to (7), wherein the image sensor adds signals output from a plurality of pixels, including pixels of different colors, on a vertical signal line when driven in the luminance detection mode. (9) The image processing device according to (8), wherein the control unit sets the drive mode of all of the image sensors to the luminance detection mode immediately after startup of the image processing device. (10) The image processing device according to any one of (1) to (3), wherein the image sensor reduces a power supply voltage of an analog circuit in the image sensor when driven in the second mode and the dynamic range of an image obtained by the image sensor driven in the first mode is equal to or less than a predetermined value. (11) The image processing device according to any one of (1) to (3), wherein the image sensor changes the number of bits during AD conversion so as to reduce the number of bits of pixel data obtained by AD conversion of signals output from pixels when the ambient brightness of the image processing device is equal to or greater than a predetermined value when driven in the second mode. (12) The image processing device according to any one of (1) to (3), wherein the image sensor, when driven in the second mode, adds capacitance to a vertical signal line provided between a pixel and an AD conversion unit when the brightness of the surroundings of the image processing device is equal to or greater than a predetermined value. (13) The image processing device according to (12), wherein the control unit determines whether the brightness of the surroundings of the image processing device is equal to or greater than the predetermined value based on an image obtained by the image sensor driven in the first mode or a detection result of the brightness of the surroundings of the image processing device obtained by a sensor different from the image sensor. (14) The image processing device according to (2), wherein, when a target subject is present, the control unit causes the image sensor driven in the second mode to generate the detection information targeting the area of ​​the target subject. (15) The image processing device according to (14), wherein the image sensor performs cropping as processing to generate the detection information targeting the area of ​​the target subject.(16) The image processing device according to (14) or (15), wherein the image sensor performs AD conversion on only signals output from pixels in a region of the image sensor corresponding to the target subject as a process for generating the detection information targeted for the region of the target subject. (17) The image processing device according to any one of (14) to (16), wherein the control unit determines whether the target subject is present based on an image obtained by the image sensor driven in the first mode, an input operation by a user, or a detection result of the target subject by a sensor different from the image sensor. (18) The image processing device according to any one of (1) to (3), wherein the second mode is a color detection mode that performs processing related to color detection for AWB control, and when driven in the color detection mode, the image sensor performs driving to add and read out signals output from multiple pixels of the same color. (19) The image processing device according to (18), wherein, when the image sensor is driven in the color detection mode, the image sensor generates information about each color of an image obtained by capturing an image with the image sensor as detection information indicating a detection result. (20) The image processing device according to any one of (1) to (6), wherein the control unit is provided within the image sensor. (21) The image processing device according to (20), wherein the control unit performs AF control based on detection information indicating a detection result obtained with the image sensor. (22) The image processing device according to (21), wherein the control unit performs calibration for AF control based on the detection information obtained with the image sensor and the detection information obtained with another image sensor, and supplies a correction value according to the calibration result to the other image sensor. (23) The image processing device according to any one of (1) to (6), wherein, immediately after startup of the image processing device, the control unit sets the drive mode of all the image sensors to the second mode.(24) The image processing device according to (4), wherein, when it is determined that the AF accuracy is insufficient based on phase difference information indicating a result of phase difference detection or information related to ambient brightness, the control unit sets the drive modes of all of the image sensors to the first mode and switches the AF method to a method based on the image. (25) A driving method in which an image processing device having a plurality of image sensors determines, for each of the image sensors, a drive mode from among a plurality of drive modes including a first mode for outputting an image and a second mode for performing only processing related to detection, based on at least one of shooting conditions and information related to the ambient environment, and when the image sensor is driven in the second mode, the image processing device stops operation of an image pipeline made up of a processing block that performs processing to generate the image to be output. (26) A recording medium having a computer-readable program recorded thereon, which causes a computer that controls an image processing device having a plurality of image sensors to execute processing including the steps of: determining, for each image sensor, a drive mode for the image sensor from a plurality of drive modes including a first mode for outputting an image and a second mode for performing only detection-related processing, based on at least one of shooting conditions and information about the surrounding environment; and, when the image sensor is driven in the second mode, stopping operation of an image pipeline consisting of processing blocks that perform processing to generate the image to be output.

[0486] 11 Image processing device, 21-1, 21-2, 21 Camera, 22 AP, 32-1, 32-2, 32 Image sensor, 41-1, 41-2, 41 Pixel unit, 42-1, 42-2, 42 Control unit, 43-1, 43-2, 43 Image processing unit, 51-1, 51-2, 51 MCCU, 52-1, 52-2, 52 Image pipeline, 53-1, 53-2, 53 Detection pipeline, 54-1, 54-2, 54 Detection value processing unit, 62 MCMU, 63 Detection information processing unit, 64 Image processing unit

Claims

1. An image processing device comprising: a plurality of image sensors; and a control unit that determines, for each image sensor, a drive mode for the image sensor from a plurality of drive modes including a first mode for outputting an image and a second mode for performing only detection processing, based on at least one of shooting conditions and information on the surrounding environment; wherein, when the image sensor is driven in the second mode, the image processing device stops operation of an image pipeline consisting of a processing block that performs processing to generate the image to be output.

2. The image processing device according to claim 1, wherein the image sensor has a detection value processing section that generates detection information indicating the detection result based on pixel data of a plurality of pixels obtained by photographing.

3. The image processing device according to claim 2, wherein the image sensor outputs only the detection information when driven in the second mode.

4. The image processing device according to claim 1, wherein the second mode is a phase difference detection mode that performs processing related to phase difference detection, and the image sensor, when driven in the phase difference detection mode, performs AD conversion only on signals output from phase difference pixels.

5. The image processing device according to claim 4, wherein when the image sensor is driven in the phase difference detection mode, an AD conversion unit to which a signal output from the phase difference pixel is not input is put into a sleep state.

6. The image processing device according to claim 4, wherein the control unit determines a pixel thinning rate in the image sensor that operates in the phase difference detection mode based on phase difference information indicating the result of phase difference detection, and the image sensor, when operating in the phase difference detection mode, generates image data consisting of pixel data for the number of phase difference pixels determined by the thinning rate.

7. The image processing device according to claim 1, wherein the second mode is a luminance detection mode that performs processing related to luminance detection for AE control, and when the image sensor is driven in the luminance detection mode, the image sensor adds and reads out signals output from a plurality of pixels, including pixels of different colors.

8. The image processing device according to claim 7, wherein the image sensor adds, on a vertical signal line, signals output from a plurality of pixels, including pixels of different colors, when driven in the luminance detection mode.

9. The image processing device according to claim 8, wherein the control unit sets the drive mode of all of the image sensors to the luminance detection mode immediately after the image processing device has been started up.

10. The image processing device according to claim 1, wherein when the image sensor is driven in the second mode, if the dynamic range of the image obtained by the image sensor driven in the first mode is equal to or less than a predetermined value, the power supply voltage of the analog circuit within the image sensor is reduced.

11. The image processing device according to claim 1, wherein when the image sensor is driven in the second mode, if the ambient brightness of the image processing device is equal to or greater than a predetermined value, the number of bits during the AD conversion is changed so that the number of bits of pixel data obtained by AD conversion of the signal output from the pixel is reduced.

12. The image processing device according to claim 1, wherein, when the image sensor is driven in the second mode, if the brightness of the surroundings of the image processing device is equal to or greater than a predetermined value, a capacitance is added to a vertical signal line provided between a pixel and an AD conversion unit.

13. The image processing device according to claim 12, wherein the control unit determines whether the brightness of the surroundings of the image processing device is equal to or greater than the predetermined value based on an image obtained by the image sensor operating in the first mode or a detection result of the brightness of the surroundings of the image processing device obtained by a sensor other than the image sensor.

14. The image processing device according to claim 2, wherein, when a target subject is present, the control unit causes the image sensor, driven in the second mode, to generate the detection information targeting the area of the target subject.

15. The image processing device according to claim 14, wherein the image sensor performs cropping as a process for generating the detection information targeted at the area of the target subject.

16. The image processing device according to claim 14, wherein the image sensor performs AD conversion on only signals output from pixels within an area of the image sensor corresponding to the subject of interest as a process for generating the detection information targeted at the area of the subject of interest.

17. The image processing device according to claim 14, wherein the control unit determines whether the target subject is present based on an image obtained by the image sensor operating in the first mode, an input operation by a user, or a detection result of the target subject by a sensor other than the image sensor.

18. An image processing device as described in claim 1, wherein the second mode is a color detection mode that performs processing related to color detection for AWB control, and when operating in the color detection mode, the image sensor operates to add and read out signals output from multiple pixels of the same color.

19. The image processing device according to claim 18, wherein when the image sensor is operated in the color detection mode, the image sensor generates information about each color of an image obtained by capturing an image with the image sensor as detection information indicating the detection result.

20. The image processing device according to claim 1, wherein the control unit is provided within the image sensor.

21. The image processing device according to claim 20, wherein the control unit performs AF control based on detection information indicating the detection result obtained by the image sensor.

22. The image processing device according to claim 21, wherein the control unit performs calibration for AF control based on the detection information obtained by the image sensor and the detection information obtained by the other image sensor, and supplies a correction value according to the result of the calibration to the other image sensor.

23. The image processing device according to claim 1, wherein the control unit sets the drive mode of all of the image sensors to the second mode immediately after the image processing device is started up.

24. The image processing device of claim 4, wherein when the control unit determines that the AF accuracy is insufficient based on phase difference information indicating the results of phase difference detection or information regarding the ambient brightness, the control unit sets the drive mode of all the image sensors to the first mode and switches the AF method to a method based on the image.

25. A driving method in which an image processing device having a plurality of image sensors determines, for each of the image sensors, a driving mode for the image sensors from among a plurality of driving modes including a first mode for outputting an image and a second mode for performing only detection-related processing, based on at least one of shooting conditions and information on the surrounding environment, and when the image sensors are driven in the second mode, the image processing device stops operation of an image pipeline consisting of a processing block that performs processing to generate the image to be output.

26. A recording medium having recorded thereon a computer-readable program that causes a computer that controls an image processing device having multiple image sensors to execute processing including the steps of: determining, for each image sensor, a drive mode for the image sensor from among a plurality of drive modes including a first mode for outputting an image and a second mode for performing only detection-related processing, based on at least one of shooting conditions and information about the surrounding environment; and, when the image sensor is driven in the second mode, halting operation of an image pipeline consisting of processing blocks that perform processing to generate the image to be output.

Images