Image pickup device, image data processing method for image pickup device, and program

Abstract

To provide an imaging apparatus, an image data processing method of an imaging apparatus, and a program, capable of achieving high-speed image processing as compared to a case where image data is output from an imaging element to a plurality of signal processing sections by only an output line of one system.SOLUTION: An imaging apparatus includes: a storage section that stores captured image data obtained by imaging a subject by an imaging element and is incorporated in the imaging element; an output section that is incorporated in the imaging element; and a plurality of signal processing sections that are disposed outside the imaging element. The output section includes a plurality of output lines each disposed in correspondence with each of the plurality of signal processing sections, and outputs each of a plurality of pieces of image data into which the captured image data stored in the storage section is divided, to a corresponding signal processing section among the plurality of signal processing sections from the plurality of output lines, and any of the plurality of signal processing sections combines the plurality of pieces of image data.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The technology disclosed herein relates to an imaging device, an image data processing method for the imaging device, and a program. [Background technology]

[0002] Japanese Patent Application Laid-Open No. 2016-158294 discloses an electronic device including an imaging element, an image processing unit, and a control unit. The imaging element has a first imaging region and a second imaging region. In the imaging element, pixels are arranged in the first imaging region and the second imaging region. The imaging element performs imaging in the first imaging region under a first imaging condition, and performs imaging in the second imaging region under a second imaging condition different from the first imaging condition. Image data consisting of pixel signals of the first imaging region and image data consisting of pixel signals of the second imaging region are output to a downstream image processing unit via one output line.

[0003] The image processing unit generates first image data by performing various image processing on image data consisting of pixel signals of the first imaging region, and generates second image data by performing various image processing on image data consisting of pixel signals of the second imaging region. The control unit causes the display unit to display a live view image or a still image obtained by combining a first image represented by the first image data and a second image represented by the second image data. Summary of the Invention

[0004] One embodiment of the technology disclosed herein provides an imaging device, an image data processing method for the imaging device, and a program that can achieve faster image processing than when image data is output from an imaging element to multiple signal processing units using only a single output line. [Means for solving the problem]

[0005] The imaging device according to the first aspect of the technique of the present disclosure is an imaging device equipped with an imaging element, which stores captured image data obtained by capturing an image of a subject with the imaging element, and includes a memory unit built in the imaging element, an output unit built in the imaging element, and a plurality of signal processing units provided outside the imaging element, and the output unit has a plurality of output lines provided corresponding to each of the plurality of signal processing units, and outputs each of the plurality of image data into which the captured image data stored in the memory unit is divided from the plurality of output lines to a corresponding one of the plurality of signal processing units, and one of the plurality of signal processing units synthesizes the plurality of image data. This makes it possible to realize high-speed image processing compared to a case in which image data is output from the imaging element to the plurality of signal processing units using only one output line.

[0006] In the imaging device according to the second aspect of the disclosed technique, each of the plurality of image data is image data representing an image having an overlapping area between adjacent images among images based on each of the plurality of image data, which suppresses the emergence of a boundary area between the two images compared to a case where the captured image is simply divided into two and the two images are joined together.

[0007] In the imaging device according to the third aspect of the technique of the present disclosure, the multiple images are divided into a designated image and an image different from the designated image, thereby suppressing the appearance of a boundary area between the designated image and the image different from the designated image.

[0008] In the imaging device according to the fourth aspect of the technique of the present disclosure, a detection unit is further included that detects face image data indicating an image of a face from the captured image data, and the specified image is an image of a face indicated by the face image data detected by the detection unit from the captured image indicated by the captured image data. This makes it possible to suppress the boundary area between the image including the face image and the image not including the face image from becoming apparent.

[0009] In the imaging device according to the fifth aspect of the technique of the present disclosure, the captured image data is divided in a different manner between the recording imaging mode and the display moving image imaging mode, thereby making it possible to achieve a different balance between image quality, power consumption, and processing speed between the recording imaging mode and the display moving image imaging mode.

[0010] In the imaging device according to the sixth aspect of the technique of the present disclosure, the captured image data is divided into a plurality of overlapping image data as a plurality of image data in the recording imaging mode, and the captured image data is divided by line in the display moving image imaging mode. As a result, the image quality can be improved in the recording imaging mode compared to the display moving image imaging mode, and the power consumption can be reduced and the processing speed can be increased in the display moving image imaging mode compared to the recording imaging mode.

[0011] In the imaging device according to the seventh aspect of the technique of the present disclosure, each of the multiple overlapping image data is image data representing an image having an overlapping area between adjacent images among the multiple images. This makes it possible to suppress the boundary area between the two images from becoming apparent, compared to a case where the captured image is simply divided into two halves and the two images are then joined together.

[0012] In the imaging device according to the eighth aspect of the technique of the present disclosure, the recording imaging mode is an operation mode in which the imaging element captures still images, thereby making it possible to achieve a different balance between image quality, power consumption, and processing speed between the operation mode in which still images are captured and the imaging mode for displaying moving images.

[0013] In an imaging device according to a ninth aspect of the technique of the present disclosure, the captured image data is color image data representing a captured color image in which a plurality of primary color pixels are periodically arranged, and the color image data is divided into a plurality of primary color pixel array image data as a plurality of image data, and each of the plurality of primary color pixel array image data is image data representing an image in which a plurality of primary color pixels are periodically arranged. This makes it possible to realize demosaic processing for the plurality of primary color pixels even when the color image data is divided into a plurality of primary color pixel array image data.

[0014] In the imaging device according to the tenth aspect of the disclosed technique, the primary color pixel array image data are a plurality of divided image data obtained by thinning and then dividing the color image data, thereby achieving faster processing than when the plurality of image data obtained by dividing the color image data without thinning it are processed by a plurality of signal processing units.

[0015] In an imaging device according to an eleventh aspect of the technique of the present disclosure, the plurality of divided image data are odd-numbered column image data indicating a set of pixels in odd-numbered columns and even-numbered column image data indicating a set of pixels in even-numbered columns, among thinned image data obtained by thinning color image data by rows. This allows each of the plurality of signal processing units to realize high-speed processing compared to processing image data obtained by irregular division.

[0016] In an imaging device according to a twelfth aspect of the technique of the present disclosure, any one of a plurality of signal processing units performs demosaic processing on composite image data obtained by combining a plurality of image data, thereby making it possible to obtain an image with higher image quality than when the demosaic processing is not performed.

[0017] In the imaging device according to the thirteenth aspect of the technique of the present disclosure, the plurality of image data are a plurality of compressed image data obtained by compressing the captured image data by dividing the captured image data into a plurality of bit ranges. This allows each of the plurality of signal processing units to achieve faster processing than when processing image data obtained by irregularly dividing the image data.

[0018] In the imaging device according to the fourteenth aspect of the technique of the present disclosure, the multiple compressed image data are image data of higher order bits and image data of lower order bits of the captured image data. This allows the image data of the higher order bits to be processed more precisely than the image data of the lower order bits, and the image data of the lower order bits can consume less power and can have a higher processing speed than the image data of the higher order bits.

[0019] In the imaging device according to the fifteenth aspect of the technique of the present disclosure, the imaging element is an imaging element in which at least the photoelectric conversion element and the storage unit are integrated into a single chip, thereby making it possible to improve portability of the imaging element compared to an imaging element in which the photoelectric conversion element and the storage unit are not integrated into a single chip.

[0020] In an imaging device according to a sixteenth aspect of the disclosed technique, the imaging element is a stacked type imaging element in which a storage unit is stacked on a photoelectric conversion element, thereby reducing the load on processing between the photoelectric conversion element and the storage unit compared to when the photoelectric conversion element and the storage unit are not stacked.

[0021] The imaging device according to the seventeenth aspect of the technique of the present disclosure further includes a control unit that controls the display unit to display an image based on the plurality of image data output by the output unit, thereby allowing a user to visually recognize the image based on the plurality of image data output by the output unit.

[0022] An image data processing method according to an eighteenth aspect of the technique of the present disclosure is an image data processing method for an imaging device including an image sensor, a storage unit that stores captured image data obtained by capturing an image of a subject with the image sensor and is built in the image sensor, an output unit that is built in the image sensor, and a plurality of signal processing units that are provided outside the image sensor, the output unit having a plurality of output lines provided corresponding to each of the plurality of signal processing units, outputting each of the plurality of image data into which the captured image data stored in the storage unit is divided from the plurality of output lines to a corresponding one of the plurality of signal processing units, and one of the plurality of signal processing units synthesizing the plurality of image data. This makes it possible to realize high-speed image processing compared to a case where image data is output from the image sensor to the plurality of signal processing units using only one output line.

[0023] A program according to a nineteenth aspect of the technique of the present disclosure is a program for making a computer function as an output unit included in an imaging device that stores an image sensor, imaged image data obtained by imaging a subject with the image sensor, and includes a storage unit built in the image sensor, an output unit built in the image sensor, and a plurality of signal processing units provided outside the image sensor, the output unit having a plurality of output lines provided corresponding to each of the plurality of signal processing units, and outputs each of the plurality of image data into which the imaged image data stored in the storage unit is divided from the plurality of output lines to a corresponding one of the plurality of signal processing units, and one of the plurality of signal processing units synthesizes the plurality of image data. This makes it possible to realize high-speed image processing compared to a case in which image data is output from the image sensor to the plurality of signal processing units using only one output line.

[0024] An imaging device according to a twentieth aspect of the technique of the present disclosure is an imaging device including an imaging element, and stores image data obtained by imaging a subject with the imaging element, and includes a memory built into the imaging element, a processor built into the imaging element, and a plurality of signal processing units provided outside the imaging element, the processor having a plurality of output lines provided corresponding to each of the plurality of signal processing units, and outputs image data from the plurality of output lines to the memory. Each of the plurality of image data into which the captured image data is divided is output to a corresponding one of the plurality of signal processing units, and one of the plurality of signal processing units synthesizes the plurality of image data. This makes it possible to realize high-speed image processing compared to a case where the image data is output from the imaging element to the plurality of signal processing units using only one output line. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is a perspective view showing an external appearance of an imaging device. [Diagram 2] FIG. 2 is a rear view showing the rear side of the imaging device. [Diagram 3] FIG. 1 is a block diagram showing a configuration of an imaging apparatus. [Figure 4]FIG. 2 is a schematic diagram showing the configuration of an imaging element. [Diagram 5] FIG. 2 is a block diagram showing a configuration of an imaging element. [Figure 6] 2 is a block diagram showing a data flow within the imaging device according to the first embodiment. FIG. [Figure 7] 4 is a conceptual diagram showing features of a left image and a right image included in a captured image. FIG. [Figure 8] 2 is a block diagram showing a controller and UI devices included in the imaging apparatus, as well as the peripheral configuration of the controller and UI devices. FIG. [Figure 9] FIG. 2 is a schematic diagram showing the configuration of a hybrid finder. [Figure 10] 4 is a flowchart showing a flow of imaging processing according to the first embodiment. [Figure 11] 5 is a flowchart showing a flow of first signal processing according to the first embodiment. [Figure 12] 6 is a flowchart showing a flow of second signal processing according to the first embodiment. [Figure 13] FIG. 11 is a block diagram showing a data flow within an imaging device according to a second embodiment. [Figure 14A] 11 is an image diagram showing the relationship between a face region image and a background region image. FIG. [Figure 14B] 11 is an image diagram showing the relationship between a face region image, a background region image, and an overlapping region. FIG. [Figure 15] 10 is a flowchart showing a flow of imaging processing according to the second embodiment. [Figure 16] 10 is a flowchart showing a flow of first signal processing according to the second embodiment. [Figure 17] 10 is a flowchart showing a flow of second signal processing according to the second embodiment. [Figure 18] FIG. 11 is a block diagram showing a data flow within an imaging device according to a third embodiment. [Figure 19] 11 is an explanatory diagram illustrating a method for generating vertically thinned image data from captured image data. FIG. [Figure 20]11 is a conceptual diagram showing the relationship between vertically thinned image data, odd-numbered column image data, and even-numbered column image data. FIG. [Figure 21] 13 is a conceptual diagram showing a manner in which vertically thinned image data is separated into odd-numbered column image data and even-numbered column image data. FIG. [Figure 22] 13 is a flowchart showing a flow of imaging processing according to the third embodiment. [Diagram 23] 13 is a flowchart showing a flow of first signal processing according to the third embodiment. [Figure 24] 13 is a flowchart showing a flow of second signal processing according to the third embodiment. [Diagram 25] FIG. 11 is a block diagram showing a data flow within an imaging device according to a third embodiment. [Figure 26] 1 is a block diagram showing a data flow within an imaging device in which a still image imaging mode is set. FIG. [Figure 27] 11 is a block diagram showing a data flow within an imaging device in which a display image imaging mode is set. FIG. [Figure 28] 13 is a flowchart showing a flow of imaging processing according to the fourth embodiment. [Figure 29] FIG. 13 is a block diagram showing a data flow within an imaging device according to a fourth embodiment. [Diagram 30] 11 is an explanatory diagram illustrating a method for generating vertically thinned image data from captured image data in a Bayer array; [Diagram 31] 10 is a conceptual diagram showing the relationship between vertically thinned image data, first horizontally thinned image data, and second horizontally thinned image data. FIG. [Diagram 32] 11 is a block diagram showing a form in which captured image data is separated into higher-order bit data and lower-order bit data and then processed. FIG. [Diagram 33] 11 is a conceptual diagram showing the relationship between an imaging element, a plurality of signal processing units, and a controller when three or more signal processing units are used. [Diagram 34] 10 is a conceptual diagram showing an example of a manner in which an imaging processing program is installed into a computer in an imaging element from a storage medium on which the imaging processing program is stored. FIG. [Diagram 35] FIG. 1 is a block diagram showing an example of a schematic configuration of a smart device incorporating an imaging element according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Hereinafter, an example of an embodiment of an imaging device according to the technique of the present disclosure will be described with reference to the accompanying drawings.

[0027] First, the meanings of terms used in the following description will be explained.

[0028] In the following description, CPU is an abbreviation for "Central Processing Unit." In the following description, RAM is an abbreviation for "Random Access Memory." In the following description, ROM is an abbreviation for "Read Only Memory." In the following description, DRAM is an abbreviation for "Dynamic Random Access Memory." In the following description, SRAM is an abbreviation for "Static Random Access Memory."

[0029] In the following description, IC is an abbreviation for "Integrated Circuit." In the following description, LSI is an abbreviation for "Large-Scale Integration." In the following description, ASIC is an abbreviation for "Application Specific Integrated Circuit." In the following description, PLD is an abbreviation for "Programmable Logic Device." In the following description, FPGA is an abbreviation for "Field-Programmable Gate Array."

[0030] In the following description, SSD is an abbreviation for "Solid State Drive." In the following description, DVD-ROM is an abbreviation for "Digital Versatile Disc Read Only Memory." In the following description, USB is an abbreviation for "Universal Serial Bus." In the following description, HDD is an abbreviation for "Hard Disk Drive." In the following description, EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory."

[0031] In the following description, CCD is an abbreviation for "Charge Coupled Device." In the following description, CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor." In the following description, EL is an abbreviation for "Electro-Luminescence." In the following description, A / D is an abbreviation for "Analog / Digital." In the following description, FIFO is an abbreviation for "First in First out." In the following description, I / F is an abbreviation for "Interface." In the following description, EIS is an abbreviation for "Electronics Image Stabilization." In the following description, AF is an abbreviation for "Auto-Focus." In the following description, AE is an abbreviation for "Automatic Exposure." In the following description, UI is an abbreviation for "User Interface."

[0032] [First embodiment] 1, the imaging device 10 is an interchangeable lens camera. The imaging device 10 is a digital camera that includes an imaging device body 12 and an interchangeable lens 14 that is replaceably attached to the imaging device body 12, and does not include a reflex mirror.

[0033] The imaging device body 12 is also provided with a hybrid finder (registered trademark) 16. The hybrid finder 16 here refers to a finder in which, for example, an optical viewfinder (hereinafter referred to as "OVF") and an electronic viewfinder (hereinafter referred to as "EVF") are selectively used. Note that OVF is an abbreviation for "optical viewfinder." Also, EVF is an abbreviation for "electronic viewfinder."

[0034] A viewfinder switching lever 18 is provided on the front of the imaging device body 12. An optical image viewable in the OVF and a live view image, which is an electronic image viewable in the EVF, can be switched by rotating the viewfinder switching lever 18 in the direction of the arrow SW. The "live view image" referred to here refers to a moving image for display obtained by capturing an image using a photoelectric conversion element 61 (see FIGS. 3 and 4) described below. A live view image is also generally referred to as a through image.

[0035] A release button 20 and a dial 22 are provided on the top surface of the imaging device body 12. The dial 22 is operated when setting the operation mode of the imaging system and the operation mode of the playback system, etc.

[0036] The release button 20 functions as an imaging preparation instruction section and an imaging instruction section, and is capable of detecting two stages of pressing operation: an imaging preparation instruction state and an imaging instruction state. The imaging preparation instruction state refers to a state in which the button is pressed from the standby position to an intermediate position (half-pressed position), for example, and the imaging instruction state refers to a state in which the button is pressed beyond the intermediate position to a final pressed position (fully-pressed position). In the following, the "state in which the button is pressed from the standby position to the half-pressed position" is referred to as the "half-pressed state", and the "state in which the button is pressed from the standby position to the fully-pressed position" is referred to as the "fully-pressed state".

[0037] In the imaging device 10, an imaging mode and a playback mode are selectively set as operation modes in response to a user instruction. The imaging modes are broadly divided into a display moving image imaging mode and a recording imaging mode.

[0038] The display moving image imaging mode is an operation mode in which a live view image based on a plurality of consecutive frames of display image data obtained by continuous imaging is displayed on a first display 32 and / or a second display 86 (see Figs. 8 and 9) described below. The display image data is image data for a live view image, and is generated, for example, by a CPU 46A (see Fig. 8) described below based on captured image data 70 (see Figs. 3 to 7) indicating an image of a subject. The captured image data 70 refers to image data obtained by capturing an image of a subject by an imaging element 44 (see Fig. 3) described below. For ease of explanation, hereinafter, an image indicated by the captured image data 70 will be referred to as a "captured image".

[0039] The recording imaging mode is broadly divided into a still image imaging mode and a moving image imaging mode. The still image imaging mode is an operation mode in which the imaging element 44 (see FIG. 3) captures still images. In the still image imaging mode, a still image obtained by imaging a subject with the imaging device 10 is stored in a specific recording device (for example, a secondary storage device (see FIG. 8)). The moving image imaging mode is an operation mode in which the image sensor 44 (see FIG. 3) captures moving images. In the moving image imaging mode, moving images obtained by imaging a subject with the imaging device 10 are stored in a specific recording device.

[0040] The record imaging mode is an operation mode in which a live view image is displayed on a first display 32 and / or a second display 86, which will be described later, and record image data is recorded in a secondary storage device 80 (see FIG. 8) and / or a memory card, which will be described later, etc. The record image data is broadly divided into still image data and moving image data, and is generated based on captured image data 70 (see FIGS. 3 to 7).

[0041] When the imaging mode is set, the imaging device 10 first enters a display moving image imaging mode. In the display moving image imaging mode, when the release button 20 is pressed, the imaging device 10 transitions from the display moving image imaging mode to a recording imaging mode.

[0042] In the imaging mode, manual focus mode and autofocus mode are selectively set according to the user's instruction. In the autofocus mode, the imaging conditions are adjusted by pressing the release button 20 halfway, and then exposure is performed by continuing to press it fully. In other words, the AE function is activated to set the exposure state by pressing the release button 20 halfway, and then the AF function is activated to control focus, and imaging is performed by pressing the release button 20 fully.

[0043] As an example, as shown in FIG. 2, a touch panel display 26, instruction keys 28, and a viewfinder eyepiece 30 are provided on the rear surface of the imaging device body 12.

[0044] The touch panel display 26 includes a first display 32 and a touch panel 34 (see also FIG. 8). The first display 32 may be a liquid crystal display or an organic EL display.

[0045] The first display 32 displays images, text information, etc. The first display 32 is used to display live view images obtained by continuous imaging when the imaging device 10 is in imaging mode. The first display 32 is also used to display still images obtained by imaging when an instruction to capture a still image is given. Furthermore, the first display 32 is also used to display playback images and menu screens, etc. when the imaging device 10 is in playback mode.

[0046] The touch panel 34 is a transmissive touch panel, and is overlaid on the surface of the display area of ​​the first display 32. The touch panel 34 detects contact with a pointing object such as a finger or a stylus pen, and outputs the detection result to a predetermined output destination such as a CPU 46A (see FIG. 8) described below.

[0047] The instruction keys 28 receive various instructions such as selecting one or more menu items, confirming a selection, erasing a selection, zooming, and frame-by-frame advancement.

[0048] 3, the imaging device 10 includes mounts 36 and 38. The mount 36 is provided on the imaging device body 12. The mount 38 is provided on the interchangeable lens 14 at a position corresponding to the position of the mount 36. The interchangeable lens 14 is attached to the imaging device body 12 in an interchangeable manner by coupling the mount 38 to the mount 36.

[0049] As an example, as shown in FIG. 3, the interchangeable lens 14 includes an imaging lens 40. The lens 40 includes an objective lens 40A, a focus lens 40B, a zoom lens 40C, and an aperture 40D. The objective lens 40A, the focus lens 40B, the zoom lens 40C, and the aperture 40D are arranged in this order along the optical axis L1 from the subject side to the imaging device body 12 side. The focus lens 40B, the zoom lens 40C, and the aperture 40D are operated by receiving power from a driving source (not shown) such as a motor under the control of a CPU 46A (see FIG. 8) described later. That is, the focus lens 40B and the zoom lens 40C move along the optical axis L1 according to the applied power. The aperture 40D adjusts the exposure by operating according to the applied power.

[0050] The imaging device body 12 includes a mechanical shutter 42 and an imaging element 44. The mechanical shutter 42 operates by receiving power from a drive source (not shown) such as a motor under the control of a CPU 46A (see FIG. 8) described below. When the interchangeable lens 14 is attached to the imaging device body 12 via the mounts 36, 38, subject light indicative of a subject passes through the imaging lens 40 and is imaged on a light receiving surface 44A of the imaging element 44 via the mechanical shutter 42.

[0051] The imaging device body 12 includes a controller 46, a UI device 48, a first signal processing unit 50, a second signal processing unit 52, and DRAMs 54 and 56. The first signal processing unit 50 and the second signal processing unit 52 are examples of the "multiple signal processing processors" according to the technology of the present disclosure.

[0052] The controller 46 controls the entire imaging device 10. The UI device 48 is a device that presents information to a user and receives instructions from the user. The UI device 48 is connected to the controller 46, and the controller 46 acquires various information from the UI device 48 and controls the UI device 48.

[0053] The imaging element 44 is connected to the controller 46 via a communication line 57, and under the control of the controller 46, captures an image of a subject and generates captured image data 70. As will be described in detail later, the imaging element 44 separates the generated captured image data 70 into two pieces of image data. In the example shown in Fig. 3, first separated image data 70A is shown as one of the two pieces of image data obtained by separating the captured image data 70, and second separated image data 70B is shown as the other piece of image data.

[0054] The imaging element 44 is connected to a first signal processing unit 50 via a first output line 53, and is connected to a second signal processing unit 52 via a second output line 55. Each of the first signal processing unit 50 and the second signal processing unit 52 is an LSI. In this embodiment, each of the first signal processing unit 50 and the second signal processing unit 52 is realized by an ASIC.

[0055] However, the technology of the present disclosure is not limited to this. For example, a PLD and / or an FPGA may be adopted instead of an ASIC. Also, an ASIC, a PLD, and / or an FPGA may be adopted. Also, a computer including a CPU, a ROM, and a RAM may be adopted. The CPU may be singular or plural. Also, the first signal processing unit 50 and / or the second signal processing unit 52 may be realized by a combination of a hardware configuration and a software configuration.

[0056] The first signal processing unit 50 and the second signal processing unit 52 are connected to each other via a communication line 58. The first signal processing unit 50 is connected to a DRAM 54, and the second signal processing unit 52 is connected to a DRAM 56. The first signal processing unit 50 is connected to the controller 46 via a communication line 60.

[0057] The imaging element 44 outputs the first separated image data 70A to the first signal processing unit 50 via the first output line 53, and outputs the second separated image data 70B to the second signal processing unit 52 via the second output line 55.

[0058] The first signal processing unit 50 performs various signal processing (for example, "specific signal processing" described later) on the inputted first separated image data 70A in cooperation with the DRAM 54. The second signal processing unit 52 performs various signal processing (for example, "specific signal processing" described later) on the inputted second separated image data 70B in cooperation with the DRAM 56. The second signal processing unit 52 then outputs the second separated image data 70B, which has been subjected to the various signal processing, to the first signal processing unit 50 via the communication line 58. The first signal processing unit 50 combines the first separated image data 70A, which has been subjected to the various signal processing, with the second separated image data 70B inputted from the second signal processing unit 52, and outputs the combined data to the controller 46 via the communication line 60.

[0059] The imaging element 44 is an example of a "stacked imaging element" according to the technology of the present disclosure. The imaging element 44 is, for example, a CMOS image sensor. As shown in FIG. 4 as an example, the imaging element 44 has a built-in photoelectric conversion element 61, a processing circuit 62, and a memory 64. The imaging element 44 is an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are integrated into a single chip. That is, the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are integrated into a single package. In the imaging element 44, the processing circuit 62 and the memory 64 are stacked on the photoelectric conversion element 61. Specifically, the photoelectric conversion element 61 and the processing circuit 62 are electrically connected to each other by conductive bumps (not shown) such as copper, and the processing circuit 62 and the memory 64 are also electrically connected to each other by conductive bumps (not shown) such as copper. The memory 64 is an example of a "storage unit" according to the technology of the present disclosure.

[0060] The processing circuit 62 is, for example, an LSI, and the memory 64 is, for example, a DRAM. However, the technology of the present disclosure is not limited to this, and an SRAM may be adopted as the memory 64 instead of the DRAM.

[0061] The processing circuit 62 is realized by an ASIC, and controls the entire image sensor 44 according to instructions from the controller 46. Although an example in which the processing circuit 62 is realized by an ASIC is given here, the technology of the present disclosure is not limited to this. For example, a PLD and / or an FPGA may be adopted instead of the ASIC. Also, an ASIC, a PLD, and / or an FPGA may be adopted. Also, a computer including a CPU, a ROM, and a RAM may be adopted. The CPU may be singular or multiple. Also, the processing circuit 62 may be realized by a combination of a hardware configuration and a software configuration.

[0062] The photoelectric conversion element 61 has a plurality of photodiodes arranged in a matrix. An example of the plurality of photodiodes is photodiodes for "4896×3265" pixels.

[0063] The photoelectric conversion element 61 is equipped with color filters, and the color filters include a G filter corresponding to G (green), which contributes most to obtaining a luminance signal, an R filter corresponding to R (red), and a B filter corresponding to B (blue). In this embodiment, the G filters, R filters, and B filters are arranged with a predetermined periodicity in the row direction (horizontal direction) and column direction (vertical direction) for the multiple photodiodes of the photoelectric conversion element 61. Therefore, when performing demosaic processing of R, G, and B signals, the imaging device 10 can perform processing according to a repetitive pattern. Note that demosaic processing refers to a process of calculating all color information for each pixel from a mosaic image corresponding to the color filter array of a single-plate color imaging element. For example, in the case of an imaging element consisting of RGB three-color filters, demosaic processing is a process of calculating all color information for each pixel from a mosaic image corresponding to the color filter array of a single-plate color imaging element. This refers to the process of calculating all RGB color information for each pixel from a mosaic image consisting of GB.

[0064] Although a CMOS image sensor is exemplified as the imaging element 44 here, the technology of the present disclosure is not limited to this, and the technology of the present disclosure also applies if the imaging element 44 is, for example, a CCD image sensor.

[0065] The imaging element 44 has a so-called electronic shutter function, and by operating the electronic shutter function under the control of the controller 46, the charge accumulation time of each photodiode in the photoelectric conversion element 61 is controlled. The charge accumulation time refers to the so-called shutter speed.

[0066] In the imaging device 10, imaging for still images and imaging for moving images are performed by a rolling shutter method. Imaging for still images is achieved by activating the electronic shutter function and operating the mechanical shutter 42 (see FIG. 3), and imaging for moving images is achieved by activating the electronic shutter function without operating the mechanical shutter 42. Note that, although the rolling shutter method is exemplified here, the technology of the present disclosure is not limited to this, and a global shutter method may be applied instead of the rolling shutter method.

[0067] The processing circuit 62 reads out captured image data 70 obtained by capturing an image of a subject with the photoelectric conversion element 61. The captured image data 70 is signal charge accumulated in the photoelectric conversion element 61. The processing circuit 62 performs A / D conversion on the captured image data 70 read out from the photoelectric conversion element 61. The processing circuit 62 stores the captured image data 70 obtained by performing A / D conversion on the captured image data 70 in the memory 64.

[0068] The processing circuit 62 acquires the captured image data 70 from the memory 64 and performs various processes on the acquired captured image data 70. The "various processes" referred to here include a process of separating the captured image data 70 into first separated image data 70A and second separated image data 70B, as shown in Fig. 4. The processing circuit 62 outputs the first separated image data 70A to the first signal processing unit 50 via the first output line 53, and outputs the second separated image data 70B to the second signal processing unit 52 via the second output line 55.

[0069] 5, the processing circuit 62 includes a photoelectric conversion element control circuit 62A, a digital processing circuit 62B, an image processing circuit 62C, and an output circuit 62D. The output circuit 62D is an example of an “output section” according to the technology of the present disclosure.

[0070] The photoelectric conversion element control circuit 62A is connected to the photoelectric conversion element 61 and the digital processing circuit 62B. The memory 64 is connected to the digital processing circuit 62B and the image processing circuit 62C. The image processing circuit 62C is connected to the output circuit 62D and the memory 64.

[0071] The output circuit 62D has a first output line 53 and a second output line 55. The first output line 53 corresponds to the first signal processing unit 50 and connects the output circuit 62D and the first signal processing unit 50. The second output line 55 corresponds to the second signal processing unit 52 and connects the output circuit 62D and the second signal processing unit 52.

[0072] The photoelectric conversion element control circuit 62A, under the control of the controller 46, controls the photoelectric conversion element 61 and reads out analog captured image data 70 from the photoelectric conversion element 61. The digital processing circuit 62B performs correlated double sampling signal processing on the analog captured image data 70 read out by the photoelectric conversion element control circuit 62A, and then performs A / D conversion to digitize the analog captured image data 70. The digital processing circuit 62B then stores the digitized captured image data 70 in the memory 64.

[0073] The memory 64 is a memory capable of storing multiple frames of captured image data 70. The memory 64 has a pixel-by-pixel storage area (not shown), and the captured image data 70 is stored in the corresponding storage area of ​​the memory 64 in pixel units by the digital processing circuit 62B.

[0074] The image processing circuit 62C acquires the captured image data 70 from the memory 64, and processes the acquired captured image data 70.

[0075] The image processing circuit 62C performs the above-mentioned various processes on the captured image data 70. The image processing circuit 62C separates the captured image data 70 into first separated image data 70A and second separated image data 70B, and outputs the first separated image data 70A and the second separated image data 70B (see FIGS. 3 and 4) obtained by separation to the output circuit 62D.

[0076] The output circuit 62D outputs the first separated image data 70A input from the image processing circuit 62C to the first signal processing unit 50 via the first output line 53. In addition, the output circuit 62D outputs the second separated image data 70B input from the image processing circuit 62C to the second signal processing unit 52 via the second output line 55.

[0077] The output frame rate of the output circuit 62D is the same as the frame rate used in a device downstream of the image sensor 44. The output frame rate is a frame rate required for the output of the first separated image data 70A and the second separated image data 70B by the output circuit 62D, and is, for example, 60 fps (frames per second). In contrast, the imaging frame rate is a frame rate required for imaging performed by the photoelectric conversion element 61, the photoelectric conversion element control circuit 62A, the digital processing circuit 62B, and the memory 64 in cooperation with each other, and is, for example, 120 fps. Note that the term "imaging" used here refers to a process from when exposure for one frame is started in the photoelectric conversion element 61 to when the captured image data 70 for one frame is stored in the memory 64.

[0078] Here, specific processing contents in the image processing circuit 62C, the output circuit 62D, the first signal processing unit 50, and the second signal processing unit 52 will be described.

[0079] 6, the image processing circuit 62C acquires captured image data 70 from the memory 64, and separates the acquired captured image data 70 into left image data 70A1 and right image data 70B1. The image processing circuit 62C outputs the left image data 70A1 and right image data 70B1 obtained by separating the captured image data 70 to the output circuit 62D.

[0080] The left image data 70A1 is an example of the above-mentioned first separated image data 70A (see FIGS. 3 to 5), and the right image data 70B1 is an example of the above-mentioned second separated image data 70B (see FIGS. 3 to 5).

[0081] The left image data 70A1 is image data showing a left image 70A1a (see FIG. 7), and the right image data 70B1 is image data showing a right image 70B1a (see FIG. 7). As an example, as shown in FIG. 7, the left image 70A1a and the right image 70B1a are a pair of images adjacent to each other in the left-right direction. The left image 70A1a and the right image 70B1a have an overlapping region 71. The overlapping region 71 is a region where the left image 70A1a and the right image 70B1a overlap in the left-right direction RL. The number of pixels in the left-right direction RL of the overlapping region 71 is, for example, several tens of pixels to several hundreds of pixels.

[0082] In the example shown in FIG. 6, the image processing circuit 62C acquires the captured image data 70 from the memory 64, and the acquired captured image data 70 is separated. However, the technology of the present disclosure is not limited to this. In this case, for example, first, the image processing circuit 62C selects the left image data 70A1 and the right image data 70B1 from the captured image data 70 according to a predetermined address in the memory 64. Then, the image processing circuit 62C directly acquires the selected left image data 70A1 and right image data 70B1 from the memory 64. Note that the predetermined address is an acquisition address for the left image data 70A1 and an acquisition address for the right image data 70B1. The acquisition address for the left image data 70A1 and the acquisition address for the right image data 70B1 are determined so that the left image data 70A1 and the right image data 70B1 each include image data indicating the overlapping area 71.

[0083] The first signal processing unit 50 has a buffer 50A, a signal processing circuit 50B, and a receiving circuit 50C. The signal processing circuit 50B is connected to a DRAM 54 and a controller 46. The second signal processing unit 52 has a buffer 52A, a signal processing circuit 52B, and a transmitting circuit 52C. The signal processing circuit 52B is connected to a DRAM 56.

[0084] The output circuit 62D outputs the left side image data 70A1 to the buffer 50A via the first output line 53. The buffer 50A holds the input left side image data 70A1 and outputs it to the signal processing circuit 50B in a FIFO manner. The signal processing circuit 50B stores the left side image data 70A1 input from the buffer 50A in the DRAM 54. The signal processing circuit 50B performs signal processing such as color correction, white balance adjustment, sharpness adjustment, gamma correction, and gradation correction (hereinafter referred to as "specific signal processing") on the left side image data 70A1 stored in the DRAM 54.

[0085] The output circuit 62D outputs the right side image data 70B1 to the buffer 52A via the second output line 55. The buffer 52A holds the input right side image data 70B1 and outputs it to the signal processing circuit 52B in a FIFO manner. The signal processing circuit 52B stores the right side image data 70B1 input from the buffer 52A in the DRAM 56. The signal processing circuit 52B performs specific signal processing on the right side image data 70B1 stored in the DRAM 56.

[0086] The transmission circuit 52C transmits the right side image data 70B1 that has been subjected to specific signal processing by the signal processing circuit 52B to the first signal processing unit 50. In the first signal processing unit 50, the reception circuit 50C receives the right side image data 70B1 transmitted from the transmission circuit 52C.

[0087] The signal processing circuit 50B synthesizes the right side image data 70B1 received by the receiving circuit 50C and the left side image data 70A1 that has been subjected to signal processing, thereby generating synthetic image data 72. The signal processing circuit 50B outputs the synthetic image data 72 obtained by the synthesis to the controller 46 via the communication line 60.

[0088] The composite image data 72 is generated by splicing together the left image data 70A1 and the right image data 70B1. Here, the image data showing the overlapping area 71 (see FIG. 7) in the left image data 70A1 and the image data showing the overlapping area 71 (see FIG. 7) in the right image data 70B1 are averaged. This suppresses the emergence of the boundary area between the two images compared to the case where two images obtained by simply dividing the captured image in the left-right direction RL (see FIG. 7) are spliced ​​together. Note that although the average is exemplified here, the technology of the present disclosure is not limited to this, and replacement may be adopted instead of the average. The term "replacement" here refers to replacing one of the image data showing the overlapping area 71 in the left image data 70A1 and the image data showing the overlapping area 71 in the right image data 70B1 with the other.

[0089] As an example, as shown in FIG. 8, the controller 46 includes a CPU 46A, a ROM 46B, The CPU 46A, the ROM 46B, the RAM 46C, the connection I / F 46D, and the input I / F 46E are connected to each other via a bus line 88.

[0090] Various programs are stored in the ROM 46B. The CPU 46A reads out the various programs from the ROM 46B and loads the read out various programs in the RAM 46C. The CPU 46A controls the entire imaging device 10 in accordance with the various programs loaded in the RAM 46C.

[0091] The connection I / F 46D is an FPGA, and is connected to the imaging element 44 via a communication line 57. The CPU 46A controls the imaging element 44 via the connection I / F 46D.

[0092] The input I / F 46E is a device having an FPGA, and is connected to the first signal processing unit 50 via the communication line 60. The input I / F 46E receives composite image data 72 (see FIG. 6) from the first signal processing unit 50 via the communication line 60. The input I / F 46E transfers the composite image data 72 input from the first signal processing unit 50 to the CPU 46A.

[0093] A secondary storage device 80 and an external I / F 82 are connected to the bus line 88. The secondary storage device 80 is a non-volatile memory such as an SSD, an HDD, or an EEPROM. The CPU 46A reads and writes various information from and to the secondary storage device 80.

[0094] The external I / F 82 is a device having an FPGA. An external device (not shown) such as a USB memory or a memory card is connected to the external I / F 82. The external I / F 82 controls the transmission and reception of various types of information between the CPU 46A and the external device.

[0095] The UI device 48 includes the hybrid finder 16, the touch panel display 26, and a reception unit 84. The first display 32 and the touch panel 34 are connected to a bus line 88. Thus, the CPU 46A causes the first display 32 to display various information, and operates in accordance with various instructions received by the touch panel 34.

[0096] The reception unit 84 includes a touch panel 34 and a hard key unit 25. The hard key unit 25 is a plurality of hard keys, and has a release button 20, a dial 22, and instruction keys 28. The hard key unit 25 is connected to a bus line 88, and the CPU 46A operates in accordance with various instructions received by the hard key unit 25.

[0097] The hybrid viewfinder 16 includes a second display 86. The CPU 46A causes the second display 86 to display various types of information.

[0098] 9 as an example, the hybrid viewfinder 16 includes an OVF 90 and an EVF 92. The OVF 90 is an inverted Galilean viewfinder and includes an eyepiece 94, a prism 96, and an objective lens 98. The EVF 92 includes a second display 86, a prism 96, and an eyepiece 94.

[0099] A liquid crystal shutter 100 is arranged on the subject side of the objective lens 98 along the optical axis L2 of the objective lens 98, and when the EVF 92 is used, the liquid crystal shutter 100 blocks light so that an optical image does not enter the objective lens 98.

[0100] The prism 96 reflects the electronic image or various information displayed on the second display 86 to guide it to the eyepiece 94, and also combines the optical image with the electronic image and various information displayed on the second display 86. The electronic image displayed on the second display 86 may be a live view image 102 based on the composite image data 72.

[0101] In the OVF mode, the CPU 46A controls the liquid crystal shutter 100 to be in a non-light-shielding state, so that an optical image can be viewed through the eyepiece 94. In the EVF mode, the CPU 46A controls the liquid crystal shutter 100 to be in a light-shielding state, so that only the electronic image displayed on the second display 86 can be viewed through the eyepiece 94.

[0102] In the following description, for convenience of explanation, when there is no need to distinguish between the first display 32 (see FIGS. 2 and 8) and the second display 86, they will be referred to as "displays" without reference numerals. The displays are an example of a "display unit" according to the technology of the present disclosure. The CPU 46A is an example of a "control unit (control processor)" according to the technology of the present disclosure.

[0103] Next, the operation of the imaging device 10 will be described.

[0104] First, the flow of the imaging process executed by the processing circuit 62 of the imaging element 44 will be described with reference to FIG.

[0105] In the imaging process shown in FIG. 10, first, in step ST10, the photoelectric conversion element control circuit 62A determines whether or not the timing to start exposure of the photoelectric conversion element 61 (hereinafter referred to as "exposure start timing") has arrived. The exposure start timing is a timing periodically determined by the imaging frame rate described above. In step ST10, if the exposure start timing has not arrived, the determination is negative, and the imaging process proceeds to step ST22. In step ST10, if the exposure start timing has arrived, the determination is positive, and the imaging process proceeds to step ST12.

[0106] In step ST12, the photoelectric conversion element control circuit 62A causes the photoelectric conversion element 61 to perform exposure for one frame.

[0107] In the next step ST14, the photoelectric conversion element control circuit 62A reads out one frame of captured image data 70 from the photoelectric conversion element 61.

[0108] In the next step ST16, the digital processing circuit 62B performs signal processing using correlated double sampling on the captured image data 70 read out in step ST14, and then performs A / D conversion to digitize the analog captured image data 70. Then, the digital processing circuit 62B stores the digitized captured image data 70 in the memory 64.

[0109] In the next step ST18, the image processing circuit 62C acquires the captured image data 70 from the memory 64, and separates the acquired captured image data 70 into left image data 70A1 (see FIG. 6) and right image data 70B1 (see FIG. 6). The image processing circuit 62C outputs the left image data 70A1 and the right image data 70B1 to the output circuit 62D.

[0110] In the next step ST20, the output circuit 62D outputs the left side image data 70A1 to the first signal processing unit 50 via the first output line 53 (see FIGS. 3 to 6 and 8). In addition, the output circuit 62D outputs the right side image data 70B1 to the second signal processing unit 52 via the second output line 55 (see FIGS. 3 to 6 and 8).

[0111] In the next step ST22, the processing circuit 62 determines whether or not a condition for ending the imaging process (hereinafter, referred to as an "imaging process ending condition") is satisfied. For example, the imaging process ending condition may be that an instruction to end the imaging process is accepted by the acceptance unit 84. In step ST22, if the imaging process end condition is not satisfied, the determination is negative, and the imaging process proceeds to step ST10. In step ST22, if the imaging process end condition is satisfied, the determination is positive, and the imaging process ends.

[0112] Next, the flow of the first signal processing executed by the first signal processing unit 50 will be described with reference to FIG.

[0113] In the first signal processing shown in Fig. 11, in step ST30, the first signal processor 50 judges whether or not the left side image data 70A1 (see Fig. 6) has been input from the processing circuit 62. In step ST30, if the left side image data 70A1 has not been input from the processing circuit 62, the judgment is negative, and the first signal processing proceeds to step ST42. In step ST30, if the left side image data 70A1 has been input from the processing circuit 62, the judgment is positive, and the first signal processing proceeds to step ST32.

[0114] In step ST32, the first signal processor 50 performs specific signal processing on the left-side image data 70A1.

[0115] In the next step ST34, the first signal processing unit 50 determines whether or not it has received the right side image data 70B1 (see FIG. 6) transmitted by executing the process of step ST54 of the second signal processing shown in FIG. 12. If the right side image data 70B1 has not been received in step ST34, the determination is negative, and the first signal processing proceeds to step ST40. If the right side image data 70B1 has been received in step ST34, the determination is positive, and the first signal processing proceeds to step ST36.

[0116] In step ST40, the first signal processing unit 50 judges whether or not a condition for terminating the first signal processing (hereinafter referred to as the "first signal processing termination condition") is satisfied. An example of the first signal processing termination condition is that an instruction to terminate the first signal processing is accepted by the acceptance unit 84. In step ST40, if the first signal processing termination condition is not satisfied, the judgment is negative, and the first signal processing proceeds to step ST34. In step ST40, if the first signal processing termination condition is satisfied, the judgment is positive, and the first signal processing ends.

[0117] In step ST36, the first signal processing unit 50 generates composite image data 72 (see FIG. 6) by performing a synthesis process to synthesize the left side image data 70A1 obtained by executing the processing of step ST32 and the right side image data 70B1 received in step ST34.

[0118] In the next step ST38, the first signal processing unit 50 outputs the composite image data 72 obtained by executing the processing of step ST36 to the controller 46 (see FIG. 6) via the communication line 60 (see FIGS. 6 and 8).

[0119] In the next step ST42, the first signal processing unit 50 judges whether or not the first signal processing end condition is satisfied. If the first signal processing end condition is not satisfied in step ST42, the judgment is negative, and the first signal processing proceeds to step ST30. If the first signal processing end condition is satisfied in step ST42, the judgment is positive, and the first signal processing ends.

[0120] Next, the flow of the second signal processing executed by the second signal processing unit 52 will be described with reference to FIG.

[0121] In the second signal processing shown in FIG. 12, in step ST50, the second signal processing unit 52 It is determined whether the right side image data 70B1 (see FIG. 6) has been input from the path 62. In step ST50, if the right side image data 70B1 has not been input from the processing circuit 62, the determination is negative, and the second signal processing proceeds to step ST56. In step ST50, if the right side image data 70B1 has been input from the processing circuit 62, the determination is positive, and the second signal processing proceeds to step ST52.

[0122] In step ST52, the second signal processor 52 performs specific signal processing on the right-side image data 70B1.

[0123] In the next step ST54, the second signal processing unit 52 transmits the right side image data 70B1 obtained by executing the processing of step ST52 to the first signal processing unit 50 via the communication line 58 (see Figures 3 to 6 and 8).

[0124] In the next step ST56, the second signal processing unit 52 judges whether or not a condition for terminating the second signal processing (hereinafter referred to as the "second signal processing termination condition") is satisfied. An example of the second signal processing termination condition is that an instruction to terminate the second signal processing is accepted by the acceptance unit 84. In step ST56, if the second signal processing termination condition is not satisfied, the judgment is negative, and the second signal processing proceeds to step ST50. In step ST56, if the second signal processing termination condition is satisfied, the judgment is positive, and the second signal processing ends.

[0125] As described above, the imaging device 10 includes the imaging element 44, the first signal processing unit 50, and the second signal processing unit 52. The imaging element 44 also includes the output circuit 62D. The left image data 70A1 is output by the output circuit 62D to the first signal processing unit 50 via the first output line 53, and the right image data 70B1 is output by the output circuit 62D to the second signal processing unit 52 via the second output line 55. The left image data 70A1 and the right image data 70B1, each of which has been subjected to specific signal processing, are then synthesized by the first signal processing unit 50, and the synthesized image data 72 is output to the controller 46.

[0126] Therefore, the traffic between the imaging element 44 and each of the first signal processing unit 50 and the second signal processing unit 52 is smaller than when the captured image data 70 is output to the first signal processing unit 50 and the second signal processing unit 52 using only one output line.

[0127] Furthermore, the amount of image data on which the first signal processing unit 50 and the second signal processing unit 52 perform specific signal processing is smaller than the captured image data 70. Therefore, compared to a case where only the first signal processing unit 50 or the second signal processing unit 52 performs specific signal processing on the entire captured image data 70, the load on each of the first signal processing unit 50 and the second signal processing unit 52 when the specific signal processing is executed is reduced.

[0128] Therefore, the imaging device 10 can realize high-speed image processing compared to a case in which image data is output from the imaging element 44 to a plurality of signal processing units using only one output line.

[0129] Moreover, each of the left image data 70A1 and the right image data 70B1 is image data showing an image having an overlapping area 71 between the left image 70A1a and the right image 70B1a, as shown in FIG. 7. When two images obtained by simply dividing a captured image in half in the left-right direction RL (see FIG. 7) are synthesized, there is a risk that the boundary area between the two images will become apparent. However, as shown in FIG. 7, each of the left image 70A1a and the right image 70B1a has an overlapping area 71. Therefore, compared to when two images obtained by simply dividing a captured image in half in the left-right direction RL are synthesized, it is possible to prevent the occurrence of a situation in which the boundary area between the two images becomes apparent.

[0130] Moreover, the imaging element 44 is an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are integrated into a single chip. This increases the portability of the imaging element 44 compared to an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are not integrated into a single chip. Also, the degree of freedom in design can be increased compared to an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are not integrated into a single chip. Furthermore, this can contribute to the miniaturization of the imaging device body 12 compared to an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are not integrated into a single chip.

[0131] 4, a stacked type imaging element in which a memory 64 is stacked on a photoelectric conversion element 61 is adopted as the imaging element 44. This can reduce the load on the processing between the photoelectric conversion element 61 and the memory 64 compared to when the photoelectric conversion element 61 and the memory 64 are not stacked. Also, compared to when the photoelectric conversion element 61 and the memory 64 are not stacked, the degree of freedom in design can be increased. Furthermore, compared to when the photoelectric conversion element 61 and the memory 64 are not stacked, this can contribute to a smaller size of the imaging device body 12.

[0132] 9, an image represented by the composite image data 72 is displayed as a live view image 102 on the second display 86 under the control of the CPU 46A. This allows the user to view the image represented by the composite image data 72. In the example shown in FIG. 9, the live view image 102 is displayed on the second display 86, but the technology of the present disclosure is not limited to this. For example, the live view image 102 may be displayed on the first display 32, or the live view image 102 may be displayed on both the first display 32 and the second display 86.

[0133] In the above-described first embodiment, an image sensor in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are integrated into one chip has been exemplified as the image sensor 44, but the technology of the present disclosure is not limited to this. For example, among the photoelectric conversion element 61, the processing circuit 62, and the memory 64, at least the photoelectric conversion element 61 and the memory 64 may be integrated into one chip.

[0134] In the above first embodiment, an example of the captured image data 70 being separated into the left image data 70A1 and the right image data 70B1 has been described, but the technology of the present disclosure is not limited thereto. For example, the captured image data 70 may be separated into upper image data indicating an image of the upper region of the captured image and lower image data indicating an image of the lower region of the captured image. The captured image data 70 may also be separated into upper left image data indicating an image of the upper left region of the captured image and lower right image data indicating an image of the lower right region of the captured image. The captured image data 70 may also be separated into upper right image data indicating an image of the upper right region of the captured image and lower left image data indicating an image of the lower left region of the captured image. Furthermore, the captured image data 70 may also be separated into central image data indicating an image of the central region of the captured image and peripheral image data indicating an image of the peripheral region of the captured image, i.e., the region other than the central region.

[0135] Even when the captured image data 70 is separated in this manner, it is preferable that each of the pair of image data obtained by separating the captured image data 70 is image data that includes overlapping image data that indicates an image that overlaps in the separation direction between the two images into which the captured image is separated.

[0136] [Second embodiment] In the first embodiment, the same signal processing is performed on each of the first separated image data 70A and the second separated image data 70B. In the second embodiment, A case where differential signal processing is performed on the first separated image data 70A and the second separated image data 70B will be described. In the second embodiment, the same components as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted. The following describes the parts that are different from the first embodiment.

[0137] As shown in FIG. 1, an imaging device 200 according to the second embodiment differs from the imaging device 10 described in the first embodiment in that it has an imaging device body 212 instead of the imaging device body 12.

[0138] The imaging device body 212 differs from the imaging device body 12 in that it has an imaging element 244 (see FIG. 13) instead of the imaging element 44. The imaging device body 212 also differs from the imaging device body 12 in that it has a first signal processing unit 250 (see FIG. 13) instead of the first signal processing unit 50, and a second signal processing unit 252 (see FIG. 13) instead of the second signal processing unit 52.

[0139] 13, the image sensor 244 differs from the image sensor 44 in that it has a processing circuit 262 instead of the processing circuit 62. The processing circuit 262 differs from the processing circuit 62 in that it has an image processing circuit 262C instead of the image processing circuit 62C, and in that it has an output circuit 262D instead of the output circuit 62D.

[0140] The image processing circuit 262C is different from the image processing circuit 62C in that it has a face detection circuit 262C1. The face detection circuit 262C1 is an example of the "detection unit (detection processor)" according to the technology of the present disclosure. The face detection circuit 262C1 is a circuit having a known face detection function. The face detection circuit 262C1 is realized by a hardware configuration such as an ASIC. The face detection circuit 262C1 is not limited to a hardware configuration, and may be realized by a software configuration, or may be realized by a software configuration and a hardware configuration. In the second embodiment, the face detection circuit 262C1 acquires the captured image data 70 from the memory 64. The face detection circuit 262C1 identifies a face image 69 (see FIG. 14A) indicating a human face from the captured image indicated by the acquired captured image data 70. That is, the face detection circuit 262C1 detects face image data indicating the face image 69 from the captured image data 70.

[0141] Then, the face detection circuit 262C1 extracts face area image data 70B2 indicating a face area image 70B2a (see FIG. 14A) within a predetermined range including the identified face image 69 from the captured image data 70, thereby separating the captured image data 70 into face area image data 70B2 and background area image data 70A2.

[0142] The background area image data 70A2 refers to image data showing an image of the background area of ​​the captured image, that is, an image other than the face area image 70B2a. The above-mentioned predetermined range refers to a range determined according to, for example, the position of the face image 69 (see FIG. 14A) in the captured image and the degree of difference between the size of the face image 69 and the size of the image other than the face image 69 in the captured image. In the example shown in FIG. 14A, the face image 69 is located in the central area of ​​the captured image, and in this case, the captured image is separated into three areas, the central area, the upper area, and the lower area. The image of the central area is specified as the face area image 70B2a, and the images of the upper area and the lower area, that is, the images other than the central area of ​​the captured image, are specified as the background area image 70A2a.

[0143] In the example shown in FIG. 14A, an image showing two faces is shown as face image 69. Face image 69 is an image that identifies an area in a captured image where the density of faces of multiple people is the highest. Note that the technology of the present disclosure is not limited to this. For example, instead of face image 69, an image showing the face of one person may be used. Alternatively, a face image showing a face with a particular expression (for example, a smile) may be applied, or a face image showing the face of a particular person may be applied.

[0144] 13, background area image data 70A2 and facial area image data 70B2 are output to the output circuit 262D by the image processing circuit 262C. The background area image data 70A2 is an example of the above-mentioned first separated image data 70A, and the facial area image data 70B2 is an example of the above-mentioned second separated image data 70B. The facial area image 70B2a is an example of the "specified image" according to the technology of the present disclosure, and the background area image 70A2a is an example of the "image different from the specified image" according to the technology of the present disclosure.

[0145] The output circuit 262D differs from the output circuit 62D in that it outputs background area image data 70A2 instead of the left side image data 70A1, and in that it outputs face area image data 70B2 instead of the right side image data 70B1.

[0146] Specifically, the output circuit 262D outputs the background area image data 70A2 to the first signal processing unit 250 via the first output line 53. In addition, the output circuit 262D outputs the face area image data 70B2 to the second signal processing unit 252 via the second output line 55.

[0147] The first signal processing section 250 differs from the first signal processing section 50 in that it has a signal processing circuit 250B instead of the signal processing circuit 50B.

[0148] The background region image data 70A2 output from the output circuit 262D is temporarily stored in the buffer 50A, and is then output from the buffer 50A to the signal processing circuit 250B. The signal processing circuit 250B then performs specific signal processing on the background region image data 70A2 input from the buffer 50A.

[0149] The second signal processing unit 252 differs from the second signal processing unit 52 in that it has a second signal processing circuit 252B instead of the signal processing circuit 52B. The signal processing circuit 252B differs from the signal processing circuit 52B in that it has a face recognition processing circuit 252B1.

[0150] The face area image data 70B2 output from the output circuit 262D is temporarily stored in the buffer 52A, and is output from the buffer 52A to the signal processing circuit 252B. Then, the signal processing circuit 252B performs specific signal processing on the face area image data 70B2 input from the buffer 52A. The face recognition processing circuit 252B1 has a known face recognition function, and executes face recognition on the face area image data 70B2 that has been subjected to the specific signal processing. By executing the face recognition, for example, it is determined whether or not the face shown by the face image 69 (see FIG. 14A) is a specific display (for example, a smile) and / or whether or not the face shown by the face image 69 is the face of a specific person.

[0151] The face authentication processing circuit 252B1 outputs face area image data 70B2 on which face authentication has been performed to the transmission circuit 52C. Face area image data 70B2 is provided with face authentication result information indicating the result of face authentication, and is used for processing in a downstream circuit. The "downstream circuit" here refers to, for example, the first signal processing unit 250 and / or the controller 46.

[0152] The transmission circuit 52C transmits to the first signal processing unit 250 the face area image data 70B2 input from the face authentication processing circuit 252B1.

[0153] In the first signal processing unit 250, the receiving circuit 50C receives the face area image data 70B2 transmitted from the transmitting circuit 52C. Then, the signal processing circuit 250B combines the face area image data 70B2 received by the receiving circuit 50C with a specific signal generated by the signal processing circuit 250B. The signal processing circuit 250B combines this with the processed background area image data 70A2 to generate combined image data 272. Then, the signal processing circuit 250B outputs the combined image data 272 to the controller 46 via the communication line 60.

[0154] Note that the background area image 70A2a and the face area image 70B2a may have an overlapping area 271 as shown in FIG. 14B as an example. The overlapping area 271 is an area where the background area image 70A2a and the face area image 70B2a overlap in the up-down direction UD. As with the overlapping area 71 described in the first embodiment, the number of pixels in the up-down direction UD of the overlapping area 271 may be, for example, several tens of pixels to several hundreds of pixels. When the background area image data 70A2 and the face area image data 70B2 are combined by the first signal processing unit 250, each image data showing the overlapping area 271 of the background area image data 70A2 and the face area image data 70B2 is averaged as in the first embodiment.

[0155] Next, the operation of the imaging device 200 will be described.

[0156] First, the flow of the imaging process executed by the processing circuit 262 of the imaging element 244 will be described with reference to Fig. 15. The imaging process shown in Fig. 15 differs from the imaging process shown in Fig. 10 in that it has a process of step ST60 instead of the process of step ST18, and in that it has a process of step ST62 instead of the process of step ST20. Therefore, in the flowchart of the imaging process shown in Fig. 15, the same steps as those in the imaging process shown in Fig. 10 are assigned the same step numbers. Hereinafter, the imaging process shown in Fig. 15 will be described only in terms of the differences from the imaging process shown in Fig. 10.

[0157] 15, in step ST60, the face detection circuit 262C1 acquires captured image data 70 from the memory 64 and separates the acquired captured image data 70 into background area image data 70A2 and face area image data 70B2. Then, the image processing circuit 262C outputs the background area image data 70A2 and the face area image data 70B2 to the output circuit 262D.

[0158] In the next step ST62, the output circuit 262D outputs the background area image data 70A2 to the first signal processing unit 250 via the first output line 53. In addition, the output circuit 262D outputs the face area image data 70B2 to the second signal processing unit 252 via the second output line 55.

[0159] Next, the flow of the first signal processing executed by the first signal processing unit 250 will be described with reference to FIG.

[0160] In the first signal processing shown in Fig. 16, in step ST70, the first signal processing unit 250 determines whether or not background region image data 70A2 (see Fig. 13) has been input from the processing circuit 262. If the background region image data 70A2 has not been input from the processing circuit 262 in step ST70, the determination is negative, and the first signal processing proceeds to step ST82. If the background region image data 70A2 has been input from the processing circuit 262 in step ST70, the determination is positive, and the first signal processing proceeds to step ST72.

[0161] In step ST72, the first signal processing unit 250 performs specific signal processing on the background region image data 70A2.

[0162] In the next step ST74, the first signal processing unit 250 judges whether or not it has received the face area image data 70B2 (see FIG. 13) transmitted by executing the process of step ST94 of the second signal processing shown in FIG. 17. In step ST74, the face area image data 70B If the face area image data 70B2 has not been received, the determination is negative, and the first signal processing proceeds to step ST80. If the face area image data 70B2 has been received in step ST74, the determination is positive, and the first signal processing proceeds to step ST76.

[0163] In step ST80, the first signal processing unit 250 judges whether or not the first signal processing end condition is satisfied. In step ST80, if the first signal processing end condition is not satisfied, the judgment is negative, and the first signal processing proceeds to step ST74. In step ST80, if the first signal processing end condition is satisfied, the judgment is positive, and the first signal processing ends.

[0164] In step ST76, the first signal processing unit 250 generates composite image data 272 (see FIG. 13) by performing a synthesis process to synthesize the background area image data 70A2 obtained by executing the processing of step ST72 and the face area image data 70B2 received in step ST74.

[0165] In the next step ST78, the first signal processing unit 250 outputs the composite image data 272 obtained by executing the process of step ST76 to the controller 46 (see FIG. 13) via the communication line 60 (see FIG. 13).

[0166] In the next step ST82, the first signal processing unit 250 judges whether or not the first signal processing end condition is satisfied. If the first signal processing end condition is not satisfied in step ST82, the judgment is negative, and the first signal processing proceeds to step ST70. If the first signal processing end condition is satisfied in step ST82, the judgment is positive, and the first signal processing ends.

[0167] Next, the flow of the second signal processing executed by the second signal processing unit 252 will be described with reference to FIG.

[0168] In the second signal processing shown in Fig. 17, in step ST90, the second signal processing unit 252 determines whether or not the face area image data 70B2 (see Fig. 13) has been input from the processing circuit 262. If the face area image data 70B2 has not been input from the processing circuit 262 in step ST90, the determination is negative, and the second signal processing proceeds to step ST96. If the face area image data 70B2 has been input from the processing circuit 262 in step ST90, the determination is positive, and the second signal processing proceeds to step ST92.

[0169] In step ST92, the second signal processing unit 252 performs specific signal processing on the face area image data 70B2. Furthermore, the second signal processing unit 252 executes face authentication on the face area image data 70B2 that has been subjected to the specific signal processing.

[0170] In the next step ST94, the second signal processing unit 252 transmits the face area image data 70B2 obtained by executing the process of step ST92 to the first signal processing unit 250 via the communication line 58 (see FIG. 13). Note that the face area image data 70B2 obtained by executing the process of step ST92 is provided with face recognition result information.

[0171] In the next step ST96, the second signal processing unit 252 judges whether or not the second signal processing end condition is satisfied. If the second signal processing end condition is not satisfied in step ST96, the judgment is negative, and the second signal processing proceeds to step ST90. If the second signal processing end condition is satisfied in step ST96, the judgment is positive, and the second signal processing ends.

[0172] As described above, in the imaging device 200, the captured image data 70 is separated into the background area image data 70A2 and the face area image data 70B2. Then, the background area image data 70A2 is output to the first signal processing unit 250 via the first output line 53, and the face area image data 70B2 is output to the second signal processing unit 252 via the second output line 55.

[0173] Generally, the face area image data 70B2 is image data that is given more importance than the background area image data 70A2, so the second signal processing unit 252 applies more complex processing to the face area image data 70B2 than to the background area image data 70A2. In the second embodiment, face recognition is executed as the complex processing. Then, the background area image data 70A2 that has been subjected to specific signal processing and the face area image data 70B2 that has been subjected to specific signal processing and face recognition are synthesized by the first signal processing unit 250.

[0174] In this way, since processing of different degrees of complexity is performed by the first signal processing unit 250 and the second signal processing unit 252, the load required for image processing can be reduced compared to the case where complex processing is always performed on the entire captured image data 70.

[0175] 14B, the background area image 70A2a and the face area image 70B2a adjacent in the up-down direction UD each have an overlapping area 271 between the background area image 70A2a and the face area image 70B2a. As a result, the signal processing circuit 250B generates composite image data 272 in which the boundary area between the background area image 70A2a and the face area image 70B2a is less obvious than when two images obtained by simply dividing the image in half are joined together.

[0176] In the above-mentioned second embodiment, an example in which the captured image data 70 is separated into the background area image data 70A2 and the face area image data 70B2 has been described, but the technology of the present disclosure is not limited thereto. For example, image data designated by the user via the reception unit 84 from among the captured image data 70 may be output to the second signal processing unit 252, and the remaining image data from the captured image data 70 may be output to the first signal processing unit 250. The image data designated by the user refers to, for example, image data from among the captured image data 70 that is determined as important image data according to an instruction received by the reception unit 84. The important image data refers to, for example, image data indicating a partial area including an image of a person and / or a building that the user pays attention to.

[0177] In this case as well, it is preferable to provide an overlapping area corresponding to the overlapping area 271 between the image represented by the image data designated by the user and the image represented by the remaining image data of the captured image data 70. As a result, the signal processing circuit 250B generates composite image data in which the boundary area between the image represented by the designated image data and the image represented by the remaining image data is less obvious than when two halves of the image are simply joined together.

[0178] In the second embodiment, face recognition is given as an example of complex processing, but the technology of the present disclosure is not limited to this. Examples of complex processing include an iris detection process for detecting a human iris, a red-eye correction process for correcting red-eye, and / or an electronic blur correction process (e.g., EIS).

[0179] [Third embodiment] In the first embodiment, an example in which the captured image data 70 is separated into left image data 70A1 and right image data 70B1 is described, but in the third embodiment, an example in which the captured image data 70 is compressed and then separated into two pieces of image data is described. In the third embodiment, the same components as those in the first embodiment are given the same reference numerals, and the description thereof is omitted. The following describes the parts that are different from the first embodiment.

[0180] As shown in FIG. 1, an imaging device 300 according to the third embodiment differs from the imaging device 10 described in the first embodiment in that it has an imaging device body 312 instead of the imaging device body 12.

[0181] The imaging device body 312 differs from the imaging device body 12 in that it has an imaging element 344 (see FIG. 18) instead of the imaging element 44, and in that it has a first signal processing unit 350 (see FIG. 18) instead of the first signal processing unit 50. The first signal processing unit 350 differs from the first signal processing unit 50 in that it has a signal processing circuit 350B instead of the signal processing circuit 50B.

[0182] 18, the image sensor 344 differs from the image sensor 44 in that it has a processing circuit 362 instead of the processing circuit 62. The processing circuit 362 differs from the processing circuit 62 in that it has an image processing circuit 362C instead of the image processing circuit 62C, and in that it has an output circuit 362D instead of the output circuit 62D.

[0183] The image processing circuit 362C acquires captured image data 70 from the memory 64. The captured image data 70 acquired by the image processing circuit 362C from the memory 64 is color image data in which R pixels, G pixels, and B pixels are periodically arranged. As an example, as shown in FIG. 19, in the captured image data 70, R pixels, G pixels, and B pixels are periodically arranged in accordance with the X-Trans (registered trademark) array. Note that the R pixels, G pixels, and B pixels are an example of "multiple primary color pixels" according to the technology of the present disclosure.

[0184] In the example shown in FIG. 19, in the first row, R pixels, G pixels, and B pixels are arranged in a cyclic manner in the row direction in the order of G pixels, B pixels, R pixels, G pixels, R pixels, and B pixels. In the second row, R pixels, G pixels, and B pixels are arranged in a cyclic manner in the row direction in the order of R pixels, G pixels, G pixels, B pixels, G pixels, and G pixels. In the third row, R pixels, G pixels, and B pixels are arranged in a cyclic manner in the row direction in the order of B pixels, G pixels, G pixels, R pixels, G pixels, and G pixels. In the fourth row, R pixels, G pixels, and B pixels are arranged in a cyclic manner in the row direction in the order of G pixels, R pixels, B pixels, G pixels, B pixels, and R pixels. In the fifth row, R pixels, G pixels, and B pixels are arranged in a cyclic manner in the row direction in the order of B pixels, G pixels, G pixels, R pixels, G pixels, and G pixels. Furthermore, in the sixth row, R pixels, G pixels, and B pixels are cyclically arranged in the row direction in the order of R pixel, G pixel, G pixel, B pixel, G pixel, and G pixel. The arrangement pattern of R pixels, G pixels, and B pixels in the first to sixth rows is repeated in units of six rows in the column direction, thereby forming the arrangement pattern of R pixels, G pixels, and B pixels of the entire captured image data 70.

[0185] The image processing circuit 362C compresses the captured image data 70 acquired from the memory 64. That is, the image processing circuit 362C generates vertically thinned image data 73 from the captured image data 70. The vertically thinned image data 73 is image data obtained by thinning the captured image data 70 row by row. Specifically, as shown in Fig. 19 as an example, the vertically thinned image data 73 is image data representing a vertical 1 / 2 thinned image obtained by thinning out pixels of even-numbered lines in the column direction from the captured image represented by the captured image data 70.

[0186] The image processing circuit 362C separates the vertically thinned image data 73 into odd-numbered column image data 73A and even-numbered column image data 73B, and outputs the odd-numbered column image data 73A and the even-numbered column image data 73B obtained by the separation to the output circuit 362D. Note that, although an example in which the captured image data 70 is thinned and then divided to obtain odd-numbered column image data 73A and even-numbered column image data 73B as a plurality of divided image data has been described here, the technology of the present disclosure is not limited to this. For example, the captured image data 70 may be directly divided to obtain a plurality of divided image data.

[0187] As an example, as shown in FIG. 20 and FIG. 21, the odd-numbered column image data 73A is image data showing an odd-numbered column image obtained by thinning out pixels of even-numbered columns from the vertical 1 / 2-thinned image shown by the vertical thinned image data 73. That is, the odd-numbered column image data 73A is image data showing a set of pixels of odd-numbered columns in the vertical 1 / 2-thinned image. As an example, as shown in FIG. 20 and FIG. 21, the even-numbered column image data 73B is image data showing an even-numbered column image obtained by thinning out pixels of odd-numbered columns from the vertical 1 / 2-thinned image shown by the vertical thinned image data 73. That is, the even-numbered column image data 73B is color image data showing a set of pixels of even-numbered columns in the vertical 1 / 2-thinned image. In other words, each of the odd-numbered column image data 73A and the even-numbered column image data 73B is color image data showing an image in which R pixels, G pixels, and B pixels are periodically arranged. The odd-numbered column image data 73A and the even-numbered column image data 73B are an example of the "plurality of primary color pixel array image data" and the "plurality of divided image data" according to the technique of the present disclosure.

[0188] The output circuit 362D outputs the odd-numbered column image data 73A input from the image processing circuit 362C to the first signal processing unit 350 via the first output line 53. In addition, the output circuit 362D outputs the even-numbered column image data 73B input from the image processing circuit 362C to the second signal processing unit 52 via the second output line 55.

[0189] In the second signal processing unit 52, processing similar to that performed on the right side image data 70B1 described in the first embodiment above is performed on the even column image data 73B, and the processed even column image data 73B is transmitted to the first signal processing unit 350.

[0190] In the first signal processing unit 350, the receiving circuit 50C receives the even-numbered column image data 73B transmitted from the second signal processing unit 52. The signal processing circuit 350B acquires the even-numbered column image data 73B received by the receiving circuit 50C.

[0191] On the other hand, the odd-numbered column image data 73A is input to the buffer 50A. The buffer 50A temporarily holds the odd-numbered column image data 73A and outputs it to the signal processing circuit 350B in a FIFO manner. The signal processing circuit 350B performs specific signal processing on the odd-numbered column image data 73A input from the buffer 50A. The signal processing circuit 350B also generates composite image data 372 by combining the odd-numbered column image data 73A that has been subjected to the specific signal processing with the even-numbered column image data 73B acquired from the receiving circuit 50C. As a result, the arrangement pattern of the R pixels, G pixels, and B pixels of the image represented by the composite image data 372 is the same as that of the vertical 1 / 2 thinned image described above. In other words, the arrangement pattern of the R pixels, G pixels, and B pixels of the image represented by the composite image data 372 is an arrangement pattern having a periodicity that allows demosaic processing of the R pixels, G pixels, and B pixels.

[0192] Therefore, the signal processing circuit 350B performs demosaic processing of the R, G, and B signals on the composite image data 372, and outputs the demosaic-processed composite image data 372 to the controller 46 via the communication line 60.

[0193] Next, the operation of the imaging device 300 will be described.

[0194] First, the flow of the imaging process executed by the processing circuit 362 of the image sensor 344 will be described with reference to Fig. 22. The imaging process shown in Fig. 22 differs from the imaging process shown in Fig. 10 in that it has a process of step ST100 instead of the process of step ST18, and in that it has a process of step ST102 instead of the process of step ST20. Therefore, in the flowchart of the imaging process shown in Fig. 22, the same steps as those in the imaging process shown in Fig. 10 are assigned the same step numbers. Hereinafter, the imaging process shown in Fig. 22 will be described with reference to Fig. 10. 2. Only the points different from the imaging process shown in FIG.

[0195] 22, in step ST100, the image processing circuit 362C generates vertically thinned image data 73 (see FIGS. 18 to 21) from the captured image data 70. Then, the image processing circuit 362C generates odd-numbered column image data 73A and even-numbered column image data 73B from the generated vertically thinned image data 73. That is, the vertically thinned image data 73 is separated into odd-numbered column image data 73A and even-numbered column image data 73B (see FIGS. 20 and 21).

[0196] In the next step ST102, the output circuit 362D outputs the odd-numbered column image data 73A to the first signal processing unit 350 via the first output line 53. In addition, the output circuit 362D outputs the even-numbered column image data 73B to the second signal processing unit 52 via the second output line 55.

[0197] Next, the flow of the first signal processing executed by the first signal processing unit 350 will be described with reference to FIG.

[0198] 23, in step ST210, the first signal processing unit 350 determines whether or not odd-numbered column image data 73A (see FIG. 18) has been input from the processing circuit 362. If odd-numbered column image data 73A has not been input from the processing circuit 362 in step ST210, the determination is negative, and the first signal processing proceeds to step ST222. If odd-numbered column image data 73A has been input from the processing circuit 362 in step ST210, the determination is positive, and the first signal processing proceeds to step ST212.

[0199] In step ST212, the first signal processing unit 350 performs specific signal processing on the odd-numbered column image data 73A.

[0200] In the next step ST214, the first signal processing unit 350 determines whether or not it has received the even-numbered column image data 73B (see FIG. 18) transmitted by executing the process of step ST234 of the second signal processing shown in FIG. 24. If the even-numbered column image data 73B has not been received in step ST214, the determination is negative, and the first signal processing proceeds to step ST220. If the even-numbered column image data 73B has been received in step ST214, the determination is positive, and the first signal processing proceeds to step ST216.

[0201] In step ST220, the first signal processing unit 350 judges whether or not the first signal processing end condition is satisfied. In step ST220, if the first signal processing end condition is not satisfied, the judgment is negative, and the first signal processing proceeds to step ST214. In step ST220, if the first signal processing end condition is satisfied, the judgment is positive, and the first signal processing ends.

[0202] In step ST216, the first signal processing unit 350 performs a synthesis process to synthesize the odd-numbered column image data 73A obtained by executing the process of step ST212 and the even-numbered column image data 73B received in step ST214, thereby generating synthetic image data 372 (see FIG. 18). Then, the first signal processing unit 350 performs a demosaic process on the synthetic image data 372.

[0203] In the next step ST218, the first signal processing unit 350 outputs the composite image data 372 obtained by executing the process of step ST216 to the controller 46 (see FIG. 18) via the communication line 60 (see FIG. 18).

[0204] In the next step ST222, the first signal processing section 350 judges whether or not the first signal processing end condition is satisfied. If not, the determination is negative, and the first signal processing proceeds to step ST210. If the first signal processing end condition is satisfied in step ST222, the determination is positive, and the first signal processing ends.

[0205] Next, the flow of the second signal processing executed by the second signal processing unit 52 will be described with reference to FIG.

[0206] In the second signal processing shown in Fig. 24, in step ST230, the second signal processor 52 determines whether or not the even-numbered column image data 73B (see Fig. 18) has been input from the processing circuit 362. If the even-numbered column image data 73B has not been input from the processing circuit 362 in step ST230, the determination is negative, and the second signal processing proceeds to step ST236. If the even-numbered column image data 73B has been input from the processing circuit 362 in step ST230, the determination is positive, and the second signal processing proceeds to step ST232.

[0207] In step ST232, the second signal processing unit 52 performs specific signal processing on the even-numbered column image data 73B.

[0208] In the next step ST234, the second signal processing unit 52 transmits the even-numbered column image data 73B obtained by executing the process of step ST232 to the first signal processing unit 350 via the communication line 58 (see FIG. 18).

[0209] In the next step ST236, the second signal processing unit 52 judges whether or not the second signal processing end condition is satisfied. If the second signal processing end condition is not satisfied in step ST236, the judgment is negative, and the second signal processing proceeds to step ST230. If the second signal processing end condition is satisfied in step ST236, the judgment is positive, and the second signal processing ends.

[0210] As described above, in the third embodiment, the captured image data 70 is color image data representing a color captured image in which a plurality of primary color pixels, R pixels, G pixels, and B pixels, are periodically arranged. The captured image data 70 is divided into a plurality of primary color pixel array image data as a plurality of image data. In the example shown in FIG. 18 and FIG. 20, odd-numbered column image data 73A and even-numbered column image data 73B are shown as a plurality of primary color pixel array image data. Each of the odd-numbered column image data 73A and the even-numbered column image data 73B is image data representing an image in which R pixels, G pixels, and B pixels are periodically arranged. By using image data representing an image in which R pixels, G pixels, and B pixels are periodically arranged, such as the odd-numbered column image data 73A and the even-numbered column image data 73B, it is possible to realize demosaic processing for the R pixels, G pixels, and B pixels.

[0211] 18 and 19, the odd column image data 73A and the even column image data 73B are image data obtained by dividing the vertically thinned image data 73. The vertically thinned image data 73 has a smaller amount of data than the captured image data 70. Therefore, the specific signal processing can be performed on the odd column image data 73A and the even column image data 73B at higher speeds than when the specific signal processing is performed on one of the two image data obtained by dividing the captured image data 70 without thinning it out.

[0212] The odd-numbered column image data 73A is image data representing a set of pixels in odd-numbered columns of the vertical 1 / 2 thinned image, and the even-numbered column image data 73B is image data representing a set of pixels in even-numbered columns of the vertical 1 / 2 thinned image. The specific signal processing and synthesis of the multiple image data obtained by irregularly dividing the vertical 1 / 2 thinned image is more complicated than the processing of the odd-numbered column image data 73A and the even-numbered column image data 73B. The column image data 73A and the even-numbered column image data 73B can be processed at a higher speed than processing a plurality of image data obtained by irregularly dividing a vertical 1 / 2 thinned image.

[0213] In addition, the first signal processing unit 350 performs demosaic processing on the composite image data 372 obtained by combining the odd-numbered column image data 73A and the even-numbered column image data 73B, so that a higher quality image can be obtained compared to the case where demosaic processing is not performed.

[0214] In the above third embodiment, the first signal processing unit 350 uses the DRAM 54, and the second signal processing unit 52 uses the DRAM 56. However, the technology of the present disclosure is not limited to this. For example, as shown in FIG. 25, the first signal processing unit 750 may be applied instead of the first signal processing unit 350, and the second signal processing unit 752 may be applied instead of the second signal processing unit 52, without using the DRAMs 54 and 56. In the example shown in FIG. 25, the first signal processing unit 750 is different from the first signal processing unit 350 in that it has a line memory 750A and does not use the DRAM 54. Also, the second signal processing unit 752 is different from the second signal processing unit 52 in that it has a line memory 752A and does not use the DRAM 56.

[0215] In the first signal processing unit 750, the line memory 750A is interposed between the buffer 50A and the signal processing circuit 350B. The buffer 50A outputs odd-numbered column image data 73A to the line memory 750A. The line memory 750A stores the odd-numbered column image data 73A input from the buffer 50A on a line-by-line basis, and outputs the data to the signal processing circuit 350B in a FIFO manner. The signal processing circuit 350B executes the processes described in the third embodiment.

[0216] On the other hand, in the second signal processing unit 752, the line memory 752A is interposed between the buffer 52A and the signal processing circuit 52B. The buffer 52A outputs the even-numbered column image data 73B to the line memory 752A. The line memory 752A stores the even-numbered column image data 73B input from the buffer 52A on a line-by-line basis, and outputs the data to the signal processing circuit 52B in a FIFO manner. The signal processing circuit 52B executes the process described in the third embodiment.

[0217] In the above third embodiment, image data representing a vertically 1 / 2 thinned image is exemplified as the compressed image data, but the technology of the present disclosure is not limited to this. For example, when n is a natural number of 3 or more, image data representing a vertically 1 / n thinned image may be applied as the compressed image data. Also, image data representing a horizontally thinned image thinned by columns may be applied as the compressed image data, or image data representing an image thinned by rows and columns may be applied as the compressed image data.

[0218] [Fourth embodiment] In the first embodiment, an example in which the captured image data 70 is separated into left image data 70A1 and right image data 70B1 is described, but in the fourth embodiment, an example in which the method of separating the captured image data 70 is changed depending on the operation mode is described. In the fourth embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted. The following describes the parts that are different from the first embodiment.

[0219] As shown in FIG. 1, an imaging device 400 according to the fourth embodiment differs from the imaging device 10 described in the first embodiment in that it has an imaging device body 412 instead of the imaging device body 12.

[0220] The imaging device body 412 differs from the imaging device body 12 in that it has an imaging element 444 (see FIG. 26) instead of the imaging element 44. The imaging element 444 differs from the imaging element 44 in that it has a processing circuit 462 instead of the processing circuit 62. The processing circuit 462 4 in that it has an image processing circuit 462C instead of the image processing circuit 62C, and an output circuit 462D instead of the output circuit 62D.

[0221] The imaging device body 412 differs from the imaging device body 12 in that it has a first signal processing unit 450 (see FIG. 26) instead of the first signal processing unit 50, and in that it has a second signal processing unit 452 (see FIG. 26) instead of the second signal processing unit 52.

[0222] The first signal processing unit 450 differs from the first signal processing unit 50 in that it has the functions of the first signal processing unit 50 and the functions of the first signal processing unit 350 described in the third embodiment. Also, the first signal processing unit 450 differs from the first signal processing unit 50 in that it selectively activates the functions of the first signal processing unit 50 and the functions of the first signal processing unit 350.

[0223] The second signal processing unit 452 differs from the second signal processing unit 52 in that it has the functions of the second signal processing unit 52 and the functions of the second signal processing unit 252 described in the second embodiment. In addition, the second signal processing unit 452 differs from the second signal processing unit 52 in that it selectively activates the functions of the second signal processing unit 52 and the functions of the second signal processing unit 252 described in the second embodiment.

[0224] The controller 46 selectively outputs a still image imaging mode signal 480A (see FIG. 26) or a display moving image imaging mode signal 480B (see FIG. 27) to the processing circuit 462 via the communication line 60. For example, the still image imaging mode signal 480A is output from the controller 46 when an instruction to set the imaging device 400 to a still image imaging mode is received by the reception unit 84. Also, for example, the display moving image imaging mode signal 480B is output from the controller 46 when an instruction to set the imaging device 400 to a display moving image imaging mode is received by the reception unit 84.

[0225] As shown in FIG. 26, when the still image imaging mode signal 480A is input from the controller 46 via the communication line 60, the processing circuit 462 operates the image processing circuit 462C in the same manner as the image processing circuit 62C described in the first embodiment. That is, the processing circuit 462 causes the image processing circuit 462C to separate the captured image data 70 into left image data 70A1 and right image data 70B1, and outputs the left image data 70A1 and the right image data 70B1 to the output circuit 462D. The left image 70A1a represented by the left image data 70A1 and the right image 70B1a represented by the right image data 70B1 have the overlapping area 71 described in the first embodiment. The left image data 70A1 and the right image data 70B1 according to the fourth embodiment are examples of "multiple overlapping image data" according to the technology of the present disclosure.

[0226] 26, the output circuit 462D outputs left side image data 70A1 to the first signal processing unit 450 via the first output line 53, similar to the output circuit 62D described in the first embodiment. Moreover, the output circuit 462D outputs right side image data 70B1 to the second signal processing unit 452 via the second output line 55, similar to the output circuit 62D described in the first embodiment.

[0227] The first signal processing unit 450 performs specific signal processing on the input left side image data 70A1, and the second signal processing unit 452 performs specific signal processing on the input right side image data 70B1 and outputs the right side image data 70B1 after the specific signal processing to the first signal processing unit 450.

[0228] The first signal processor 450 synthesizes the left image data 70A1 that has been subjected to specific signal processing and the right image data 70B1 input from the second signal processor 452, and outputs the result in the first embodiment. The first signal processing unit 450 generates the above-described composite image data 72. The first signal processing unit 450 outputs the generated composite image data 72 to the controller .

[0229] 27, when a display moving image capture mode signal 480B is input from the controller 46 via the communication line 60, the processing circuit 462 operates the image processing circuit 462C in the same manner as the image processing circuit 362C described in the third embodiment. That is, the processing circuit 462 causes the image processing circuit 462C to compress the captured image data 70 into vertically thinned image data 73, and then separates the vertically thinned image data 73 into odd-numbered column image data 73A and even-numbered column image data 73B. The processing circuit 462 then causes the image processing circuit 462C to output the odd-numbered column image data 73A and the even-numbered column image data 73B to the output circuit 462D.

[0230] 27, like the output circuit 362D described in the third embodiment, the output circuit 462D outputs odd-numbered column image data 73A to the first signal processing unit 450 via the first output line 53. Also, like the output circuit 362D described in the third embodiment, the output circuit 462D outputs even-numbered column image data 73B to the second signal processing unit 452 via the second output line 55.

[0231] The first signal processing unit 450 performs specific signal processing on the input odd-numbered column image data 73A, and the second signal processing unit 452 performs specific signal processing on the input even-numbered column image data 73B and outputs the even-numbered column image data 73B after the specific signal processing to the first signal processing unit 450.

[0232] The first signal processing unit 450 combines the odd-numbered column image data 73A that has been subjected to specific signal processing with the even-numbered column image data 73B input from the second signal processing unit 452 to generate the combined image data 372 described in the above third embodiment. The first signal processing unit 450 outputs the generated combined image data 372 to the controller 46.

[0233] Next, the operation of the imaging device 400 will be described.

[0234] First, the flow of imaging processing executed by the processing circuit 462 of the image sensor 444 will be described with reference to Fig. 28. For convenience of explanation, it is assumed that the imaging device 400 is set to a still image imaging mode or a display moving image imaging mode.

[0235] In the imaging process shown in Fig. 28, first, in step ST300, the image processing circuit 462C judges whether the imaging device 400 is in a still image imaging mode. In step ST300, if the imaging device 400 is not in a still image imaging mode, that is, if the imaging device 400 is in a display moving image imaging mode, the judgment is negative and the imaging process proceeds to step ST304. In step ST300, if the imaging device 400 is in a still image imaging mode, the judgment is positive and the imaging process proceeds to step ST302.

[0236] In step ST302, the processing circuit 462 executes imaging processing for a still image, and then the imaging processing proceeds to step ST306. The imaging processing for a still image is similar to the imaging processing (see FIG. 10) described in the first embodiment.

[0237] In step ST304, the processing circuit 462 executes imaging processing for a display moving image, and then the imaging processing proceeds to step ST306. The imaging processing for a display moving image is similar to the imaging processing (see FIG. 22) described in the third embodiment.

[0238] In step ST306, the processing circuit 462 determines whether or not the imaging process end condition is satisfied. In step ST306, if the imaging process end condition is not satisfied, the determination is negative, and the imaging process proceeds to step ST300. In step ST306, if the imaging process end condition is satisfied, the determination is positive, and the imaging process ends.

[0239] In the still image imaging mode, the first signal processing unit 450 executes the same processing as the first signal processing (see FIG. 11) described in the first embodiment. In the still image imaging mode, the second signal processing unit 452 executes the same processing as the second signal processing (see FIG. 12) described in the first embodiment.

[0240] On the other hand, in the display moving image imaging mode, the first signal processing unit 450 executes the same processing as the first signal processing (FIG. 23) described in the third embodiment. Also, in the display moving image imaging mode, the second signal processing unit 452 executes the same processing as the second signal processing (FIG. 24) described in the third embodiment.

[0241] As described above, in the fourth embodiment, in the still image imaging mode, the captured image data 70 is separated into the left image data 70A1 and the right image data 70B1 described in the first embodiment. Also, in the display moving image imaging mode, the captured image data 70 is separated into the odd column image data 73A and the even column image data 73B described in the third embodiment. This makes it possible to achieve a different balance between image quality, power consumption, and processing speed between the still image imaging mode and the display moving image imaging mode.

[0242] In addition, in the still image imaging mode, the captured image data 70 is separated into left image data 70A1 and right image data 70B1, each of which includes image data indicating the overlapping region 71. In contrast, in the display moving image imaging mode, the captured image data 70 is separated in units of lines. Therefore, in the still image imaging mode, processing is performed on the left image data 70A1 and right image data 70B1, each of which includes image data indicating the overlapping region 71, so that the image quality can be improved compared to the display moving image imaging mode. In addition, in the display moving image imaging mode, processing is performed on the odd column image data 73A and the even column image data 73B, which have a smaller data amount than the left image data 70A1 and the right image data 70B1. Therefore, in the display moving image imaging mode, power consumption can be reduced and processing speed can be increased compared to the still image imaging mode.

[0243] In the above third embodiment, the case where the imaging process (see FIG. 10) described in the above first embodiment is applied as the process of step ST302 (see FIG. 28) has been described, but the technology of the present disclosure is not limited thereto. For example, the imaging process (see FIG. 15) described in the above second embodiment may be applied as the process of step ST302 (see FIG. 28). In this case, in the still image imaging mode, the first signal processing unit 450 executes the first signal processing (see FIG. 16) described in the above second embodiment, and the second signal processing unit 452 executes the second signal processing (see FIG. 17) described in the above second embodiment.

[0244] In the third embodiment, the captured image data 70 is separated into the left image data 70A1 and the right image data 70B1 in the still image imaging mode, but the technology of the present disclosure is not limited to this. For example, in the still image imaging mode, the captured image data 70 may be separated into the background area image data 70A2 and the face area image data 70B2 as described in the second embodiment. In this case, in the still image imaging mode, the first signal processing unit 450 may be operated in the same manner as the first signal processing unit 250 described in the second embodiment, and the second signal processing unit 452 may be operated in the same manner as the second signal processing unit 252 described in the second embodiment.

[0245] [Fifth embodiment] In the third embodiment, an example in which the captured image data 70 is compressed into odd-numbered row image data 73A and even-numbered row image data 73B has been described, but in the fifth embodiment, an example in which the captured image data 70 is compressed into two pieces of image data by another method will be described. In the fifth embodiment, the same components as those in the third embodiment are denoted by the same reference numerals, and the description thereof will be omitted. The following describes the parts that are different from the third embodiment.

[0246] As shown in FIG. 1, an imaging device 500 according to the fifth embodiment differs from the imaging device 300 described in the third embodiment in that an imaging device body 512 is provided instead of the imaging device body 312.

[0247] The imaging device main body 512 differs from the imaging device main body 312 in that it has an imaging element 544 (see FIG. 29) instead of the imaging element 344. The imaging element 544 differs from the imaging element 344 in that it has a processing circuit 562 instead of the processing circuit 362. The processing circuit 562 differs from the processing circuit 362 in that it has an image processing circuit 562C instead of the image processing circuit 362C, and in that it has an output circuit 562D instead of the output circuit 362D.

[0248] The imaging device main body 512 differs from the imaging device main body 312 in that it has a first signal processing unit 550 (see FIG. 29) instead of the first signal processing unit 50.

[0249] The memory 64 stores captured image data 570, and the image processing circuit 562C acquires the captured image data 570 from the memory 64. The captured image data 570 is color image data having R, G, and B pixels. As an example, as shown in Fig. 30, in the captured image data 570, the R, G, and B pixels are arranged with a periodicity corresponding to the Bayer array.

[0250] 30, in the first row, R pixels and G pixels are arranged in a cyclical manner in the row direction in the order of R pixels and G pixels. In addition, in the second row, B pixels and G pixels are arranged in a cyclical manner in the row direction in the order of G pixels and B pixels. The arrangement pattern of R pixels and G pixels in the first row is repeated every other row in the column direction, and the arrangement pattern of B pixels and G pixels in the second row is repeated every other row in the column direction, thereby forming the arrangement pattern of R pixels, G pixels, and B pixels of the entire captured image data 570.

[0251] The image processing circuit 562C compresses the captured image data 570 acquired from the memory 64. That is, the image processing circuit 562C generates vertically thinned image data 573 from the captured image data 570. As an example, as shown in Fig. 30, the vertically thinned image data 573 is image data representing a vertical 1 / 2 thinned image obtained by thinning out every two lines of two rows adjacent in the column direction from the captured image represented by the captured image data 570.

[0252] 29 and 31, the image processing circuit 562C separates the vertically thinned image data 753 into first horizontally thinned image data 573A and second horizontally thinned image data 573B. The image processing circuit 562C outputs the first horizontally thinned image data 573A and second horizontally thinned image data 573B obtained by the separation to the output circuit 562D.

[0253] 31, the first horizontally thinned image data 573A is image data showing one of a pair of horizontally 1 / 2 thinned images obtained by alternately thinning out every other column in units of two columns in the row direction from the vertically 1 / 2 thinned image shown by the vertically thinned image data 573. The second horizontally thinned image data 573B is image data showing the other of the pair of horizontally 1 / 2 thinned images.

[0254] The output circuit 562D converts the first horizontally thinned image data 5 The output circuit 562D outputs the second horizontally thinned image data 573A input from the image processing circuit 562C to the first signal processing unit 550 via the first output line 53. The output circuit 562D also outputs the second horizontally thinned image data 573B input from the image processing circuit 562C to the second signal processing unit 52 via the second output line 55.

[0255] In the second signal processing unit 52, processing similar to that performed on the even column image data 73B described in the third embodiment above is performed on the second horizontally thinned image data 573B, and the processed second horizontally thinned image data 573B is transmitted to the first signal processing unit 550.

[0256] The first signal processing unit 550 receives the second horizontally thinned image data 573B transmitted from the second signal processing unit 52.

[0257] In the first signal processing unit 550, the first horizontally thinned image data 573A is subjected to the same processing as that performed on the odd-numbered column image data 73A described in the third embodiment. Then, in the first signal processing unit 550, the first horizontally thinned image data 573A and the second horizontally thinned image data 573B are combined to generate composite image data 572. As a result, the arrangement pattern of R pixels, G pixels, and B pixels of the image represented by the composite image data 572 is the same as the arrangement pattern of the vertically 1 / 2 thinned image represented by the vertically thinned image data 573. In other words, the arrangement pattern of R pixels, G pixels, and B pixels of the image represented by the composite image data 572 is an arrangement pattern having a periodicity that allows demosaic processing of the R pixels, G pixels, and B pixels.

[0258] Therefore, as in the third embodiment described above, the first signal processing unit 550 performs demosaic processing of the R, G, and B signals on the composite image data 572, and outputs the demosaic-processed composite image data 572 to the controller 46 via the communication line 60.

[0259] As described above, the Bayer array is adopted for the captured image data 570. In this case, as with the odd-numbered column image data 73A and the even-numbered column image data 73B described in the third embodiment, the first horizontally thinned image data 573A and the second horizontally thinned image data 573B are obtained as two image data that can be demosaiced. Therefore, even if the captured image data 570 is image data in the Bayer array, the same effect as in the third embodiment can be obtained.

[0260] In the above fifth embodiment, the first horizontally thinned image data 573A and the second horizontally thinned image data 573B are exemplified as two pieces of image data obtained by compressing the captured image data 570, but the technology of the present disclosure is not limited thereto. For example, as shown in Fig. 32, the image processing circuit 562C may separate the captured image data 570 into higher-order bit data 570C and lower-order bit data 570D. When the number of bits per pixel for the captured image data 570 is 16 bits, the higher-order bit data 570C is, for example, image data having the number of bits per pixel of the higher 8 bits, and the lower-order bit data 570D is, for example, image data having the number of bits per pixel of the lower 8 bits.

[0261] As shown in FIG. 32, the output circuit 562D outputs the higher-order bit data 570C to the first signal processing unit 550 via the first output line 53, and outputs the lower-order bit data 570D to the second signal processing unit 52 via the second output line 55. Similarly to the fifth embodiment, the second signal processing unit 52 performs specific signal processing on the lower-order bit data 570D and then transmits the lower-order bit data 570D to the first signal processing unit 550 via the communication line 58. Similarly to the fifth embodiment, the first signal processing unit 550 receives the lower-order bit data 570D transmitted from the second signal processing unit 52 and performs specific signal processing on the higher-order bit data 570C. Similarly to the fifth embodiment, the first signal processing unit 550 combines the higher-order bit data 570C that has been subjected to the specific signal processing with the received lower-order bit data 570D to obtain a signal output signal. The first signal processing unit 550 outputs the generated composite image data 572A to the controller 46.

[0262] Note that, although the captured image data 570 is illustrated here as having a bit number of 16 bits per pixel, the present invention is not limited to this, and the image data may have a bit number of less than 16 bits per pixel, or may have a bit number of more than 16 bits per pixel. Also, the method of dividing the upper bits and the lower bits may be determined according to, for example, the application and / or specifications.

[0263] Here, an example has been described in which the image processing circuit 562C separates the captured image data 570 into the higher-order bit data 570C and the lower-order bit data 570D, but the technology of the present disclosure is not limited to this. For example, the captured image data 570 may be separated into higher-order bit image data, middle-order bit image data, and lower-order bit image data. The higher-order bit image data, middle-order bit image data, and lower-order bit image data refer to three compressed image data obtained by compressing the captured image data 570 by dividing it into three bit ranges. The captured image data 570 may also be compressed by dividing it into four or more bit ranges. In this way, the captured image data 570 may be divided into a plurality of bit ranges to obtain a plurality of compressed image data.

[0264] In this way, the captured image data 570 is separated into a plurality of compressed image data (higher bit data 570C and lower bit data 570D in the example shown in FIG. 32) obtained by dividing the captured image data 570 into a plurality of bit ranges. This allows each of the first signal processing unit 550 and the second signal processing unit 52 to perform faster processing than when processing image data obtained by irregular division.

[0265] 32, by separating the captured image data 570 into higher order bit data 570C and lower order bit data 570D, it is possible to perform processing with higher accuracy on the higher order bit data 570C than on the lower order bit data 570D. On the other hand, it is possible to reduce power consumption and increase the processing speed on the lower order bit data 570D compared to the higher order bit data 570C.

[0266] In addition, in each of the above embodiments, an example in which the captured image data 70, 570 (hereinafter simply referred to as "captured image data") is separated into two pieces of image data has been described, but the technology of the present disclosure is not limited to this. For example, when N is a natural number of 2 or more, the captured image data may be separated into N pieces of image data. As a separation method, for example, a method of dividing the captured image data into N equal parts, or a method of separating the captured image data into N bit ranges, etc., may be considered.

[0267] When the captured image data is separated into N pieces of image data, for example, as shown in Fig. 33, each of the first signal processing unit 650 to the Nth signal processing unit 650N is connected to the imaging element 44 via a corresponding output line. In the example shown in Fig. 33, the first signal processing unit 650 is connected to the imaging element 44 via the first output line 53, and the Nth signal processing unit 650N is connected to the imaging element 44 via the Nth output line 53N. The Nth signal processing unit 650N transmits image data that has been subjected to signal processing to the first signal processing unit 650, and the first signal processing unit 650 combines the N pieces of image data and outputs the combined combined image data to the controller 46.

[0268] In addition, in each of the above embodiments, the processing circuits 62, 262, 362, 462, and 562 (hereinafter simply referred to as "processing circuits") are implemented by ASICs. However, the technology of the present disclosure is not limited to this. For example, the above-described imaging process may be implemented by a computer. This may be realized by a software configuration.

[0269] In this case, for example, as shown in FIG. 34, an imaging processing program 802 for causing a computer 652 built in the imaging element 44, 244, 344, 444, 544 to execute the above-mentioned imaging processing is stored in a storage medium 800. The computer 652 includes a CPU 652A, a ROM 652B, and a RAM 652C. The imaging processing program 802 in the storage medium 800 is installed in the computer 652, and the CPU 652A of the computer 652 executes the above-mentioned imaging processing according to the imaging processing program 802. Here, a single CPU is exemplified as the CPU 652A, but the technology of the present disclosure is not limited to this, and multiple CPUs may be adopted instead of the CPU 652A. An example of the storage medium 800 is any portable storage medium such as an SSD or a USB memory.

[0270] 34, the image capturing processing program 802 is stored in the storage medium 800, but the technology of the present disclosure is not limited to this. For example, the image capturing processing program 802 may be stored in advance in the ROM 652B, and the CPU 652A may read out the image capturing processing program 802 from the ROM 652B, load it into the RAM 652C, and execute the loaded image capturing processing program 802.

[0271] Furthermore, the imaging processing program 802 may be stored in a storage unit of another computer or a server device or the like connected to the computer 652 via a communication network (not shown), and the imaging processing program 802 may be downloaded to the computer 652 in response to a request from an imaging device 700 having a similar configuration to any one of the imaging devices 10, 200, 300, 400, 500. In this case, the downloaded imaging processing program 802 is executed by the CPU 652A of the computer 652.

[0272] Moreover, the computer 652 may be provided outside the image pickup device 44, 244, 344, 444, 544. In this case, the computer 652 may control the processing circuit according to the image pickup processing program 802.

[0273] As the hardware resource for executing the imaging process described in each of the above embodiments, the following various processors can be used. As the processor, for example, as described above, there is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the imaging process by executing software, i.e., a program. In addition, as the processor, there is a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing a specific process, such as an FPGA, a PLD, or an ASIC.

[0274] The hardware resource for executing the imaging process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the various processes related to the technology of the present disclosure may be a single processor.

[0275] As an example of a configuration using one processor, first, there is a form in which one processor is configured by combining one or more CPUs and software, as typified by computers such as clients and servers, and this processor functions as a hardware resource that executes processing within the image sensor. Secondly, there is a form in which a processor is used that realizes the functions of the entire system, including multiple hardware resources that execute image processing, on a single IC chip, as typified by SoC (System-on-a-chip). In this way, processing within the image sensor is realized using one or more of the various processors mentioned above as hardware resources. do.

[0276] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0277] In addition, in each of the above embodiments, a lens-interchangeable camera is exemplified as an imaging device, but the technology of the present disclosure is not limited thereto. For example, the technology of the present disclosure may be applied to a smart device 900 shown in FIG. 35. As an example, the smart device 900 shown in FIG. 35 is an example of an imaging device according to the technology of the present disclosure. The smart device 900 is equipped with the imaging elements 44, 244, 344, 444, and 544 described in the above embodiments. Even with the smart device 900 configured in this way, the same action and effect as the imaging device described in the above embodiments can be obtained. Note that the technology of the present disclosure is applicable not only to the smart device 900 but also to a personal computer or a wearable terminal device.

[0278] In addition, in each of the above-described embodiments, the first display 32 and the second display 86 are exemplified, but the technology of the present disclosure is not limited thereto. For example, a separate display that is later attached to the imaging device body 12 may be used as the "display unit" according to the technology of the present disclosure.

[0279] The above-mentioned imaging process, first signal process, and second signal process are merely examples, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of the processes may be changed without departing from the spirit of the invention.

[0280] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0281] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0282] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

[0283] The following supplementary notes are further disclosed regarding the above embodiment.

[0284] (Appendix 1) An imaging device (10, 200, 300, 400, 500, 700) having an imaging element (44, 244, 344, 444, 544), a storage unit (64) that stores captured image data (70, 570) obtained by capturing an image of a subject using the imaging element (44, 244, 344, 444, 544) and is built into the imaging element (44, 244, 344, 444, 544); a processing unit (62, 262, 362, 462, 562) that processes captured image data (70, 570) and is built into an image sensor (44, 244, 344, 444, 544); an output unit (62D, 262D, 362D, 462D, 562D) that outputs processed image data obtained by processing the captured image data (70, 570) by the processing unit (62, 262, 362, 462, 562) and is built in the image sensor (44, 244, 344, 444, 544); A plurality of signal processing units (50, 250, 350, 450, 550, 650, 52, 252, 452) provided outside the imaging element, The processing unit (62, 262, 362, 462, 562) performs processing to divide the captured image data (70, 570) stored in the storage unit (64) into a plurality of image data (70A, 70B); The output unit (62D, 262D, 362D, 462D, 562D) has a plurality of output lines (53, 55) provided corresponding to each of the plurality of signal processing units (50, 250, 350, 450, 550, 650, 52, 252, 452), and outputs each of the plurality of image data (70A, 70B) as processed image data from the plurality of output lines (53, 55) to a corresponding signal processing unit among the plurality of signal processing units (50, 250, 350, 450, 550, 650, 52, 252, 452, 750, 752); An imaging device (10, 200, 300, 400, 500, 700) in which any one of a plurality of signal processing units (50, 250, 350, 450, 550, 650, 52, 252, 452, 750, 752) combines a plurality of image data (70A, 70B).

[0285] (Appendix 2) An imaging device (200) as described in Appendix 1, wherein each of the multiple image data (70A, 70B) is image data indicating an image having an overlapping area (71, 271) between adjacent images (70A1a, 70B1a, 70A2a, 70B2a) among multiple images (70A1a, 70B1a, 70A2a, 70B2a) based on each of the multiple image data (70A, 70B).

[0286] (Appendix 3) The imaging device (200) according to appendix 1 or 2, wherein the plurality of images (70A2a, 70B2a) are divided into a designated image (70B2a) and an image different from the designated image.

[0287] (Appendix 4) The processing unit detects facial image data indicating a facial image (69) from the captured image data (70), and the specified image (70B2a) is an image including the facial image (69) indicated by the facial image data detected by the processing unit (262) from the captured image indicated by the captured image data (70), in the imaging device (200) described in Appendix 3.

[0288] (Appendix 5) The imaging device according to any one of claims 1 to 4, wherein the processing unit changes how captured image data is divided between a recording imaging mode and a display moving image imaging mode.

[0289] (Appendix 6) The processing unit (462) divides the captured image data (70) into a plurality of overlapping image data (70A1, 70B1) as a plurality of image data in the recording imaging mode, and divides the captured image data (70) by line in the display moving image imaging mode, 6. The imaging device according to claim 5, wherein each of the plurality of overlapping image data is image data representing an image having an overlapping area (71) between adjacent images among the plurality of images.

Claims

1. An imaging element, a memory that stores image data obtained by capturing an image of a subject using the image sensor; a processor; the processor includes an output circuit; the output circuit has a plurality of output lines, and outputs, from the plurality of output lines, each of a plurality of image data into which the captured image data stored in the memory is divided; The imaging element is a stacked type imaging element in which a photoelectric conversion element, the processor, and the memory are integrated into a single chip, and the processor and the memory are stacked on the photoelectric conversion element. Image sensor.

2. The imaging element according to claim 1 , wherein each of the plurality of image data is image data representing an image having an overlapping area between adjacent images among images based on each of the plurality of image data.

3. 3. The image sensor according to claim 1, wherein the plurality of images based on the plurality of image data are divided into a designated image and an image different from the designated image.

4. a detection processor for detecting face image data indicative of an image of a face from the captured image data; The imaging element according to claim 3 , wherein the specified image is an image including an image of a face represented by the face image data detected by the detection processor from the captured image represented by the captured image data.

5. 5. The image sensor according to claim 1, wherein the captured image data is divided in a different manner between an image capture mode for recording and an image capture mode for displaying a moving image.

6. 6. The image sensor according to claim 5, wherein the captured image data is divided into a plurality of overlapped image data as the plurality of image data in the record imaging mode, and the captured image data is divided on a line-by-line basis in the display moving image imaging mode.

7. The imaging element according to claim 6 , wherein each of the plurality of overlapping image data is image data representing an image having an overlapping area between adjacent images among images based on each of the plurality of image data.

8. 8. The imaging element according to claim 5, wherein the recording imaging mode is an operation mode for causing the imaging element to capture a still image.

9. the captured image data is color image data representing a color captured image in which a plurality of primary color pixels are periodically arranged, The color image data is divided into a plurality of primary color pixel array image data as the plurality of image data, 9. The imaging element according to claim 1, wherein each of the plurality of primary color pixel array image data is image data representing an image in which the plurality of primary color pixels are periodically arranged.

10. The image sensor according to claim 9 , wherein the plurality of primary color pixel array image data are a plurality of divided image data obtained by thinning out and then dividing the color image data.

11. The plurality of divided image data are obtained by thinning out the color image data on a row-by-row basis.

11. The image sensor according to claim 10, wherein the thinned-out image data includes odd-numbered column image data indicating a set of pixels in odd-numbered columns and even-numbered column image data indicating a set of pixels in even-numbered columns.

12. 2. The image sensor according to claim 1, wherein the plurality of pieces of image data are a plurality of pieces of compressed image data obtained by compressing the captured image data by dividing the captured image data into a plurality of bit ranges.

13. The image pickup device according to claim 12 , wherein the plurality of compressed image data are image data of most significant bits and image data of least significant bits of the captured image data.

14. An image data processing method for an image sensor including a memory that stores captured image data obtained by capturing an image of a subject, and a processor, comprising: outputting, from a plurality of output lines, each of a plurality of image data into which the captured image data stored in the memory is divided; the processor includes an output circuit; the output circuit has the plurality of output lines; The imaging element is a stacked type imaging element in which a photoelectric conversion element, the processor, and the memory are integrated into a single chip, and the processor and the memory are stacked on the photoelectric conversion element. A method for processing image data from an imaging element.

15. A program for causing a computer to execute a process, the program being applied to an image sensor including a memory for storing captured image data obtained by capturing an image of a subject, the processor including an output circuit, the output circuit having a plurality of output lines, The process comprises: outputting, from the plurality of output lines, each of a plurality of image data into which the captured image data stored in the memory is divided; The imaging element is a stacked type imaging element in which a photoelectric conversion element, the processor, and the memory are integrated into a single chip, and the processor and the memory are stacked on the photoelectric conversion element. program.

Images