Camera device, image data processing method of camera device, and storage medium

Abstract

The camera device of the present invention includes: a storage unit, which stores camera image data obtained by photographing a subject with an imaging element and is built into the imaging element; an output unit, which is built into the imaging element; and multiple signal processing units, which are arranged outside the imaging element, the output unit has multiple output lines respectively arranged corresponding to each of the multiple signal processing units, and multiple image data divided from the camera image data stored in the storage unit are output from the multiple output lines to corresponding signal processing units in the multiple signal processing units, respectively, and any one of the multiple signal processing units synthesizes the multiple image data.

Description

Technical Field

[0001] The technology of the present invention relates to an imaging device, an image data processing method of the imaging device, and a program. Background Art

[0002] Japanese Patent Publication No. 2016-158294 discloses an electronic device including an imaging element, an image processing unit, and a control unit. The imaging element has a first imaging area and a second imaging area. In the imaging element, pixels are arranged in the first imaging area and the second imaging area. The imaging element shoots in the first imaging area under a first imaging condition, and shoots in the second imaging area under a second imaging condition different from the first imaging condition. Image data composed of pixel signals of the first imaging area and image data composed of pixel signals of the second imaging area are output to a subsequent image processing unit via an output line of a system.

[0003] The image processing unit generates first image data by performing various image processing on image data composed of pixel signals of the first imaging area, and generates second image data by performing various image processing on image data composed of pixel signals of the second imaging area. The control unit causes the display unit to display a live preview image or a still image synthesized by the first image represented by the first image data and the second image represented by the second image data. Summary of the invention

[0004] One embodiment of the technology involved in the present invention provides an imaging device, an image data processing method of the imaging device, and a program that can realize high-speed image processing compared to the case where image data is output from an imaging element to multiple signal processing units through only one system of output lines.

[0005] Means for solving technical problems

[0006] The first aspect of the present invention is a camera device having an imaging element, the camera device including: a storage unit that stores image data obtained by photographing a subject by the imaging element and is built into the imaging element; an output unit that is built into the imaging element; and a plurality of signal processing units that are provided outside the imaging element, the output unit having a plurality of output lines that are respectively provided corresponding to each of the plurality of signal processing units, and outputting a plurality of image data obtained by dividing the image data stored in the storage unit from the plurality of output lines to corresponding signal processing units in the plurality of signal processing units, and any one of the plurality of signal processing units synthesizing the plurality of image data. Thus, high-speed image processing can be achieved compared to a case where image data is output from the imaging element to the plurality of signal processing units through only one system of output lines.

[0007] In the imaging device according to the second aspect of the present invention, the plurality of image data are image data representing images having overlapping regions between adjacent images in each of the images based on the plurality of image data. Thus, compared with the case where two images obtained by simply dividing the captured image into two are connected, the boundary region between the two images is suppressed from becoming obvious.

[0008] In the imaging device according to the third aspect of the present invention, a plurality of images are divided into a designated image and an image different from the designated image, thereby suppressing the boundary region between the designated image and the image different from the designated image from becoming conspicuous.

[0009] In the imaging device according to the fourth aspect of the technology of the present invention, a detection unit is further included, which detects facial image data representing a facial image from the captured image data, and the designated image is an image including a facial image represented by the facial image data detected by the detection unit in the captured image represented by the captured image data. Thus, the boundary area between the image including the facial image and the image not including the facial image is suppressed from becoming obvious.

[0010] In the imaging device according to the fifth aspect of the present invention, the method of dividing the image data differs between the recording imaging mode and the animation display imaging mode. Thus, the balance between image quality, power consumption and processing speed can be made different between the recording imaging mode and the animation display imaging mode.

[0011] In the imaging device according to the sixth aspect of the present invention, in the recording shooting mode, the captured image data is divided into a plurality of repeated image data as a plurality of image data, and in the display animation shooting mode, the captured image data is divided in line units. Thus, in the recording shooting mode, the image quality can be improved compared to the display animation shooting mode, and in the display animation shooting mode, the power consumption can be suppressed and the processing speed can be increased compared to the recording shooting mode.

[0012] In the imaging device according to the seventh aspect of the present invention, the plurality of overlapping image data are image data representing images having overlapping regions between adjacent images among the plurality of images. Thus, compared with the case where two images obtained by simply dividing the captured image into two are connected, the boundary region between the two images is suppressed from becoming obvious.

[0013] In the imaging device according to the eighth aspect of the present invention, the recording shooting mode is an operation mode in which the imaging element performs still image shooting. Thus, the balance between image quality, power consumption and processing speed can be made different between the still image shooting operation mode and the video display shooting mode.

[0014] In the imaging device according to the ninth aspect of the present invention, the imaging image data is color image data representing a color imaging image in which a plurality of primary color pixels are periodically arranged, and the color image data is divided into a plurality of primary color pixel arrangement image data as a plurality of image data, and the plurality of primary color pixel arrangement image data are image data representing an image in which a plurality of primary color pixels are periodically arranged. Thus, even when the color image data is divided into a plurality of primary color pixel arrangement image data, demosaicing processing can be performed on a plurality of primary color pixels.

[0015] In the camera device according to the tenth aspect of the present invention, the plurality of primary color pixel arrangement image data are a plurality of segmented image data obtained by segmenting the color image data after thinning out the color image data. Thus, high-speed processing can be achieved compared to the case where the plurality of image data obtained by segmenting the color image data without thinning out the color image data is processed by a plurality of signal processing units.

[0016] In the imaging device according to the 11th aspect of the technology of the present invention, the plurality of divided image data are odd-numbered column image data representing a set of pixels of odd-numbered columns and even-numbered column image data representing a set of pixels of even-numbered columns in the thinned image data obtained by thinning the color image data in units of rows. Thus, each of the plurality of signal processing units can perform high-speed processing compared to the case of processing image data obtained by irregular division.

[0017] In the imaging device according to the twelfth aspect of the present invention, any one of the plurality of signal processing units performs demosaic processing on the synthesized image data obtained by synthesizing the plurality of image data. This makes it possible to obtain a higher quality image than when demosaic processing is not performed.

[0018] In the camera device according to the 13th aspect of the present invention, the plurality of image data are compressed image data obtained by dividing the camera image data within a plurality of bit ranges and compressing them. Thus, each of the plurality of signal processing units can perform high-speed processing compared to the case where image data obtained by irregular division is processed.

[0019] In the camera device according to the 14th aspect of the technology of the present invention, the plurality of compressed image data are high-order image data and low-order image data in the captured image data. Thus, the high-order image data can be processed with higher precision than the low-order image data, and the power consumption of the low-order image data can be suppressed more than that of the high-order image data, and the processing speed can be increased.

[0020] In the imaging device according to the fifteenth aspect of the present invention, the imaging element is an imaging element in which at least the photoelectric conversion element and the storage unit are integrated into a single chip. This can improve the portability of the imaging element compared to an imaging element in which the photoelectric conversion element and the storage unit are not integrated into a single chip.

[0021] In the imaging device according to the 16th aspect of the present invention, the imaging element is a stacked imaging element in which the storage unit is stacked on the photoelectric conversion element. Thus, compared with a case where the photoelectric conversion element and the storage unit are not stacked, the load imposed by the processing between the photoelectric conversion element and the storage unit is reduced.

[0022] The imaging device according to the seventeenth aspect of the present invention further includes a control unit that controls the display unit to display images based on the plurality of image data output by the output unit, thereby enabling a user to visually recognize images based on the plurality of image data output by the output unit.

[0023] The image data processing method involved in the 18th aspect of the technology of the present invention is an image data processing method of an image pickup device, the image pickup device including: an imaging element; a storage unit that stores imaged image data obtained by photographing a subject by the imaging element and is built into the imaging element; an output unit that is built into the imaging element; and a plurality of signal processing units that are provided outside the imaging element, the output unit having a plurality of output lines that are respectively provided corresponding to each of the plurality of signal processing units, the image data processing method including the following processing: a plurality of image data divided into the imaged image data stored in the storage unit are respectively outputted from the plurality of output lines to corresponding signal processing units in the plurality of signal processing units, and any one of the plurality of signal processing units synthesizes the plurality of image data. Thus, high-speed image processing can be achieved compared to the case where the image data is outputted from the imaging element to the plurality of signal processing units through only one system of output lines.

[0024] The program involved in the 19th aspect of the technology of the present invention is a program for causing a computer to function as an output unit included in an imaging device, the imaging device including: an imaging element; a storage unit that stores the image data obtained by photographing a subject by the imaging element and is built into the imaging element; an output unit that is built into the imaging element; and a plurality of signal processing units that are provided outside the imaging element, the output unit having a plurality of output lines that are respectively provided corresponding to each of the plurality of signal processing units, and outputting a plurality of image data divided from the image data stored in the storage unit from the plurality of output lines to the corresponding signal processing units in the plurality of signal processing units, and any one of the plurality of signal processing units synthesizing the plurality of image data. Thus, high-speed image processing can be achieved compared to the case where the image data is output from the imaging element to the plurality of signal processing units through only one system of output lines.

[0025] The imaging device according to the 20th aspect of the technology of the present invention has an imaging element, and the imaging device includes: a memory that stores the image data obtained by photographing a subject by the imaging element, and is built into the imaging element; a processor that is built into the imaging element; and a plurality of signal processing units that are provided outside the imaging element, the processor having a plurality of output lines that are respectively provided corresponding to each of the plurality of signal processing units, and outputting a plurality of image data obtained by dividing the image data stored in the memory from the plurality of output lines to the corresponding signal processing units in the plurality of signal processing units, and any one of the plurality of signal processing units synthesizes the plurality of image data. Thus, compared with the case where the image data is output from the imaging element to the plurality of signal processing units through only one system of output lines, high-speed image processing can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a perspective view showing the appearance of the imaging device.

[0027] Figure 2 It is a rear view showing the rear side of the imaging device.

[0028] Figure 3 It is a block diagram showing the structure of an imaging device.

[0029] Figure 4 This is a schematic structural diagram showing the structure of an imaging element.

[0030] Figure 5 It is a block diagram showing the structure of an imaging element.

[0031] Figure 6 This is a block diagram showing the flow of data in the imaging device according to the first embodiment.

[0032] Figure 7 A conceptual diagram showing the characteristics of the left image and the right image included in the captured image.

[0033] Figure 8 This is a block diagram showing a controller and UI system devices included in the imaging device, and the peripheral configuration of the controller and UI system devices.

[0034] Fig. 9 This is a schematic structural diagram showing the structure of a hybrid viewfinder.

[0035] Fig.10 This is a flowchart showing the flow of imaging processing according to the first embodiment.

[0036] Fig.11 This is a flowchart showing the first signal processing flow involved in the first embodiment.

[0037] Fig.12 This is a flowchart showing the flow of the second signal processing according to the first embodiment.

[0038] Fig.13 This is a block diagram showing the flow of data in the imaging device according to the second embodiment.

[0039] Fig.14A This is an image diagram showing the relationship between the face area image and the background area image.

[0040] Fig. 14B This is an image diagram showing the relationship between the face region image, the background region image, and the overlapping region.

[0041] Fig.15 This is a flowchart showing the flow of imaging processing according to the second embodiment.

[0042] Fig.16 This is a flowchart showing the first signal processing flow involved in the second embodiment.

[0043] Fig.17 This is a flowchart showing the second signal processing flow involved in the second embodiment.

[0044] Fig.18 This is a block diagram showing the flow of data in the imaging device according to the third embodiment.

[0045] Fig.19 This is an explanatory diagram for explaining a method of generating vertical thinning image data from captured image data.

[0046] Fig. 20 This is a conceptual diagram showing the relationship between vertical thinning image data, odd-numbered column image data, and even-numbered column image data.

[0047] Fig.21 This is a conceptual diagram showing a method of separating vertical thinning image data into odd-numbered column image data and even-numbered column image data.

[0048] Fig. 22 This is a flowchart showing the flow of imaging processing according to the third embodiment.

[0049] Fig.23 This is a flowchart showing the first signal processing flow involved in the third embodiment.

[0050] Fig.24 This is a flowchart showing the second signal processing flow involved in the third embodiment.

[0051] Fig.25 This is a block diagram showing the flow of data in the imaging device according to the third embodiment.

[0052] Fig.26 This is a block diagram showing the flow of data within the imaging device in which the still image shooting mode is set.

[0053] Fig. 27 This is a block diagram showing the flow of data within the imaging device in which the imaging mode for displaying an image is set.

[0054] Fig.28 This is a flowchart showing the flow of imaging processing according to the fourth embodiment.

[0055] Fig.29 This is a block diagram showing the flow of data within the imaging device according to the fourth embodiment.

[0056] Fig.30 This is an explanatory diagram for explaining a method of generating vertical thinning image data from captured image data in the Bayer arrangement.

[0057] Fig.31 It is a conceptual diagram showing the relationship among vertical thinning image data, first horizontal thinning image data, and second horizontal thinning image data.

[0058] Fig.32 This is a block diagram showing a method of separating captured image data into high-order data and low-order data and then processing them.

[0059] Fig.33 This is a conceptual diagram showing the relationship between an imaging element, a plurality of signal processing units, and a controller when three or more signal processing units are used.

[0060] Fig.34 This is a conceptual diagram showing an example of a method of installing an image pickup processing program from a storage medium storing the image pickup processing program into a computer in an imaging element.

[0061] Fig.35 This is a block diagram showing an example of a schematic configuration of a smart device incorporating an imaging element according to an embodiment. DETAILED DESCRIPTION

[0062] Hereinafter, an example of an embodiment of an imaging device according to the technology of the present invention will be described with reference to the drawings.

[0063] First, the meanings of terms used in the following description are explained.

[0064] Also, in the following description, CPU is the abbreviation of “Central Processing Unit”. Also, in the following description, RAM is the abbreviation of “Random Access Memory”. Also, in the following description, ROM is the abbreviation of “Read Only Memory”. Also, in the following description, DRAM is the abbreviation of “Dynamic Random Access Memory”. Also, in the following description, SRAM is the abbreviation of “Static Random Access Memory”.

[0065] Furthermore, in the following description, IC is an abbreviation of "Integrated Circuit". Furthermore, in the following description, LSI is an abbreviation of "Large-Scale Integration". Furthermore, in the following description, ASIC is an abbreviation of "Application Specific Integrated Circuit". Furthermore, in the following description, PLD is an abbreviation of "Programmable Logic Device". Furthermore, in the following description, FPGA is an abbreviation of "Field-Programmable Gate Array".

[0066] Furthermore, in the following description, SSD refers to the abbreviation of “Solid State Drive”. Furthermore, in the following description, DVD-ROM refers to the abbreviation of “Digital Versatile Disc Read Only Memory”. Furthermore, in the following description, USB refers to the abbreviation of “Universal Serial Bus”. Furthermore, in the following description, HDD refers to the abbreviation of “Hard Disk Drive”. Furthermore, in the following description, EEPROM refers to the abbreviation of “Electrically Erasable and Programmable Read Only Memory”.

[0067] Furthermore, in the following description, CCD refers to the abbreviation of “Charge Coupled Device”. Furthermore, in the following description, CMOS refers to the abbreviation of “Complementary Metal Oxide Semiconductor”. Furthermore, in the following description, EL refers to the abbreviation of “Electro-Luminescence”. Furthermore, in the following description, A / D refers to the abbreviation of “Analog / Digital”. Furthermore, in the following description, FIFO refers to the abbreviation of “First in First out”. Furthermore, in the following description, I / F refers to the abbreviation of “Interface”. Furthermore, in the following description, EIS refers to the abbreviation of “Electronics Image Stabilization”. Furthermore, in the following description, AF refers to the abbreviation of “Auto-Focus”. Furthermore, in the following description, AE refers to the abbreviation of “Automatic Exposure”. In addition, in the following description, UI is an abbreviation of "User Interface".

[0068] [First embodiment]

[0069] As an example, Figure 1 As shown, the imaging device 10 is a lens-interchangeable camera. The imaging device 10 is a digital camera, which includes an imaging device body 12, an interchangeable lens 14 replaceably mounted on the imaging device body 12, and a mirror is omitted.

[0070] Furthermore, a hybrid viewfinder (registered trademark) 16 is provided on the camera body 12. The hybrid viewfinder 16 mentioned here refers to a viewfinder that selectively uses an optical viewfinder (hereinafter referred to as "OVF") and an electronic viewfinder (hereinafter referred to as "EVF"), for example. In addition, OVF is an abbreviation of "optical viewfinder". And EVF is an abbreviation of "electronic viewfinder".

[0071] A viewfinder switching lever 18 is provided on the front surface of the camera body 12. By rotating the viewfinder switching lever 18 in the direction of arrow SW, the optical image that can be visually recognized by the OVF and the electronic image that can be visually recognized by the EVF, i.e., the live preview image, are switched. The "live preview image" referred to here refers to the image that is visually recognized by the photoelectric conversion element 61 (see FIG. 6 ) described later. Figure 3 and Figure 4) is captured and displayed as a dynamic image. The instant preview image is also commonly referred to as a live view image.

[0072] A release button 20 and a dial 22 are provided on the upper surface of the imaging device body 12. The dial 22 is operated when setting the operating mode of the imaging system and the operating mode of the playback system.

[0073] The release button 20 functions as a shooting preparation indication unit and a shooting indication unit, and can detect pressing operations in two stages: a shooting preparation indication state and a shooting indication state. The shooting preparation indication state refers to, for example, a state in which the button is pressed from the standby position to an intermediate position (half-pressed position), and the shooting indication state refers to a state in which the button is pressed to a final pressed position (full-pressed position) beyond the intermediate position. In addition, hereinafter, "a state in which the button is pressed from the standby position to a half-pressed position" is referred to as "a half-pressed state", and "a state in which the button is pressed from the standby position to a full-pressed position" is referred to as "a full-pressed state".

[0074] In the imaging device 10, a shooting mode and a playback mode are selectively set as operation modes according to a user's instruction. The shooting mode is roughly divided into a shooting mode for displaying a moving image and a shooting mode for recording.

[0075] The video display shooting mode is a mode in which a live preview image based on a plurality of frames of display image data obtained by continuous shooting is displayed on the first display 32 and / or the second display 86 (see FIG. Figure 8 and Fig. 9 ) on the action mode. The display image data is image data for real-time preview images, for example, by the CPU 46A (reference Figure 8 ) based on the captured image data 70 (reference Figure 3 to Figure 7 The captured image data 70 refers to the image captured by the imaging element 44 (see Figure 3 ) Image data obtained by photographing a subject. In addition, hereinafter, for the sake of convenience of explanation, the image represented by the photographed image data 70 is referred to as a "photographed image".

[0076] The recording shooting mode is roughly divided into a still image shooting mode and a moving image shooting mode. The still image shooting mode is to use the imaging element 44 (refer to Figure 3 ) is an action mode for still image shooting. In the still image shooting mode, a still image obtained by shooting a subject by the camera device 10 is recorded in a specific recording device (for example, an auxiliary storage device (refer to Figure 8 ) etc.). The dynamic image shooting mode is to make the imaging element 44 (reference Figure 3) is an action mode for performing animation shooting. In the animation shooting mode, a dynamic image obtained by shooting a subject by the camera device 10 is stored in a specific recording device.

[0077] The recording shooting mode is a mode in which a live preview image is displayed on the first display 32 and / or the second display 86 described later, and the recording image data is recorded in the auxiliary storage device 80 described later (see Figure 8 ) and / or the operation mode of the memory card, etc. The image data for recording is roughly divided into still image data and dynamic image data. Figure 3 to Figure 7 ) is generated.

[0078] When the shooting mode is set, the imaging device 10 first enters the shooting mode for displaying moving images. In the shooting mode for displaying moving images, when the release button 20 is pressed, the imaging device 10 shifts from the shooting mode for displaying moving images to the shooting mode for recording.

[0079] In the shooting mode, the manual focus mode and the automatic focus mode are selectively set according to the user's instruction. In the automatic focus mode, the shooting conditions are adjusted by half-pressing the release button 20, and then, if it is fully pressed, exposure is performed. That is, after the AE function is activated by half-pressing the release button 20 to set the exposure state, the AF function is activated to control the focus, and the release button 20 is fully pressed to perform shooting.

[0080] As an example, Figure 2 As shown, a touch panel display 26 , instruction keys 28 , and a viewfinder eyepiece portion 30 are provided on the back surface of the imaging device body 12 .

[0081] The touch panel display 26 includes a first display 32 and a touch panel 34 (see also Figure 8 ). As the first display 32, a liquid crystal display or an organic EL display can be cited.

[0082] The first display 32 displays images and character information, etc. The first display 32 is used to display a live preview image obtained by continuous shooting when the camera device 10 is in shooting mode. In addition, the first display 32 is also used to display a still image obtained by shooting when an instruction for shooting a still image is given. In addition, the first display 32 is also used to display a playback image and a menu screen, etc. when the camera device 10 is in playback mode.

[0083] The touch panel 34 is a transmissive touch panel and overlaps the surface of the display area of ​​the first display 32. The touch panel 34 detects contact with a pointer such as a finger or a stylus pen, and outputs the detection result to the CPU 46A (see Figure 8 ) and other specified export destinations.

[0084] The instruction key 28 receives various instructions such as selection of one or more menus, confirmation of selected contents, deletion of selected contents, zooming, frame transfer, etc.

[0085] As an example, Figure 3 As shown, the imaging device 10 includes mounts 36 and 38. The mount 36 is provided on the imaging device body 12. The mount 38 is provided at a position of the interchangeable lens 14 corresponding to the position of the mount 36. The interchangeable lens 14 is mounted on the imaging device body 12 in an interchangeable manner by the mount 38 being keyed to the mount 36.

[0086] As an example, Figure 3 As shown in FIG. 1 , the interchangeable lens 14 includes an imaging lens 40. The imaging lens 40 includes an objective lens 40A, a focusing lens 40B, a zoom lens 40C, and an aperture 40D. The objective lens 40A, the focusing lens 40B, the zoom lens 40C, and the aperture 40D are arranged in order along the optical axis L1 from the subject side to the imaging device body 12 side. The focusing lens 40B, the zoom lens 40C, and the aperture 40D are controlled by the CPU 46A (see FIG. 46A ) described later. Figure 8 ), and operates by receiving power from a driving source such as a motor (not shown). That is, the focus lens 40B and the zoom lens 40C move along the optical axis L1 according to the applied power. In addition, the aperture 40D adjusts the exposure by operating according to the applied power.

[0087] The imaging device body 12 includes a mechanical shutter 42 and an imaging element 44. The mechanical shutter 42 is controlled by a CPU 46A (see Figure 8 ) and operates by receiving power from a driving source such as a motor (not shown). When the interchangeable lens 14 is mounted on the camera body 12 via the mounts 36 and 38, subject light representing the subject passes through the imaging lens 40 and is imaged on the light receiving surface 44A of the imaging element 44 via the mechanical shutter 42.

[0088] The camera body 12 includes a controller 46, a UI system device 48, a first signal processing unit 50, a second signal processing unit 52, and DRAMs 54 and 56. The first signal processing unit 50 and the second signal processing unit 52 are examples of "a plurality of signal processing processors" involved in the technology of the present invention.

[0089] The controller 46 controls the entire imaging apparatus 10. The UI system device 48 is a device that presents information to the user or receives instructions from the user. The UI system device 48 is connected to the controller 46, and the controller 46 acquires various information from the UI system device 48 and controls the UI system device 48.

[0090] The imaging element 44 is connected to the controller 46 via a communication line 57, and under the control of the controller 46, captures a subject to generate captured image data 70. The details will be described later, and the imaging element 44 separates the generated captured image data 70 into two image data. Figure 3 In the example shown, first separated image data 70A is shown as one of two image data obtained by separating the captured image data 70 , and second separated image data 70B is shown as the other image data.

[0091] The imaging element 44 is connected to the first signal processing unit 50 via the first output line 53, and is connected to the second signal processing unit 52 via the second output line 55. The first signal processing unit 50 and the second signal processing unit 52 are LSIs. In addition, in the present embodiment, the first signal processing unit 50 and the second signal processing unit 52 are respectively implemented by ASICs.

[0092] However, the technology of the present invention is not limited thereto. For example, PLD and / or FPGA may be used instead of ASIC. Also, ASIC, PLD and / or FPGA may be used. Also, a computer including a CPU, ROM and RAM may be used. The CPU may be one or more. Also, the first signal processing unit 50 and / or the second signal processing unit 52 may be implemented by a combination of hardware structure and software structure.

[0093] The first signal processing unit 50 and the second signal processing unit 52 are connected to each other via a communication line 58. The first signal processing unit 50 is connected to a DRAM 54, and the second signal processing unit 52 is connected to a DRAM 56. The first signal processing unit 50 is connected to the controller 46 via a communication line 60.

[0094] The imaging element 44 outputs the first separated image data 70A to the first signal processing unit 50 via the first output line 53 , and outputs the second separated image data 70B to the second signal processing unit 52 via the second output line 55 .

[0095] The first signal processing unit 50 cooperates with the DRAM 54 to perform various signal processing (for example, "specific signal processing" described later) on the input first separated image data 70A. The second signal processing unit 52 cooperates with the DRAM 56 to perform various signal processing (for example, "specific signal processing" described later) on the input second separated image data 70B. And, the second signal processing unit 52 outputs the second separated image data 70B subjected to various signal processing to the first signal processing unit 50 via the communication line 58. The first signal processing unit 50 synthesizes the first separated image data 70A subjected to various signal processing and the second separated image data 70B input from the second signal processing unit 52, and outputs the synthesized data to the controller 46 via the communication line 60.

[0096] The imaging element 44 is an example of a "stacked imaging element" according to the present invention. The imaging element 44 is, for example, a CMOS image sensor. Figure 4 As shown, a photoelectric conversion element 61, a processing circuit 62, and a memory 64 are built into the imaging element 44. The imaging element 44 is an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are integrated into a single chip. That is, the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are packaged. In the imaging element 44, the processing circuit 62 and the memory 64 are stacked on the photoelectric conversion element 61. Specifically, the photoelectric conversion element 61 and the processing circuit 62 are electrically connected to each other through conductive bumps such as copper (not shown in the figure), and the processing circuit 62 and the memory 64 are also electrically connected to each other through conductive bumps such as copper (not shown in the figure). In addition, the memory 64 is an example of a "storage unit" involved in the technology of the present invention.

[0097] The processing circuit 62 is, for example, an LSI, and the memory 64 is, for example, a DRAM. However, the technology of the present invention is not limited thereto, and an SRAM may be used as the memory 64 instead of the DRAM.

[0098] The processing circuit 62 is implemented by an ASIC, and controls the entire imaging element 44 according to the instructions of the controller 46. In addition, here, an example is given in which the processing circuit 62 is implemented by an ASIC, but the technology of the present invention is not limited to this. For example, a PLD and / or FPGA can be used instead of an ASIC. In addition, an ASIC, a PLD and / or an FPGA can also be used. In addition, a computer including a CPU, a ROM and a RAM can be used. The CPU can be one or more. In addition, the processing circuit 62 can be implemented by a combination of a hardware structure and a software structure.

[0099] The photoelectric conversion element 61 has a plurality of photodiodes arranged in a matrix. An example of the plurality of photodiodes is photodiodes for "4896×3265" pixels.

[0100] The photoelectric conversion element 61 is provided with a color filter, and the color filter includes a G filter corresponding to G (green), an R filter corresponding to R (red), and a B filter corresponding to B (blue), which are most helpful for obtaining a brightness signal. In the present embodiment, for the multiple photodiodes of the photoelectric conversion element 61, the G filter, the R filter, and the B filter are respectively arranged with a prescribed periodicity in the row direction (horizontal direction) and the column direction (vertical direction). Therefore, when performing a demosaicing process of the R, G, and B signals, the camera device 10 can process in a repetitive mode. In addition, the demosaicing process refers to the following process: all color information is calculated for each pixel from a mosaic image corresponding to the color filter arrangement of a single-plate color imaging element. For example, in the case of an imaging element composed of RGB three-color filters, the demosaicing process refers to the following process: all RGB color information is calculated for each pixel from a mosaic image composed of RGB.

[0101] In addition, here, a CMOS image sensor is exemplified as the imaging element 44 , but the technology of the present invention is not limited thereto, and even if the imaging element 44 is a CCD image sensor, for example, the technology of the present invention is also applicable.

[0102] The imaging element 44 has a so-called electronic shutter function, and controls the charge accumulation time of each photodiode in the photoelectric conversion element 61 by activating the electronic shutter function under the control of the controller 46. The charge accumulation time is a so-called shutter speed.

[0103] In the imaging device 10, still image shooting and moving image shooting are performed by a rolling shutter method. Still image shooting is performed by activating the electronic shutter function and making the mechanical shutter 42 (refer to Figure 3 ) is implemented by operating, and dynamic image shooting is implemented by activating the electronic shutter function without operating the mechanical shutter 42. In addition, here, the rolling shutter method is illustrated, but the technology of the present invention is not limited to this, and the global shutter method can be applied instead of the rolling shutter method.

[0104] The processing circuit 62 reads out the captured image data 70 obtained by photographing the subject by the photoelectric conversion element 61. The captured image data 70 is the signal charge accumulated in the photoelectric conversion element 61. The processing circuit 62 performs A / D conversion on the captured image data 70 read out from the photoelectric conversion element 61. The processing circuit 62 stores the captured image data 70 obtained by performing A / D conversion on the captured image data 70 in the memory 64.

[0105] The processing circuit 62 acquires the camera image data 70 from the memory 64 and performs various processes on the acquired camera image data 70. Figure 4As shown, the “various processing” mentioned here includes the processing of separating the captured image data 70 into the first separated image data 70A and the second separated image data 70B. The processing circuit 62 outputs the first separated image data 70A to the first signal processing unit 50 via the first output line 53, and outputs the second separated image data 70B to the second signal processing unit 52 via the second output line 55.

[0106] As an example, Figure 5 As shown, the processing circuit 62 includes a photoelectric conversion element control circuit 62A, a digital processing circuit 62B, an image processing circuit 62C, and an output circuit 62D. The output circuit 62D is an example of an "output unit" involved in the technology of the present invention.

[0107] The photoelectric conversion element control circuit 62A is connected to the photoelectric conversion element 61 and the digital processing circuit 62B. The memory 64 is connected to the digital processing circuit 62B and the image processing circuit 62C. The image processing circuit 62C is connected to the output circuit 62D and the memory 64.

[0108] The output circuit 62D has a first output line 53 and a second output line 55. The first output line 53 corresponds to the first signal processing unit 50 and connects the output circuit 62D to the first signal processing unit 50. The second output line 55 corresponds to the second signal processing unit 52 and connects the output circuit 62D to the second signal processing unit 52.

[0109] The photoelectric conversion element control circuit 62A controls the photoelectric conversion element 61 under the control of the controller 46, and reads out the analog camera image data 70 from the photoelectric conversion element 61. The digital processing circuit 62B performs correlated double sampling signal processing on the analog camera image data 70 read out by the photoelectric conversion element control circuit 62A, and then performs A / D conversion, thereby digitizing the analog camera image data 70. The digital processing circuit 62B stores the digitized camera image data 70 in the memory 64.

[0110] The memory 64 is a memory capable of storing a plurality of frames of the captured image data 70. The memory 64 has a pixel-based storage area (not shown), and the captured image data 70 is stored in the corresponding storage area in the memory 64 in pixel units by the digital processing circuit 62B.

[0111] The image processing circuit 62C acquires the captured image data 70 from the memory 64 , and processes the acquired captured image data 70 .

[0112] The image processing circuit 62C performs the above-mentioned various processes on the captured image data 70. The image processing circuit 62C separates the captured image data 70 into first separated image data 70A and second separated image data 70B, and transmits the separated first separated image data 70A and second separated image data 70B (see Figure 3 and Figure 4 ) is output to output circuit 62D.

[0113] The output circuit 62D outputs the first separated image data 70A input from the image processing circuit 62C to the first signal processing unit 50 via the first output line 53. The output circuit 62D outputs the second separated image data 70B input from the image processing circuit 62C to the second signal processing unit 52 via the second output line 55.

[0114] The output frame rate in the output circuit 62D is the same frame rate as the frame rate used in the device at the subsequent stage of the imaging element 44. The output frame rate is the frame rate required for the output circuit 62D to output the first separated image data 70A and the second separated image data 70B, for example, 60fps (frames per second). In contrast, the image capture frame rate is the frame rate required for the photoelectric conversion element 61, the photoelectric conversion element control circuit 62A, the digital processing circuit 62B and the memory 64 to cooperate in shooting, for example, 120fps. In addition, the "shooting" mentioned here refers to the process from the start of exposure of one frame in the photoelectric conversion element 61 to the storage of one frame of the captured image data 70 in the memory 64.

[0115] Here, specific processing contents in the image processing circuit 62C, the output circuit 62D, the first signal processing unit 50 , and the second signal processing unit 52 will be described.

[0116] As an example, Figure 6 As shown, the image processing circuit 62C obtains the captured image data 70 from the memory 64 and separates the obtained captured image data 70 into left image data 70A1 and right image data 70B1. The image processing circuit 62C outputs the left image data 70A1 and the right image data 70B1 obtained by separating the captured image data 70 to the output circuit 62D.

[0117] The left image data 70A1 is the first separated image data 70A (see Figure 3 to Figure 5 ), the right image data 70B1 is the second separated image data 70B (reference Figure 3 to Figure 5 ).

[0118] The left image data 70A1 represents the left image 70A1a (reference Figure 7) image data, the right image data 70B1 represents the right image 70B1a (reference Figure 7 ) image data. As an example, Figure 7 As shown, the left image 70A1a and the right image 70B1a are a pair of images adjacent to each other. The left image 70A1a and the right image 70B1a have an overlapping region 71. The overlapping region 71 is an area that overlaps the left image 70A1a and the right image 70B1a in the left-right direction RL. The number of pixels in the left-right direction RL of the overlapping region 71 is, for example, tens to hundreds of pixels.

[0119] In addition, Figure 6 In the example shown, the image processing circuit 62C acquires the captured image data 70 from the memory 64 and separates the acquired captured image data 70, but the technology of the present invention is not limited to this. In this case, for example, first, the image processing circuit 62C selects the left image data 70A1 and the right image data 70B1 from the captured image data 70 according to the predetermined address in the memory 64. Then, the image processing circuit 62C directly acquires the selected left image data 70A1 and the right image data 70B1 from the memory 64. In addition, the predetermined address is the acquisition address of the left image data 70A1 and the acquisition address of the right image data 70B1. The acquisition address of the left image data 70A1 and the acquisition address of the right image data 70B1 are determined so that each of the left image data 70A1 and the right image data 70B1 also includes the image data representing the overlapping area 71.

[0120] The first signal processing unit 50 includes a buffer 50A, a signal processing circuit 50B, and a receiving circuit 50C. The DRAM 54 and the controller 46 are connected to the signal processing circuit 50B. The second signal processing unit 52 includes a buffer 52A, a signal processing circuit 52B, and a transmitting circuit 52C. The DRAM 56 is connected to the signal processing circuit 52B.

[0121] The output circuit 62D outputs the left image data 70A1 to the buffer 50A via the first output line 53. The buffer 50A holds the input left image data 70A1 and outputs it to the signal processing circuit 50B in a FIFO manner. The signal processing circuit 50B stores the left image data 70A1 input from the buffer 50A in the DRAM 54. The signal processing circuit 50B performs signal processing such as color tone correction, white balance adjustment, sharpness adjustment, gamma correction, and grayscale correction on the left image data 70A1 stored in the DRAM 54 (hereinafter referred to as "specific signal processing").

[0122] The output circuit 62D outputs the right image data 70B1 to the buffer 52A via the second output line 55. The buffer 52A holds the input right image data 70B1 and outputs it to the signal processing circuit 52B in a FIFO manner. The signal processing circuit 52B stores the right image data 70B1 input from the buffer 52A in the DRAM 56. The signal processing circuit 52B performs specific signal processing on the right image data 70B1 stored in the DRAM 56.

[0123] The transmission circuit 52C transmits the right image data 70B1 subjected to specific signal processing to the first signal processing unit 50 via the signal processing circuit 52B. In the first signal processing unit 50, the reception circuit 50C receives the right image data 70B1 transmitted from the transmission circuit 52C.

[0124] The signal processing circuit 50B generates composite image data 72 by combining the right image data 70B1 received by the receiving circuit 50C and the left image data 70A1 subjected to signal processing. The signal processing circuit 50B outputs the composite image data 72 to the controller 46 via the communication line 60 .

[0125] The composite image data 72 is generated by connecting the left image data 70A1 and the right image data 70B1. Figure 7 ) and the overlapping area 71 in the right image data 70B1 (reference Figure 7 ) is averaged by addition operation. Thus, the image data of the camera image is averaged in the left and right direction RL (reference Figure 7 ) is connected, the boundary area between the two images is suppressed from becoming obvious. In addition, although the addition operation average is exemplified here, the technology of the present invention is not limited to this, and replacement can be used instead of the addition operation average. The "replacement" mentioned here means replacing one of the image data representing the overlapping area 71 in the left image data 70A1 and the image data representing the overlapping area 71 in the right image data 70B1 with the other.

[0126] As an example, Figure 8 As shown, the controller 46 includes a CPU 46A, a ROM 46B, a RAM 46C, a connection I / F 46D, and an input I / F 46E. The CPU 46A, the ROM 46B, the RAM 46C, the connection I / F 46D, and the input I / F 46E are connected to each other via a bus 88 .

[0127] Various programs are stored in the ROM 46B. The CPU 46A reads various programs from the ROM 46B and expands the read various programs into the RAM 46C. The CPU 46A controls the entire imaging device 10 according to the various programs expanded into the RAM 46C.

[0128] The connection I / F 46D is an FPGA, and is connected to the imaging element 44 via a communication line 57. The CPU 46A controls the imaging element 44 via the connection I / F 46D.

[0129] The input I / F 46E is a device having an FPGA, and is connected to the first signal processing unit 50 via a communication line 60. The synthesized image data 72 (refer to Figure 6 ) is input from the first signal processing unit 50 to the input I / F 46E via the communication line 60. The input I / F 46E transfers the synthesized image data 72 input from the first signal processing unit 50 to the CPU 46A.

[0130] The auxiliary storage device 80 and the external I / F 82 are connected to the bus 88. The auxiliary storage device 80 is a nonvolatile memory such as an SSD, an HDD, or an EEPROM. The CPU 46A reads and writes various information to and from the auxiliary storage device 80.

[0131] The external I / F 82 is a device having an FPGA. An external device (not shown) such as a USB memory and a memory card is connected to the external I / F 82. The external I / F 82 controls the exchange of various information between the CPU 46A and the external device.

[0132] The UI system device 48 includes the hybrid viewfinder 16, the touch panel display 26, and the receiving unit 84. The first display 32 and the touch panel 34 are connected to the bus 88. Therefore, the CPU 46A causes the first display 32 to display various information, and operates according to various instructions received from the touch panel 34.

[0133] Receiving unit 84 includes touch panel 34 and hard key unit 25. Hard key unit 25 is a plurality of hard keys, and includes release button 20, dial 22, and instruction key 28. Hard key unit 25 is connected to bus 88, and CPU 46A operates according to various instructions received by hard key unit 25.

[0134] The hybrid viewfinder 16 includes a second display 86. The CPU 46A causes the second display 86 to display various information.

[0135] As an example, Fig. 9 As shown, the hybrid viewfinder 16 includes an OVF 90 and an EVF 92. The OVF 90 is a reverse Galilean viewfinder and includes an eyepiece lens 94, a prism 96, and an objective lens 98. The EVF 92 includes a second display 86, a prism 96, and an eyepiece lens 94.

[0136] A liquid crystal shutter 100 is disposed on the subject side of the objective lens 98 along the optical axis L2 of the objective lens 98 . When the EVF 92 is used, the liquid crystal shutter 100 blocks light to prevent an optical image from being incident on the objective lens 98 .

[0137] The prism 96 reflects the electronic image or various information displayed on the second display 86 and guides them to the eyepiece lens 94, and synthesizes the optical image with the electronic image and / or various information displayed on the second display 86. As the electronic image displayed on the second display 86, a live preview image 102 based on the synthesized image data 72 can be cited.

[0138] In the case of the OVF mode, the CPU 46A controls the liquid crystal shutter 100 to be in a non-light-shielding state so that the optical image can be visually recognized from the eyepiece lens 94. In the case of the EVF mode, the CPU 46A controls the liquid crystal shutter 100 to be in a light-shielding state so that only the electronic image displayed on the second display 86 can be visually recognized from the eyepiece lens 94.

[0139] In addition, for the sake of convenience, the following description will not be made on the first display 32 (see Figure 2 and Figure 8 ) and the second display 86, they are referred to as "displays" without being marked with symbols. The display is an example of a "display unit" involved in the technology of the present invention. In addition, CPU 46A is an example of a "control unit (control processor)" involved in the technology of the present invention.

[0140] Next, the operation of the imaging device 10 will be described.

[0141] First, refer to Fig.10 , the imaging processing flow executed by the processing circuit 62 of the imaging element 44 is described.

[0142] exist Fig.10 In the imaging process shown, first, in step ST10, the photoelectric conversion element control circuit 62A determines whether the time to start exposure of the photoelectric conversion element 61 (hereinafter referred to as "exposure start time") has arrived. The exposure start time is a time periodically specified according to the above-mentioned imaging frame rate. In step ST10, if the exposure start time has not arrived, it is determined as "no", and the imaging process is transferred to step ST22. In step ST10, if the exposure start time has arrived, it is determined as "yes", and the imaging process is transferred to step ST12.

[0143] In step ST12 , the photoelectric conversion element control circuit 62A causes the photoelectric conversion element 61 to perform exposure for one frame.

[0144] In the next step ST14 , the photoelectric conversion element control circuit 62A reads out one frame of captured image data 70 from the photoelectric conversion element 61 .

[0145] In the next step ST16 , the digital processing circuit 62B performs correlated double sampling signal processing and A / D conversion on the image data 70 read out in step ST14 , thereby digitizing the analog image data 70 . The digital processing circuit 62B then stores the digitized image data 70 in the memory 64 .

[0146] In the next step ST18, the image processing circuit 62C acquires the captured image data 70 from the memory 64, and separates the acquired captured image data 70 into left image data 70A1 (refer to Figure 6 ) and right image data 70B1 (reference Figure 6 ). The image processing circuit 62C outputs the left image data 70A1 and the right image data 70B1 to the output circuit 62D.

[0147] In the next step ST20, the output circuit 62D outputs the signal through the first output line 53 (reference Figure 3 to Figure 6 and Figure 8 ) outputs the left image data 70A1 to the first signal processing unit 50. Furthermore, the output circuit 62D outputs the left image data 70A1 via the second output line 55 (reference Figure 3 to Figure 6 and Figure 8 ) outputs the right image data 70B1 to the second signal processing unit 52.

[0148] In the next step ST22, the processing circuit 62 determines whether the condition for ending the image capture process (hereinafter referred to as "image capture process ending condition") is satisfied. As the image capture process ending condition, for example, the condition that the receiving unit 84 receives an instruction to end the image capture process can be cited. In step ST22, if the image capture process ending condition is not satisfied, it is determined as "No", and the image capture process is transferred to step ST10. In step ST22, if the image capture process ending condition is satisfied, it is determined as "Yes", and the image capture process is ended.

[0149] Next, refer to Fig.11 , the first signal processing flow performed by the first signal processing unit 50 is described.

[0150] exist Fig.11 In the first signal processing shown in FIG. 1 , in step ST30, the first signal processing unit 50 determines whether the left image data 70A1 (reference Figure 6). In step ST30, if the left image data 70A1 is not input from the processing circuit 62, the determination is "No", and the first signal processing is transferred to step ST42. In step ST30, if the left image data 70A1 is input from the processing circuit 62, the determination is "Yes", and the first signal processing is transferred to step ST32.

[0151] In step ST32 , the first signal processing unit 50 performs specific signal processing on the left image data 70A1 .

[0152] In the next step ST34, the first signal processing unit 50 determines whether the signal received by executing Fig.12 The right image data 70B1 (reference Figure 6 ). In step ST34, if the right image data 70B1 is not received, it is determined as "No", and the first signal processing is transferred to step ST40. In step ST34, if the right image data 70B1 is received, it is determined as "Yes", and the first signal processing is transferred to step ST36.

[0153] In step ST40, the first signal processing unit 50 determines whether a condition for terminating the first signal processing (hereinafter referred to as "the first signal processing termination condition") is satisfied. As the first signal processing termination condition, for example, a condition that the receiving unit 84 receives an instruction to terminate the first signal processing can be cited. In step ST40, if the first signal processing termination condition is not satisfied, it is determined as "no", and the first signal processing is transferred to step ST34. In step ST40, if the first signal processing termination condition is satisfied, it is determined as "yes", and the first signal processing is terminated.

[0154] In step ST36, the first signal processing unit 50 generates synthesized image data 72 (see FIG. 1 ) by performing a synthesis process of synthesizing the left image data 70A1 obtained by executing the process of step ST32 and the right image data 70B1 received in step ST34. Figure 6 ).

[0155] In the next step ST38, the first signal processing unit 50 transmits the signal to the first signal processing unit 50 via the communication line 60 (refer to Figure 6 and Figure 8 ) outputs the synthesized image data 72 obtained by executing the process of step ST36 to the controller 46 (refer to Figure 6 ).

[0156] In the next step ST42, the first signal processing unit 50 determines whether the first signal processing end condition is satisfied. In step ST42, if the first signal processing end condition is not satisfied, the determination is "No", and the first signal processing is transferred to step ST30. In step ST42, if the first signal processing end condition is satisfied, the determination is "Yes", and the first signal processing is terminated.

[0157] Next, refer to Fig.12 , the second signal processing flow performed by the second signal processing unit 52 is described.

[0158] exist Fig.12 In the second signal processing shown in FIG. 1 , in step ST50, the second signal processing unit 52 determines whether the right image data 70B1 (reference Figure 6 ). In step ST50, if the right image data 70B1 is not input from the processing circuit 62, the determination is "No", and the second signal processing is transferred to step ST56. In step ST50, if the right image data 70B1 is input from the processing circuit 62, the determination is "Yes", and the second signal processing is transferred to step ST52.

[0159] In step ST52 , the second signal processing unit 52 performs specific signal processing on the right image data 70B1 .

[0160] In the next step ST54, the second signal processing unit 52 receives the signal through the communication line 58 (refer to Figure 3 to Figure 6 and Figure 8 ) The right image data 70B1 obtained by executing the processing of step ST52 is sent to the first signal processing unit 50.

[0161] In the next step ST56, the second signal processing unit 52 determines whether the condition for terminating the second signal processing (hereinafter referred to as "the second signal processing termination condition") is satisfied. As the second signal processing termination condition, for example, the condition that the receiving unit 84 receives an instruction to terminate the second signal processing can be cited. In step ST56, if the second signal processing termination condition is not satisfied, it is determined as "no", and the second signal processing is transferred to step ST50. In step ST56, if the second signal processing termination condition is satisfied, it is determined as "yes", and the second signal processing is terminated.

[0162] As described above, the imaging device 10 includes the imaging element 44, the first signal processing unit 50, and the second signal processing unit 52. In addition, the imaging element 44 includes the output circuit 62D. The left image data 70A1 is output to the first signal processing unit 50 via the first output line 53 through the output circuit 62D, and the right image data 70B1 is output to the second signal processing unit 52 via the output circuit 62D via the second output line 55. In addition, the left image data 70A1 and the right image data 70B1, which have been subjected to specific signal processing, are synthesized by the first signal processing unit 50, and the synthesized image data 72 is output to the controller 46.

[0163] Therefore, compared with the case where the camera image data 70 is output to the first signal processing unit 50 and the second signal processing unit 52 using only one system of output lines, the communication volume between the imaging element 44 and each of the first signal processing unit 50 and the second signal processing unit 52 is reduced.

[0164] Furthermore, the amount of image data on which the first signal processing unit 50 and the second signal processing unit 52 respectively perform specific signal processing is smaller than the captured image data 70. Therefore, compared with a case where only the first signal processing unit 50 or the second signal processing unit 52 performs specific signal processing on the entire captured image data 70, the load applied to the first signal processing unit 50 and the second signal processing unit 52 when performing the specific signal processing is reduced.

[0165] Therefore, the image pickup device 10 can realize high-speed image processing compared to a case where image data is output from the imaging element 44 to a plurality of signal processing sections through only one system of output lines.

[0166] And, if Figure 7 As shown, the left image data 70A1 and the right image data 70B1 are image data respectively representing an image having an overlapping area 71 between the left image 70A1a and the right image 70B1a. Figure 7 ) is simply divided into two images to obtain two images, it is possible that the boundary area between the two images is obvious. Figure 7 As shown, the left image 70A1a and the right image 70B1a each have an overlapping region 71. Therefore, compared with the case of synthesizing two images obtained by simply dividing the captured image into two in the left-right direction RL, it is possible to suppress the boundary region between the two images from becoming noticeable.

[0167] Furthermore, the imaging element 44 is an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are integrated into a single chip. Thus, the portability of the imaging element 44 is improved compared to an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are not integrated into a single chip. Furthermore, the degree of freedom in design can be improved compared to an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are not integrated into a single chip. Furthermore, compared to an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are not integrated into a single chip, it is also possible to contribute to the miniaturization of the camera body 12.

[0168] And, if Figure 4 As shown in the figure, as the imaging element 44, a stacked type imaging element in which a memory 64 is stacked on a photoelectric conversion element 61 is adopted. As a result, compared with a case where the photoelectric conversion element 61 and the memory 64 are not stacked, the load imposed by the processing between the photoelectric conversion element 61 and the memory 64 can be reduced. In addition, compared with a case where the photoelectric conversion element 61 and the memory 64 are not stacked, the degree of freedom of design can also be improved. In addition, compared with a case where the photoelectric conversion element 61 and the memory 64 are not stacked, it can also contribute to the miniaturization of the imaging device body 12.

[0169] And, if Fig. 9 As shown, the image represented by the composite image data 72 is displayed as a live preview image 102 on the second display 86 under the control of the CPU 46A. This enables the user to visually recognize the image represented by the composite image data 72. Fig. 9 In the example shown, the live preview image 102 is displayed on the second display 86, but the technology of the present invention is not limited to this. For example, the live preview image 102 can be displayed on the first display 32, or the live preview image 102 can be displayed on both the first display 32 and the second display 86.

[0170] In the first embodiment, as the imaging element 44, an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are integrated into a single chip is exemplified, but the technology of the present invention is not limited thereto. For example, at least the photoelectric conversion element 61 and the memory 64 may be integrated into a single chip.

[0171] In the first embodiment described above, an example of separating the camera image data 70 into the left image data 70A1 and the right image data 70B1 is given for explanation, but the technology of the present invention is not limited to this. For example, the camera image data 70 may also be separated into upper image data representing an image of the upper area in the camera image and lower image data representing an image of the lower area in the camera image. Furthermore, the camera image data 70 may also be separated into upper left image data representing an image of the upper left area in the camera image and lower right image data representing an image of the lower right area in the camera image. Furthermore, the camera image data 70 may also be separated into upper right image data representing an image of the upper right area in the camera image and lower left image data representing an image of the lower left area in the camera image. In addition, the camera image data 70 may also be separated into central image data representing an image of the central area in the camera image and peripheral image data representing an image of the peripheral area in the camera image, i.e., an area other than the central area.

[0172] Even when the camera image data 70 is separated in this way, each of a pair of image data obtained by separating the camera image data 70 preferably includes repeated image data indicating an image repeated in the separation direction between the two images of the separated camera image.

[0173] [Second embodiment]

[0174] In the first embodiment described above, the case where the same signal processing is performed on the first separated image data 70A and the second separated image data 70B, respectively, is described. However, in the second embodiment, the case where different signal processing is performed on the first separated image data 70A and the second separated image data 70B is described. In addition, in the second embodiment, the same reference numerals are given to the same components as those in the first embodiment described above, and their description is omitted. The following describes the parts different from the first embodiment described above.

[0175] like Figure 1 As shown, the imaging device 200 according to the second embodiment is different from the imaging device 10 described in the first embodiment in that it includes an imaging device body 212 instead of the imaging device body 12 .

[0176] The camera body 212 is different from the camera body 12 in that it has an imaging element 244 (see Fig.13 ) is used to replace the imaging element 44. In addition, the imaging device body 212 is different from the imaging device body 12 in that it has a first signal processing unit 250 (see Fig.13 ) instead of the first signal processing unit 50 and having a second signal processing unit 252 (reference Fig.13) instead of the second signal processing unit 52.

[0177] like Fig.13 As shown, imaging element 244 is different from imaging element 44 in that it has processing circuit 262 instead of processing circuit 62. Processing circuit 262 is different from processing circuit 62 in that it has image processing circuit 262C instead of image processing circuit 62C and output circuit 262D instead of output circuit 62D.

[0178] The image processing circuit 262C is different from the image processing circuit 62C in that it has a face detection circuit 262C1. The face detection circuit 262C1 is an example of a "detection unit (detection processor)" involved in the technology of the present invention. The face detection circuit 262C1 is a circuit having a well-known face detection function. The face detection circuit 262C1 is implemented by a hardware structure such as an ASIC. In addition, the face detection circuit 262C1 is not limited to a hardware structure, and can be implemented by a software structure, or by a software structure and a hardware structure. In the second embodiment, the face detection circuit 262C1 acquires the camera image data 70 from the memory 64. The face detection circuit 262C1 determines a facial image 69 (reference) representing a human face in the camera image represented by the acquired camera image data 70. Fig.14A ). That is, the face detection circuit 262C1 detects the face image data representing the face image 69 from the captured image data 70 .

[0179] Then, the face detection circuit 262C1 extracts a face region image 70B2a (see FIG. 2A ) representing a predetermined range including the identified face image 69 from the captured image data 70. Fig.14A )'s facial area image data 70B2, separating the captured image data 70 into the facial area image data 70B2 and the background area image data 70A2.

[0180] The background area image data 70A2 refers to an image representing the background area in the camera image, that is, image data of an image other than the face area image 70B2a. The above-mentioned predetermined range refers to, for example, the face image 69 (refer to Fig.14A ) and the difference between the size of the facial image 69 and the size of the images other than the facial image 69 in the captured image. Fig.14A In the example shown, the facial image 69 is located in the central area of ​​the captured image. In this case, the captured image is separated into three areas: the central area, the upper area, and the lower area. The image of the central area is determined as the facial area image 70B2a, and the image of the upper area and the image of the lower area, that is, the image other than the central area in the captured image is determined as the background area image 70A2a.

[0181] exist Fig.14A In the example shown, an image showing two faces is shown as the face image 69. The face image 69 is an image that identifies the area with the highest density of faces of multiple people in the camera image. In addition, the technology of the present invention is not limited to this. For example, a face image showing the face of one person may be used instead of the face image 69, a face image showing a face with a specific expression (for example, a smiling face) may be used, and a face image showing the face of a specific person may be used.

[0182] like Fig.13 As shown, the background area image data 70A2 and the face area image data 70B2 are output to the output circuit 262D via the image processing circuit 262C. The background area image data 70A2 is an example of the first separated image data 70A, and the face area image data 70B2 is an example of the second separated image data 70B. Furthermore, the face area image 70B2a is an example of the "specified image" involved in the technology of the present invention, and the background area image 70A2a is an example of the "image different from the specified image" involved in the technology of the present invention.

[0183] The output circuit 262D is different from the output circuit 62D in that it outputs background area image data 70A2 instead of the left image data 70A1 and outputs face area image data 70B2 instead of the right image data 70B1.

[0184] Specifically, the output circuit 262D outputs the background area image data 70A2 to the first signal processing unit 250 via the first output line 53 . Furthermore, the output circuit 262D outputs the face area image data 70B2 to the second signal processing unit 252 via the second output line 55 .

[0185] The first signal processing unit 250 is different from the first signal processing unit 50 in that it includes a signal processing circuit 250B instead of the signal processing circuit 50B.

[0186] The background area image data 70A2 output from the output circuit 262D is temporarily held in the buffer 50A and is output from the buffer 50A to the signal processing circuit 250B. Then, the signal processing circuit 250B performs specific signal processing on the background area image data 70A2 input from the buffer 50A.

[0187] The second signal processing unit 252 is different from the second signal processing unit 52 in that it includes a second signal processing circuit 252B instead of the signal processing circuit 52B. The signal processing circuit 252B is different from the signal processing circuit 52B in that it includes a face recognition processing circuit 252B1.

[0188] The facial region image data 70B2 output from the output circuit 262D is temporarily held in the buffer 52A and is output from the buffer 52A to the signal processing circuit 252B. The signal processing circuit 252B performs specific signal processing on the facial region image data 70B2 input from the buffer 52A. The facial authentication processing circuit 252B1 has a well-known facial authentication function and performs facial authentication on the facial region image data 70B2 that has been subjected to specific signal processing. By performing facial authentication, for example, a facial image 69 (see FIG. 1 ) is generated. Fig.14A ) is a determination of whether the face represented by the facial image 69 is a specific display (e.g., a smiling face) and / or whether the face represented by the facial image 69 is the face of a specific person.

[0189] The face recognition processing circuit 252B1 outputs the face region image data 70B2 on which the face recognition has been performed to the transmission circuit 52C. The face region image data 70B2 is provided with face recognition result information indicating the result of the face recognition, and is used for processing in the subsequent circuit. The "subsequent circuit" mentioned here refers to, for example, the first signal processing unit 250 and / or the controller 46.

[0190] The transmission circuit 52C transmits the facial area image data 70B2 input from the facial recognition processing circuit 252B1 to the first signal processing unit 250 .

[0191] In the first signal processing unit 250, the receiving circuit 50C receives the face area image data 70B2 transmitted from the transmitting circuit 52C. Then, the signal processing circuit 250B synthesizes the face area image data 70B2 received by the receiving circuit 50C and the background area image data 70A2 subjected to specific signal processing by the signal processing circuit 250B to generate synthesized image data 272. Then, the signal processing circuit 250B outputs the synthesized image data 272 to the controller 46 via the communication line 60.

[0192] In addition, as an example, Fig. 14B As shown, the background region image 70A2a and the face region image 70B2a may have an overlapping region 271. The overlapping region 271 is an area overlapping the background region image 70A2a and the face region image 70B2a in the up-down direction UD. As with the overlapping region 71 described in the first embodiment, the number of pixels in the up-down direction UD of the overlapping region 271 may be, for example, tens to hundreds of pixels. When the background region image data 70A2 and the face region image data 70B2 are synthesized by the first signal processing unit 250, each image data representing the overlapping region 271 of each of the background region image data 70A2 and the face region image data 70B2 is averaged by addition as in the first embodiment.

[0193] Next, the operation of the imaging device 200 will be described.

[0194] First, refer to Fig.15 , the imaging process flow executed by the processing circuit 262 of the imaging element 244 is described. Fig.15 The camera processing shown is Fig.10 Compared with the imaging process shown in FIG. 1 , the difference is that the process of step ST18 is replaced by the process of step ST60 and the process of step ST20 is replaced by the process of step ST62. Fig.15 In the flowchart of the imaging process shown in FIG. Fig.10 The same steps in the imaging process shown are marked with the same step numbers. Fig.15 The camera processing shown is only for Fig.10 The differences in the imaging processing shown will be described.

[0195] exist Fig.15 In the imaging process shown, in step ST60, the face detection circuit 262C1 acquires the image data 70 from the memory 64, and separates the acquired image data 70 into background area image data 70A2 and face area image data 70B2. Then, the image processing circuit 262C outputs the background area image data 70A2 and the face area image data 70B2 to the output circuit 262D.

[0196] In the next step ST62 , the output circuit 262D outputs the background area image data 70A2 to the first signal processing unit 250 via the first output line 53 . Furthermore, the output circuit 262D outputs the face area image data 70B2 to the second signal processing unit 252 via the second output line 55 .

[0197] Next, refer to Fig.16 , the first signal processing flow performed by the first signal processing unit 250 is described.

[0198] exist Fig.16 In the first signal processing shown in FIG. 1 , in step ST70, the first signal processing unit 250 determines whether the background area image data 70A2 (see FIG. 1 ) is input from the processing circuit 262. Fig.13 ). In step ST70, when the background area image data 70A2 is not input from the processing circuit 262, it is determined as "No", and the first signal processing is transferred to step ST82. In step ST70, when the background area image data 70A2 is input from the processing circuit 262, it is determined as "Yes", and the first signal processing is transferred to step ST72.

[0199] In step ST72 , the first signal processing unit 250 performs specific signal processing on the background area image data 70A2 .

[0200] In the next step ST74, the first signal processing unit 250 determines whether the received Fig.17 The facial area image data 70B2 (reference Fig.13 ). In step ST74, if the face area image data 70B2 is not received, it is determined as "No", and the first signal processing moves to step ST80. In step ST74, if the face area image data 70B2 is received, it is determined as "Yes", and the first signal processing moves to step ST76.

[0201] In step ST80, the first signal processing unit 250 determines whether the first signal processing end condition is satisfied. In step ST80, if the first signal processing end condition is not satisfied, the determination is "No", and the first signal processing is transferred to step ST74. In step ST80, if the first signal processing end condition is satisfied, the determination is "Yes", and the first signal processing is terminated.

[0202] In step ST76, the first signal processing unit 250 generates synthesized image data 272 (see FIG. 2 ) by performing a synthesis process of synthesizing the background area image data 70A2 obtained by executing the process of step ST72 and the face area image data 70B2 received in step ST74. Fig.13 ).

[0203] In the next step ST78, the first signal processing unit 250 transmits the signal to the first signal processing unit 250 via the communication line 60 (refer to Fig.13 ) outputs the synthesized image data 272 obtained by executing the process of step ST76 to the controller 46 (refer to Fig.13 ).

[0204] In the next step ST82, the first signal processing unit 250 determines whether the first signal processing end condition is satisfied. In step ST82, if the first signal processing end condition is not satisfied, the determination is "No", and the first signal processing is transferred to step ST70. In step ST82, if the first signal processing end condition is satisfied, the determination is "Yes", and the first signal processing is terminated.

[0205] Next, refer to Fig.17 , the second signal processing flow performed by the second signal processing unit 252 is described.

[0206] exist Fig.17In the second signal processing shown in FIG. 1 , in step ST90, the second signal processing unit 252 determines whether the face area image data 70B2 (see FIG. 1 ) is input from the processing circuit 262. Fig.13 ). In step ST90, if the facial region image data 70B2 is not input from the processing circuit 262, the determination is "No", and the second signal processing is transferred to step ST96. In step ST90, if the facial region image data 70B2 is input from the processing circuit 262, the determination is "Yes", and the second signal processing is transferred to step ST92.

[0207] In step ST92, the second signal processing unit 252 performs specific signal processing on the face area image data 70B2. Then, the second signal processing unit 252 performs face authentication on the face area image data 70B2 that has been subjected to the specific signal processing.

[0208] In the next step ST94, the second signal processing unit 252 receives the signal through the communication line 58 (reference Fig.13 ) sends the face area image data 70B2 obtained by executing the process of step ST92 to the first signal processing unit 250. In addition, the face area image data 70B2 obtained by executing the process of step ST92 is provided with face authentication result information.

[0209] In the next step ST96, the second signal processing unit 252 determines whether the second signal processing end condition is satisfied. In step ST96, if the second signal processing end condition is not satisfied, the determination is "No", and the second signal processing is transferred to step ST90. In step ST96, if the second signal processing end condition is satisfied, the determination is "Yes", and the second signal processing ends.

[0210] As described above, in the imaging device 200, the captured image data 70 is separated into the background area image data 70A2 and the face area image data 70B2. The background area image data 70A2 is output to the first signal processing unit 250 via the first output line 53, and the face area image data 70B2 is output to the second signal processing unit 252 via the second output line 55.

[0211] Generally, since the face area image data 70B2 is image data that is more important than the background area image data 70A2, the face area image data 70B2 is subjected to a more complex process than the background area image data 70A2 in the second signal processing unit 252. In the second embodiment, face recognition is performed as the complex process. In addition, the first signal processing unit 250 synthesizes the background area image data 70A2 subjected to a specific signal process and the face area image data 70B2 subjected to face recognition in addition to the specific signal process.

[0212] In this way, since processes of different complexity are executed in the first signal processing unit 250 and the second signal processing unit 252 , the load required for image processing can be reduced compared to a case where complex processes are always executed on the entire captured image data 70 .

[0213] And, if Fig. 14B As shown in FIG. 2 , the background region image 70A2a and the face region image 70B2a adjacent to each other in the up-down direction UD have an overlapping region 271 between the background region image 70A2a and the face region image 70B2a. Thus, in the signal processing circuit 250B, compared with the case where two images obtained by simply dividing into two are connected, the synthesized image data 272 in which the boundary region between the background region image 70A2a and the face region image 70B2a is suppressed from being made obvious is generated.

[0214] In addition, in the second embodiment described above, an example of separating the captured image data 70 into the background area image data 70A2 and the face area image data 70B2 is given for explanation, but the technology of the present invention is not limited to this. For example, the image data designated by the user via the receiving unit 84 in the captured image data 70 may be output to the second signal processing unit 252, and the remaining image data in the captured image data 70 may be output to the first signal processing unit 250. The image data designated by the user refers to, for example, image data in the captured image data 70 that is designated as important image data according to the instruction received by the receiving unit 84. The important image data refers to, for example, image data representing a partial area of ​​an image including a person and / or a building that the user is interested in.

[0215] In this case, it is also preferable to provide an overlapping region corresponding to the overlapping region 271 between the image represented by the image data designated by the user and the image represented by the remaining image data in the captured image data 70. Thus, in the signal processing circuit 250B, compared with the case where two images obtained by simply dividing into two are connected, synthesized image data is generated in which the boundary region between the image represented by the designated image data and the image represented by the remaining image data is suppressed from being obvious.

[0216] Furthermore, in the second embodiment, face recognition is illustrated as an example of complex processing, but the technology of the present invention is not limited thereto. Examples of complex processing include pupil detection processing for detecting the pupils of a person, red eye correction processing for correcting red eyes, and / or electronic shake correction processing (e.g., EIS).

[0217] [Third Embodiment]

[0218] In the first embodiment, an example of a method for separating the captured image data 70 into the left image data 70A1 and the right image data 70B1 is described, but in the third embodiment, an example of a method for separating the captured image data 70 into two image data after compression is described. In addition, in the third embodiment, the same reference numerals are given to the same components as those in the first embodiment, and their description is omitted. The following describes the parts different from the first embodiment.

[0219] like Figure 1 As shown, the imaging device 300 according to the third embodiment is different from the imaging device 10 described in the first embodiment in that it includes an imaging device body 312 instead of the imaging device body 12 .

[0220] The camera body 312 is different from the camera body 12 in that it has an imaging element 344 (see Fig.18 ) to replace the imaging element 44 and having a first signal processing unit 350 (reference Fig.18 ) instead of the first signal processing unit 50. The first signal processing unit 350 is different from the first signal processing unit 50 in that it has a signal processing circuit 350B instead of the signal processing circuit 50B.

[0221] like Fig.18 As shown, imaging element 344 differs from imaging element 44 in that it has a processing circuit 362 instead of processing circuit 62. Processing circuit 362 differs from processing circuit 62 in that it has an image processing circuit 362C instead of image processing circuit 62C and an output circuit 362D instead of output circuit 62D.

[0222] The image processing circuit 362C acquires the captured image data 70 from the memory 64. The captured image data 70 acquired from the memory 64 by the image processing circuit 362C is color image data in which R pixels, G pixels, and B pixels are periodically arranged. Fig.19 As shown, in the captured image data 70, R pixels, G pixels, and B pixels are arranged in a periodic pattern corresponding to the X-Trans (registered trademark) arrangement. The R pixels, G pixels, and B pixels are examples of "a plurality of primary color pixels" involved in the technique of the present invention.

[0223] exist Fig.19In the example shown, in the first row, the R pixel, the G pixel, and the B pixel are sequentially arranged in a row direction in a circular manner with G pixel, B pixel, R pixel, G pixel, R pixel, and B pixel. Also, in the second row, the R pixel, the G pixel, and the B pixel are sequentially arranged in a row direction in a circular manner with R pixel, G pixel, G pixel, B pixel, G pixel, and G pixel. Also, in the third row, the R pixel, the G pixel, and the B pixel are sequentially arranged in a row direction in a circular manner with B pixel, G pixel, G pixel, R pixel, G pixel, and G pixel. Also, in the fourth row, the R pixel, the G pixel, and the B pixel are sequentially arranged in a row direction in a circular manner with G pixel, R pixel, B pixel, G pixel, B pixel, and R pixel. Also, in the fifth row, the R pixel, the G pixel, and the B pixel are sequentially arranged in a row direction in a circular manner with B pixel, G pixel, G pixel, R pixel, G pixel, and G pixel. Also, in the sixth row, the R pixel, the G pixel, and the B pixel are sequentially arranged in a row direction in a circular manner with R pixel, G pixel, G pixel, B pixel, G pixel, and G pixel. Then, the arrangement pattern of R pixels, G pixels, and B pixels in the first to sixth rows is repeated in the column direction in units of six rows, thereby forming the arrangement pattern of R pixels, G pixels, and B pixels in the entire captured image data 70 .

[0224] The image processing circuit 362C compresses the captured image data 70 obtained from the memory 64. That is, the image processing circuit 362C generates vertical thinning image data 73 based on the captured image data 70. The vertical thinning image data 73 is image data obtained by thinning the captured image data 70 in units of lines. Specifically, as an example, Fig.19 As shown, the vertical thinning image data 73 is image data indicating a vertical 1 / 2 thinning image obtained by thinning out pixels of even-numbered rows in the column direction from the captured image indicated by the captured image data 70 .

[0225] The image processing circuit 362C separates the vertical thinning image data 73 into odd-numbered column image data 73A and even-numbered column image data 73B, and outputs the separated odd-numbered column image data 73A and even-numbered column image data 73B to the output circuit 362D. In addition, here, the odd-numbered column image data 73A and even-numbered column image data 73B are obtained by thinning the captured image data 70 and then dividing them as an example of a method for obtaining a plurality of divided image data, but the technology of the present invention is not limited to this. For example, a plurality of divided image data can also be obtained by directly dividing the captured image data 70.

[0226] As an example, Fig. 20 and Fig.21As shown in FIG. 1 , the odd-numbered column image data 73A is image data representing an odd-numbered column image obtained by thinning out pixels of rows of even-numbered columns from the vertical 1 / 2 thinned-out image represented by the vertical thinned-out image data 73. That is, the odd-numbered column image data 73A is image data representing a set of pixels representing odd-numbered columns in the vertical 1 / 2 thinned-out image. And, as an example, Fig. 20 and Fig.21 As shown, the even column image data 73B is image data representing an even column image obtained by thinning out pixels of rows of odd columns from the vertical 1 / 2 thinned-out image represented by the vertical thinned-out image data 73. That is, the even column image data 73B is color image data representing a set of pixels of even columns in the vertical 1 / 2 thinned-out image. In other words, the odd column image data 73A and the even column image data 73B are color image data representing an image in which R pixels, G pixels, and B pixels are arranged periodically, respectively. In addition, the odd column image data 73A and the even column image data 73B are examples of "multiple primary color pixel arrangement image data" and "multiple segmented image data" involved in the technology of the present invention.

[0227] The output circuit 362D outputs the odd-numbered column image data 73A input from the image processing circuit 362C to the first signal processing unit 350 via the first output line 53. The output circuit 362D outputs the even-numbered column image data 73B input from the image processing circuit 362C to the second signal processing unit 52 via the second output line 55.

[0228] In the second signal processing unit 52 , the same processing as that performed on the right image data 70B1 described in the first embodiment is performed on the even-numbered column image data 73B, and the processed even-numbered column image data 73B is sent to the first signal processing unit 350 .

[0229] In the first signal processing section 350, the reception circuit 50C receives the even-numbered column image data 73B transmitted from the second signal processing section 52. The signal processing circuit 350B acquires the even-numbered column image data 73B received by the reception circuit 50C.

[0230] On the other hand, the odd-numbered column image data 73A is input to the buffer 50A. The buffer 50A temporarily holds the odd-numbered column image data 73A and outputs it to the signal processing circuit 350B in a FIFO manner. The signal processing circuit 350B performs specific signal processing on the odd-numbered column image data 73A input from the buffer 50A. Furthermore, the signal processing circuit 350B generates the composite image data 372 by synthesizing the odd-numbered column image data 73A to which the specific signal processing has been performed and the even-numbered column image data 73B obtained from the receiving circuit 50C. As a result, the arrangement pattern of the R pixels, G pixels, and B pixels of the image represented by the composite image data 372 becomes the same arrangement pattern as the vertical 1 / 2 interval elimination image described above. That is, the arrangement pattern of the R pixels, G pixels, and B pixels of the image represented by the composite image data 372 becomes a periodic arrangement pattern that enables the demosaicing of the R pixels, G pixels, and B pixels.

[0231] Therefore, the signal processing circuit 350B performs a demosaicing process on the R, G, and B signals of the synthesized image data 372 , and outputs the synthesized image data 372 on which the demosaicing process has been performed to the controller 46 via the communication line 60 .

[0232] Next, the operation of the imaging device 300 will be described.

[0233] First, refer to Fig. 22 , the imaging process flow executed by the processing circuit 362 of the imaging element 344 is described. Fig. 22 The camera processing shown is Fig.10 Compared with the imaging process shown in FIG. 1 , the difference is that the process of step ST100 is substituted for the process of step ST18 and the process of step ST102 is substituted for the process of step ST20. Fig. 22 In the flowchart of the imaging process shown in FIG. Fig.10 The same steps in the imaging process shown are marked with the same step numbers. Fig. 22 The camera processing shown is only for Fig.10 The differences in the imaging processing shown will be described.

[0234] exist Fig. 22 In the imaging process shown in FIG. 1 , in step ST100, the image processing circuit 362C generates vertical thinning image data 73 (see FIG. 1 ) based on the imaging image data 70. Figure 18 to Figure 21 ). Then, the image processing circuit 362C generates odd-numbered column image data 73A and even-numbered column image data 73B based on the generated vertical thinning image data 73. That is, the vertical thinning image data 73 is separated into odd-numbered column image data 73A and even-numbered column image data 73B (refer to Fig. 20 and Fig.21 ).

[0235] In the next step ST102 , the output circuit 362D outputs the odd-numbered column image data 73A to the first signal processing unit 350 via the first output line 53 . Furthermore, the output circuit 362D outputs the even-numbered column image data 73B to the second signal processing unit 52 via the second output line 55 .

[0236] Next, refer to Fig.23 , the first signal processing flow performed by the first signal processing unit 350 is described.

[0237] exist Fig.23 In the first signal processing shown in FIG. 1 , in step ST210, the first signal processing unit 350 determines whether the odd-numbered column image data 73A (see FIG. 1 ) is input from the processing circuit 362. Fig.18 ). In step ST210, if the odd-numbered column image data 73A is not input from the processing circuit 362, the determination is "No", and the first signal processing is transferred to step ST222. In step ST210, if the odd-numbered column image data 73A is input from the processing circuit 362, the determination is "Yes", and the first signal processing is transferred to step ST212.

[0238] In step ST212 , the first signal processing unit 350 performs specific signal processing on the odd-numbered column image data 73A.

[0239] In the next step ST214, the first signal processing unit 350 determines whether the received Fig.24 The even-numbered column image data 73B (reference Fig.18 ). In step ST214, if the even-numbered column image data 73B is not received, the determination is "No", and the first signal processing moves to step ST220. In step ST214, if the even-numbered column image data 73B is received, the determination is "Yes", and the first signal processing moves to step ST216.

[0240] In step ST220, the first signal processing unit 350 determines whether the first signal processing end condition is satisfied. In step ST220, if the first signal processing end condition is not satisfied, the determination is "No", and the first signal processing is transferred to step ST214. In step ST220, if the first signal processing end condition is satisfied, the determination is "Yes", and the first signal processing ends.

[0241] In step ST216, the first signal processing unit 350 generates synthesized image data 372 (see FIG. 3 ) by performing a synthesis process of synthesizing the odd-numbered column image data 73A obtained by executing the process of step ST212 and the even-numbered column image data 73B received in step ST214. Fig.18 ). Furthermore, the first signal processing unit 350 performs a demosaicing process on the synthesized image data 372.

[0242] In the next step ST218, the first signal processing unit 350 transmits the signal to the first signal processing unit 350 via the communication line 60 (reference Fig.18 ) outputs the synthesized image data 372 obtained by executing the processing of step ST216 to the controller 46 (reference Fig.18 ).

[0243] In the next step ST222, the first signal processing unit 350 determines whether the first signal processing end condition is satisfied. In step ST222, if the first signal processing end condition is not satisfied, the determination is "No", and the first signal processing is transferred to step ST210. In step ST222, if the first signal processing end condition is satisfied, the determination is "Yes", and the first signal processing ends.

[0244] Next, refer to Fig.24 , the second signal processing flow performed by the second signal processing unit 52 is described.

[0245] exist Fig.24 In the second signal processing shown in FIG. 2 , in step ST230, the second signal processing unit 52 determines whether the even-numbered column image data 73B (see FIG. 24 ) is input from the processing circuit 362. Fig.18 ). In step ST230, if the even-numbered column image data 73B is not input from the processing circuit 362, the determination is "No", and the second signal processing is transferred to step ST236. In step ST230, if the even-numbered column image data 73B is input from the processing circuit 362, the determination is "Yes", and the second signal processing is transferred to step ST232.

[0246] In step ST232 , the second signal processing unit 52 performs specific signal processing on the even-numbered column image data 73B.

[0247] In the next step ST234, the second signal processing unit 52 receives the signal through the communication line 58 (reference Fig.18 ) The even-numbered column image data 73B obtained by executing the processing of step ST232 is sent to the first signal processing unit 350.

[0248] In the next step ST236, the second signal processing unit 52 determines whether the second signal processing end condition is satisfied. In step ST236, if the second signal processing end condition is not satisfied, the determination is "No", and the second signal processing is transferred to step ST230. In step ST236, if the second signal processing end condition is satisfied, the determination is "Yes", and the second signal processing ends.

[0249] As described above, in the third embodiment, the camera image data 70 is color image data representing a color camera image in which R pixels, G pixels, and B pixels as a plurality of primary color pixels are periodically arranged. Furthermore, the camera image data 70 is divided into a plurality of primary color pixel arrangement image data as a plurality of image data. Fig.18 and Fig. 20 In the example shown, odd-numbered column image data 73A and even-numbered column image data 73B are shown as a plurality of primary color pixel arrangement image data. The odd-numbered column image data 73A and even-numbered column image data 73B are image data representing an image in which R pixels, G pixels, and B pixels are arranged periodically, respectively. By using image data representing an image in which R pixels, G pixels, and B pixels are arranged periodically, as in the odd-numbered column image data 73A and even-numbered column image data 73B, de-mosaicing of R pixels, G pixels, and B pixels can be achieved.

[0250] And, if Figure 18-19 As shown, the odd-numbered column image data 73A and the even-numbered column image data 73B are image data obtained by dividing the vertical thinning image data 73. The data amount of the vertical thinning image data 73 is smaller than that of the camera image data 70. Therefore, compared with the case where a specific signal processing is performed on one of the two image data obtained by dividing the camera image data 70 without thinning, it is possible to perform specific signal processing on the odd-numbered column image data 73A and the even-numbered column image data 73B at a high speed.

[0251] Furthermore, the odd-numbered column image data 73A is image data representing a set of pixels of odd-numbered columns in the vertical 1 / 2 thinned-out image, and the even-numbered column image data 73B is image data representing a set of pixels of even-numbered columns in the vertical 1 / 2 thinned-out image. Compared with the processing of the odd-numbered column image data 73A and the even-numbered column image data 73B, the specific signal processing and synthesis processing contents of the plurality of image data obtained by irregularly dividing the vertical 1 / 2 thinned-out image are complicated. Therefore, the odd-numbered column image data 73A and the even-numbered column image data 73B can be processed at a high speed compared with the processing of the plurality of image data obtained by irregularly dividing the vertical 1 / 2 thinned-out image.

[0252] Furthermore, the first signal processing unit 350 performs demosaic processing on the synthesized image data 372 obtained by synthesizing the odd-numbered column image data 73A and the even-numbered column image data 73B. Therefore, a higher-quality image can be obtained compared to a case where demosaic processing is not performed.

[0253] In the third embodiment, the first signal processing unit 350 uses the DRAM 54 and the second signal processing unit 52 uses the DRAM 56, but the technology of the present invention is not limited to this. Fig.25 As shown, DRAMs 54 and 56 may not be used, and a first signal processing unit 750 may be used instead of the first signal processing unit 350, and a second signal processing unit 752 may be used instead of the second signal processing unit 52. Fig.25 In the example shown, the first signal processing unit 750 is different from the first signal processing unit 350 in that it has a line memory 750A and does not use the DRAM 54. Also, the second signal processing unit 752 is different from the second signal processing unit 52 in that it has a line memory 752A and does not use the DRAM 56.

[0254] In the first signal processing unit 750, a line memory 750A is provided between the buffer 50A and the signal processing circuit 350B. The buffer 50A outputs the odd-numbered column image data 73A to the line memory 750A. The line memory 750A stores the odd-numbered column image data 73A input from the buffer 50A in units of lines, and outputs the data to the signal processing circuit 350B in a FIFO manner. The signal processing circuit 350B performs the processing described in the third embodiment.

[0255] On the other hand, in the second signal processing unit 752, a line memory 752A is interposed between the buffer 52A and the signal processing circuit 52B. The buffer 52A outputs the even-numbered column image data 73B to the line memory 752A. The line memory 752A stores the even-numbered column image data 73B input from the buffer 52A in units of lines, and outputs it to the signal processing circuit 52B in a FIFO manner. The signal processing circuit 52B performs the processing described in the third embodiment.

[0256] Furthermore, in the third embodiment, image data representing a vertical 1 / 2 thinned image is exemplified as compressed image data, but the technology of the present invention is not limited thereto. For example, when n is a natural number greater than 3, image data representing a vertical 1 / n thinned image may also be used as compressed image data. Furthermore, image data representing a horizontal thinned image thinned in column units may be used as compressed image data, and image data representing an image thinned in row units and column units may also be used as compressed image data.

[0257] [Fourth embodiment]

[0258] In the first embodiment, an example of a method for separating the captured image data 70 into the left image data 70A1 and the right image data 70B1 is described, but in the fourth embodiment, an example of a method for separating the captured image data 70 differently depending on the operation mode is described. In addition, in the fourth embodiment, the same reference numerals are attached to the same components as those in the first embodiment, and the description thereof is omitted. The following describes the parts different from the first embodiment.

[0259] like Figure 1 As shown, the imaging device 400 according to the fourth embodiment is different from the imaging device 10 described in the first embodiment in that it includes an imaging device body 412 instead of the imaging device body 12 .

[0260] The camera body 412 is different from the camera body 12 in that it has an imaging element 444 (see Fig.26 ) instead of the imaging element 44. The imaging element 444 is different from the imaging element 44 in that it has a processing circuit 462 instead of the processing circuit 62. The processing circuit 462 is different from the processing circuit 62 in that it has an image processing circuit 462C instead of the image processing circuit 62C and an output circuit 462D instead of the output circuit 62D.

[0261] The camera body 412 is different from the camera body 12 in that it has a first signal processing unit 450 (see Fig.26 ) instead of the first signal processing unit 50 and having a second signal processing unit 452 (reference Fig.26 ) instead of the second signal processing unit 52.

[0262] The first signal processing unit 450 is different from the first signal processing unit 50 in that it has the functions of the first signal processing unit 50 and the functions of the first signal processing unit 350 described in the third embodiment. Furthermore, the first signal processing unit 450 is different from the first signal processing unit 50 in that the functions of the first signal processing unit 50 and the functions of the first signal processing unit 350 are selectively operated.

[0263] The second signal processing unit 452 is different from the second signal processing unit 52 in that it has the functions of the second signal processing unit 52 and the functions of the second signal processing unit 252 described in the second embodiment. Furthermore, the second signal processing unit 452 is different from the second signal processing unit 52 in that the functions of the second signal processing unit 52 and the functions of the second signal processing unit 252 described in the second embodiment are selectively operated.

[0264] The controller 46 transmits the still image shooting mode signal 480A (reference Fig.26 ) and the shooting mode signal 480B for displaying animation (reference Fig. 27 ) is selectively output to the processing circuit 462. For example, when the receiving unit 84 receives an instruction to set the camera device 400 to the still image shooting mode, the controller 46 outputs the still image shooting mode signal 480A. And, for example, when the receiving unit 84 receives an instruction to set the camera device 400 to the display animation shooting mode, the controller 46 outputs the display animation shooting mode signal 480B.

[0265] like Fig.26 As shown in FIG. 4 , when a still image shooting mode signal 480A is input from the controller 46 via the communication line 60, the processing circuit 462 causes the image processing circuit 462C to operate in the same manner as the image processing circuit 62C described in the first embodiment. That is, the processing circuit 462 causes the image processing circuit 462C to separate the captured image data 70 into the left image data 70A1 and the right image data 70B1, and causes the output circuit 462D to output the left image data 70A1 and the right image data 70B1. The left image 70A1a represented by the left image data 70A1 and the right image 70B1a represented by the right image data 70B1 have the overlapping area 71 described in the first embodiment. In addition, the left image data 70A1 and the right image data 70B1 involved in this fourth embodiment are examples of the "plurality of overlapping image data" involved in the technology of the present invention.

[0266] like Fig.26 As shown, the output circuit 462D outputs the left image data 70A1 to the first signal processing unit 450 via the first output line 53, similarly to the output circuit 62D described in the first embodiment. Also, the output circuit 462D outputs the right image data 70B1 to the second signal processing unit 452 via the second output line 55, similarly to the output circuit 62D described in the first embodiment.

[0267] The first signal processing unit 450 performs specific signal processing on the input left image data 70A1 , and the second signal processing unit 452 performs specific signal processing on the input right image data 70B1 , and outputs the right image data 70B1 subjected to the specific signal processing to the first signal processing unit 450 .

[0268] The first signal processing unit 450 combines the left image data 70A1 subjected to specific signal processing with the right image data 70B1 input from the second signal processing unit 452 to generate the synthesized image data 72 described in the first embodiment. The first signal processing unit 450 outputs the generated synthesized image data 72 to the controller 46 .

[0269] On the other hand, Fig. 27 As shown in FIG. 4 , when the display animation shooting mode signal 480B is input from the controller 46 via the communication line 60, the processing circuit 462 causes the image processing circuit 462C to operate in the same manner as the image processing circuit 362C described in the third embodiment. That is, the processing circuit 462 causes the image processing circuit 462C to compress the captured image data 70 into the vertical thinning image data 73, and then separate the vertical thinning image data 73 into the odd-numbered column image data 73A and the even-numbered column image data 73B. Furthermore, the processing circuit 462 causes the image processing circuit 462C to output the odd-numbered column image data 73A and the even-numbered column image data 73B to the output circuit 462D.

[0270] like Fig. 27 As shown, similarly to the output circuit 362D described in the third embodiment, the output circuit 462D outputs the odd-numbered column image data 73A to the first signal processing unit 450 via the first output line 53. Also, similarly to the output circuit 362D described in the third embodiment, the output circuit 462D outputs the even-numbered column image data 73B to the second signal processing unit 452 via the second output line 55.

[0271] The first signal processing unit 450 performs specific signal processing on the input odd-column image data 73A, and the second signal processing unit 452 performs specific signal processing on the input even-column image data 73B, and outputs the even-column image data 73B subjected to the specific signal processing to the first signal processing unit 450 .

[0272] The first signal processing unit 450 combines the odd-numbered column image data 73A subjected to specific signal processing with the even-numbered column image data 73B input from the second signal processing unit 452 to generate the synthesized image data 372 described in the third embodiment. The first signal processing unit 450 outputs the generated synthesized image data 372 to the controller 46 .

[0273] Next, the operation of the imaging device 400 will be described.

[0274] First, refer to Fig.28 , the imaging process flow executed by the processing circuit 462 of the imaging element 444 will be described. In addition, for the sake of convenience, the description here is based on the premise that the imaging device 400 is set to the still image shooting mode or the video display shooting mode.

[0275] exist Fig.28 In the imaging process shown, first, in step ST300, the image processing circuit 462C determines whether the imaging device 400 is in the still image shooting mode. In step ST300, if the imaging device 400 is not in the still image shooting mode, that is, if the imaging device 400 is in the shooting mode for displaying moving images, the determination is "No", and the imaging process moves to step ST304. In step ST300, if the imaging device 400 is in the still image shooting mode, the determination is "Yes", and the imaging process moves to step ST302.

[0276] In step ST302, the processing circuit 462 executes the still image pickup process, and then the pickup process moves to step ST306. The still image pickup process is the same as the pickup process described in the first embodiment (see FIG. 1 ). Fig.10 )Same treatment.

[0277] In step ST304, the processing circuit 462 executes the image capture processing for displaying an animated image, and then the image capture processing is transferred to step ST306. The image capture processing for displaying an animated image is the same as the image capture processing described in the third embodiment (see FIG. Fig. 22 )Same treatment.

[0278] In step ST306, the processing circuit 462 determines whether the imaging process end condition is satisfied. In step ST306, if the imaging process end condition is not satisfied, the determination is "No", and the imaging process is transferred to step ST300. In step ST306, if the imaging process end condition is satisfied, the determination is "Yes", and the imaging process ends.

[0279] In the still image shooting mode, the first signal processing unit 450 performs the first signal processing described in the first embodiment (see FIG. 1 ). Fig.11 ). In the still image shooting mode, the second signal processing unit 452 performs the same processing as that described in the first embodiment (see Fig.12 )Same treatment.

[0280] On the other hand, in the image capture mode for displaying moving images, the first signal processing unit 450 performs the first signal processing ( Fig.23 ) is performed. In addition, in the shooting mode for displaying an animated image, the second signal processing unit 452 performs the second signal processing ( Fig.24 )Same treatment.

[0281] As described above, in the fourth embodiment, in the still image shooting mode, the captured image data 70 is separated into the left image data 70A1 and the right image data 70B1 described in the first embodiment. In addition, in the video display shooting mode, the captured image data 70 is separated into the odd-numbered column image data 73A and the even-numbered column image data 73B described in the third embodiment. Thus, the balance between the still image shooting mode and the video display shooting mode can be made different.

[0282] Furthermore, in the still image shooting mode, the captured image data 70 is separated into the left image data 70A1 and the right image data 70B1, each of which includes image data representing the overlapping region 71. In contrast, in the animation display shooting mode, the captured image data 70 is separated in row units. Therefore, in the still image shooting mode, since the left image data 70A1 and the right image data 70B1, each of which includes image data representing the overlapping region 71, are processed, the image quality can be improved compared with the animation display shooting mode. Furthermore, in the animation display shooting mode, the odd-numbered column image data 73A and the even-numbered column image data 73B, which have less data volume than the left image data 70A1 and the right image data 70B1, are processed. Therefore, in the animation display shooting mode, power consumption can be suppressed and the processing speed can be improved compared with the still image shooting mode.

[0283] In the third embodiment described above, step ST302 (reference Fig.28 ) is processed, the imaging process described in the first embodiment above is applied (reference Fig.10 ) is described above, but the technology of the present invention is not limited to this. For example, as step ST302 (reference Fig.28 ) processing, the imaging processing described in the second embodiment above may also be applied (refer to Fig.15 In this case, in the still image shooting mode, the first signal processing unit 450 executes the first signal processing described in the second embodiment (see Fig.16 ), in the second signal processing unit 452, the second signal processing described in the second embodiment is performed (refer to Fig.17 ).

[0284] Furthermore, in the third embodiment, an example of separating the captured image data 70 into the left image data 70A1 and the right image data 70B1 in the still image shooting mode is given for explanation, but the technology of the present invention is not limited to this. For example, in the still image shooting mode, as described in the second embodiment, the captured image data 70 may be separated into the background area image data 70A2 and the face area image data 70B2. In this case, in the still image shooting mode, the first signal processing unit 450 may be operated in the same manner as the first signal processing unit 250 described in the second embodiment, and the second signal processing unit 452 may be operated in the same manner as the second signal processing unit 252 described in the second embodiment.

[0285] [Fifth embodiment]

[0286] In the third embodiment, an example of a method in which the captured image data 70 is compressed into odd-numbered column image data 73A and even-numbered column image data 73B is described. However, in the fifth embodiment, an example of a method in which the captured image data 70 is compressed into two image data by another method is described. In addition, in the fifth embodiment, the same reference numerals are given to the same components as those in the third embodiment, and their description is omitted. The following describes the parts different from the third embodiment.

[0287] like Figure 1 As shown, the imaging device 500 according to the fifth embodiment is different from the imaging device 300 described in the third embodiment in that it includes an imaging device body 512 instead of the imaging device body 312 .

[0288] The camera body 512 is different from the camera body 312 in that it has an imaging element 544 (see Fig.29 ) instead of imaging element 344. Imaging element 544 differs from imaging element 344 in that it includes processing circuit 562 instead of processing circuit 362. Processing circuit 562 differs from processing circuit 362 in that it includes image processing circuit 562C instead of image processing circuit 362C and output circuit 562D instead of output circuit 362D.

[0289] The camera body 512 is different from the camera body 312 in that it has a first signal processing unit 550 (see Fig.29 ) to replace the first signal processing unit 50.

[0290] The memory 64 stores the captured image data 570, and the image processing circuit 562C obtains the captured image data 570 from the memory 64. The captured image data 570 is color image data having R pixels, G pixels, and B pixels. Fig.30 As shown, in the captured image data 570, R pixels, G pixels, and B pixels are arranged in a periodicity corresponding to the Bayer arrangement.

[0291] exist Fig.30 In the example shown, in the first row, R pixels and G pixels are sequentially arranged in a cyclic manner with R pixels and G pixels in the row direction. Also, in the second row, B pixels and G pixels are sequentially arranged in a cyclic manner with G pixels and B pixels in the row direction. Furthermore, the arrangement pattern of R pixels and G pixels in the first row is repeated in the column direction every other row, and the arrangement pattern of B pixels and G pixels in the second row is repeated in the column direction every other row, thereby forming the arrangement pattern of R pixels, G pixels, and B pixels of the entire captured image data 570.

[0292] The image processing circuit 562C compresses the captured image data 570 obtained from the memory 64. That is, the image processing circuit 562C generates vertical thinning image data 573 based on the captured image data 570. As an example, Fig.30 As shown, the vertical thinning image data 573 is image data indicating a vertical 1 / 2 thinning image obtained by thinning out two adjacent rows in the column direction from the captured image indicated by the captured image data 570 every two rows.

[0293] As an example, Fig.29 and Fig.31 As shown, the image processing circuit 562C separates the vertical thinning image data 753 into the first horizontal thinning image data 573A and the second horizontal thinning image data 573B. The image processing circuit 562C outputs the separated first horizontal thinning image data 573A and the second horizontal thinning image data 573B to the output circuit 562D.

[0294] As an example, Fig.31 As shown, the first horizontal thinning image data 573A is image data representing one of a pair of horizontal 1 / 2 thinning images obtained by alternately thinning out every 2 columns in the row direction in units of 2 columns from the vertical 1 / 2 thinning image represented by the vertical thinning image data 573. And, the second horizontal thinning image data 573B is image data representing the other of the pair of horizontal 1 / 2 thinning images.

[0295] The output circuit 562D outputs the first horizontal thinning image data 573A input from the image processing circuit 562C to the first signal processing unit 550 via the first output line 53. Furthermore, the output circuit 562D outputs the second horizontal thinning image data 573B input from the image processing circuit 562C to the second signal processing unit 52 via the second output line 55.

[0296] In the second signal processing unit 52 , the second horizontal thinning image data 573B is processed in the same manner as the processing performed on the even-numbered column image data 73B described in the third embodiment, and the processed second horizontal thinning image data 573B is sent to the first signal processing unit 550 .

[0297] The first signal processing unit 550 receives the second horizontal thinning image data 573B transmitted from the second signal processing unit 52 .

[0298] In the first signal processing unit 550, the first horizontal thinning image data 573A is subjected to the same processing as that performed on the odd-numbered column image data 73A described in the third embodiment. Furthermore, in the first signal processing unit 550, the first horizontal thinning image data 573A and the second horizontal thinning image data 573B are synthesized to generate synthesized image data 572. As a result, the arrangement pattern of R pixels, G pixels, and B pixels of the image represented by the synthesized image data 572 becomes the same arrangement pattern as that of the vertical 1 / 2 thinning image represented by the vertical thinning image data 573. That is, the arrangement pattern of R pixels, G pixels, and B pixels of the image represented by the synthesized image data 572 becomes a periodic arrangement pattern that enables de-mosaicing of the R pixels, G pixels, and B pixels.

[0299] Therefore, similarly to the third embodiment, the first signal processing unit 550 performs demosaicing of the R, G, and B signals on the synthesized image data 572 , and outputs the demosaiced synthesized image data 572 to the controller 46 via the communication line 60 .

[0300] As described above, the captured image data 570 adopts the Bayer arrangement. In this case, as well as the odd-numbered column image data 73A and the even-numbered column image data 73B described in the third embodiment, the first horizontal thinning image data 573A and the second horizontal thinning image data 573B are obtained as two image data that can be demosaiced. Therefore, even if the captured image data 570 is the image data of the Bayer arrangement, the same effect as the third embodiment can be obtained.

[0301] In the fifth embodiment, the first horizontal thinning image data 573A and the second horizontal thinning image data 573B are exemplified as two image data obtained by compressing the captured image data 570, but the technology of the present invention is not limited to this. Fig.32 As shown, the image processing circuit 562C may separate the captured image data 570 into high-order data 570C and low-order data 570D. When the number of bits per pixel of the captured image data 570 is 16 bits, the high-order data 570C is, for example, image data with the upper 8 bits per pixel, and the low-order data 570D is, for example, image data with the lower 8 bits per pixel.

[0302] like Fig.32 As shown, the output circuit 562D outputs the upper data 570C to the first signal processing unit 550 via the first output line 53, and outputs the lower data 570D to the second signal processing unit 52 via the second output line 55. Also, similarly to the fifth embodiment described above, the second signal processing unit 52 performs specific signal processing on the lower data 570D, and then transmits the lower data 570D to the first signal processing unit 550 via the communication line 58. Also, similarly to the fifth embodiment described above, the first signal processing unit 550 receives the lower data 570D transmitted from the second signal processing unit 52, and performs specific signal processing on the upper data 570C. Similar to the fifth embodiment described above, the first signal processing unit 550 generates the synthesized image data 572A by synthesizing the upper data 570C subjected to the specific signal processing and the received lower data 570D. The first signal processing unit 550 outputs the generated synthesized image data 572A to the controller 46.

[0303] In addition, here, the case where the number of bits per pixel of the captured image data 570 is 16 bits is exemplified, but the present invention is not limited thereto, and the number of bits per pixel may be less than 16 bits, or the number of bits per pixel may be more than 16 bits. Furthermore, the method of dividing the high bit and the low bit may be a method determined according to the application and / or specification, etc.

[0304] Furthermore, here, an example is given in which the image processing circuit 562C separates the camera image data 570 into high-order data 570C and low-order data 570D, but the technology of the present invention is not limited to this. For example, the camera image data 570 can also be separated into high-order image data, middle-order image data, and low-order image data. The high-order image data, the middle-order image data, and the low-order image data refer to three compressed image data obtained by dividing the camera image data 570 within a 3-bit range and compressing it. Furthermore, the camera image data 570 can also be compressed by dividing the camera image data 570 within a 4 or more bit range. In this way, the camera image data 570 can also be divided within a plurality of bit ranges to obtain a plurality of compressed image data.

[0305] In this way, the camera image data 570 is separated into a plurality of compressed image data (in Fig.32 In the example shown, the upper data 570C and the lower data 570D). Thus, compared with the case of processing irregularly divided image data, each of the first signal processing unit 550 and the second signal processing unit 52 can perform high-speed processing.

[0306] And, if Fig.32 As shown, by separating the captured image data 570 into upper data 570C and lower data 570D, the upper data 570C can be processed with higher precision than the lower data 570D. On the other hand, the lower data 570D can reduce power consumption and increase processing speed compared to the upper data 570C.

[0307] Furthermore, in each of the above-mentioned embodiments, an example of a method in which the camera image data 70, 570 (hereinafter referred to as "camera image data") is separated into two image data is given for explanation, but the technology of the present invention is not limited to this. For example, when N is set to a natural number greater than 2, the camera image data may also be separated into N image data. As a separation method, for example, in addition to a method of dividing the camera image data into N equal parts, a method of separating the camera image data within an N-bit range is also considered.

[0308] When the camera image data is separated into N image data, for example, Fig.33 As shown, the first signal processing unit 650 to the Nth signal processing unit 650N are connected to the imaging element 44 through corresponding output lines. Fig.33In the example shown, the first signal processing unit 650 is connected to the imaging element 44 via the first output line 53, and the Nth signal processing unit 650N is connected to the imaging element 44 via the Nth output line 53N. The Nth signal processing unit 650N sends the image data that has completed the signal processing to the first signal processing unit 650, and the first signal processing unit 650 synthesizes the N image data and outputs the synthesized synthesized image data to the controller 46.

[0309] Furthermore, in the above-mentioned embodiments, the processing circuits 62, 262, 362, 462, 562 (hereinafter referred to as "processing circuits") are implemented by ASICs, but the technology of the present invention is not limited thereto. For example, the above-mentioned imaging processing may also be implemented by a computer software structure.

[0310] In this case, for example, Fig.34 As shown, a video processing program 802 is stored in a storage medium 800, and the video processing program 802 is used to enable a computer 652 built into an imaging element 44, 244, 344, 444, 544 to perform the above-mentioned video processing. The computer 652 includes a CPU 652A, a ROM 652B, and a RAM 652C. Furthermore, the video processing program 802 of the storage medium 800 is installed in the computer 652, and the CPU 652A of the computer 652 performs the above-mentioned video processing according to the video processing program 802. Here, one CPU is illustrated as the CPU 652A, but the technology of the present invention is not limited to this, and multiple CPUs can also be used instead of the CPU 652A. In addition, as an example of the storage medium 800, any portable storage medium such as an SSD or a USB memory can be cited.

[0311] exist Fig.34 In the example shown, the image processing program 802 is stored in the storage medium 800, but the technology of the present invention is not limited to this. For example, the image processing program 802 may be pre-stored in the ROM 652B, the CPU 652A may read the image processing program 802 from the ROM 652B, expand it to the RAM 652C, and execute the expanded image processing program 802.

[0312] Furthermore, the image processing program 802 may be stored in a storage unit such as another computer or a server device connected to the computer 652 via a communication network (not shown), and the image processing program 802 may be downloaded to the computer 652 in response to a request from the image pickup device 700 having the same structure as any one of the image pickup devices 10, 200, 300, 400, and 500. In this case, the downloaded image processing program 802 is executed by the CPU 652A of the computer 652.

[0313] Furthermore, the computer 652 may be provided outside the imaging element 44, 244, 344, 444, 544. In this case, the computer 652 may control the processing circuit according to the imaging processing program 802.

[0314] As hardware resources for executing the image processing described in the above embodiments, various processors shown below can be used. As a processor, for example, a general-purpose processor, i.e., a CPU, can be cited, which functions as a hardware resource for executing the image processing by executing software, i.e., a program, as described above. Furthermore, as a processor, for example, a dedicated circuit as a processor can be cited, and the processor has a circuit structure specially designed for executing specific processing, such as an FPGA, a PLD, or an ASIC.

[0315] The hardware resource for executing the video processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware resource for executing the various processing involved in the technology of the present invention may be a processor.

[0316] As an example of a system composed of one processor, there is the following method: as represented by computers such as clients and servers, a processor is composed of a combination of one or more CPUs and software, and the processor functions as a hardware resource for executing processing within the imaging element. Second, there is the following method: as represented by SoC (System-on-a-chip), a processor that implements the functions of the entire system including multiple hardware resources for executing camera processing is used by one IC chip. In this way, the processing within the imaging element is realized by using one or more of the above-mentioned various processors as hardware resources.

[0317] Furthermore, as the hardware configuration of these various processors, more specifically, a circuit combining circuit elements such as semiconductor elements can be used.

[0318] Furthermore, in the above-mentioned embodiments, a lens-interchangeable camera is exemplified as the imaging device, but the technology of the present invention is not limited to this. Fig.35 As an example, Fig.35The smart device 900 shown is an example of a camera device involved in the technology of the present invention. The imaging elements 44, 244, 344, 444, and 544 described in the above embodiments are mounted in the smart device 900. Even the smart device 900 constructed in this way can obtain the same function and effect as the camera device described in the above embodiments. In addition, the technology of the present invention is not limited to the smart device 900, but can also be applied to a personal computer or a wearable terminal device.

[0319] Furthermore, in the above-mentioned embodiments, the first display 32 and the second display 86 are exemplified, but the technology of the present invention is not limited thereto. For example, a separate display attached to the camera body 12 may be used as the "display unit" involved in the technology of the present invention.

[0320] Furthermore, the above-mentioned imaging process, the first signal process and the second signal process are merely examples, and therefore, it is of course possible to delete unnecessary steps, add new steps, or switch the processing order without departing from the scope of the subject matter.

[0321] The recorded contents and illustrated contents shown above are detailed descriptions of the parts involved in the technology of the present invention, and are only an example of the technology of the present invention. For example, the descriptions related to the above-mentioned structure, function, action and effect are descriptions related to an example of the structure, function, action and effect of the parts involved in the technology of the present invention. Therefore, without departing from the scope of the main purpose of the technology of the present invention, it is of course possible to delete unnecessary parts of the recorded contents and illustrated contents shown above, or to add new elements, or to replace them. In addition, in order to avoid complication and to make it easy to understand the parts involved in the technology of the present invention, in the recorded contents and illustrated contents shown above, descriptions related to technical common sense, etc. that do not need to be specifically explained in terms of the technology that can implement the present invention are omitted.

[0322] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" may mean only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" is added to express three or more items, the same concept as "A and / or B" may also be applied.

[0323] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each document, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0324] Regarding the above embodiment, the following supplementary notes are further disclosed.

[0325] (Note 1)

[0326] A camera device (10, 200, 300, 400, 500, 700) having an imaging element (44, 244, 344, 444, 544), comprising:

[0327] a storage unit (64) which stores captured image data (70, 570) obtained by capturing an object with an imaging element (44, 244, 344, 444, 544) and is built into the imaging element (44, 244, 344, 444, 544);

[0328] a processing unit (62, 262, 362, 462, 562) which processes the camera image data (70, 570) and is built into the imaging element (44, 244, 344, 444, 544);

[0329] an output unit (62D, 262D, 362D, 462D, 562D) which outputs processed image data obtained by processing the camera image data (70, 570) by the processing unit (62, 262, 362, 462, 562) and is built into the imaging element (44, 244, 344, 444, 544); and

[0330] A plurality of signal processing units (50, 250, 350, 450, 550, 650, 52, 252, 452) are arranged outside the imaging element,

[0331] The processing unit (62, 262, 362, 462, 562) performs processing to divide the captured image data (70, 570) stored in the storage unit (64) into a plurality of image data (70A, 70B).

[0332] The output unit (62D, 262D, 362D, 462D, 562D) has a plurality of output lines (53, 55) respectively provided corresponding to each of the plurality of signal processing units (50, 250, 350, 450, 550, 650, 52, 252, 452), and outputs each of the plurality of image data (70A, 70B) as processed image data from the plurality of output lines (53, 55) to a corresponding signal processing unit among the plurality of signal processing units (50, 250, 350, 450, 550, 650, 52, 252, 452, 750, 752).

[0333] Any one of a plurality of signal processing units (50, 250, 350, 450, 550, 650, 52, 252, 452, 750, 752) synthesizes a plurality of image data (70A, 70B).

[0334] (Note 2)

[0335] The camera device (200) according to Supplement 1, wherein:

[0336] The plurality of image data (70A, 70B) are image data respectively representing images having overlapping areas (71, 271) between adjacent images (70A1a, 70B1a, 70A2a, 70B2a) in a plurality of images (70A1a, 70B1a, 70A2a, 70B2a) based on each of the plurality of image data (70A, 70B).

[0337] (Note 3)

[0338] The camera device (200) according to Appendix 1 or 2, wherein:

[0339] A plurality of images (70A2a, 70B2a) are divided into a designated image (70B2a) and images different from the designated image.

[0340] (Note 4)

[0341] The camera device (200) according to Supplement 3, wherein:

[0342] The processing unit detects facial image data representing a facial image (69) from the camera image data (70), and the designated image (70B2a) is an image including the facial image (69) represented by the facial image data detected by the processing unit (262) in the camera image represented by the camera image data (70).

[0343] (Note 5)

[0344] The camera device (400) according to any one of Supplementary Notes 1 to 4, wherein:

[0345] A processing unit (462) changes the method of dividing the captured image data in a recording shooting mode and a video display shooting mode.

[0346] (Note 6)

[0347] The imaging device according to Supplement 5, wherein:

[0348] The processing unit (462) divides the captured image data (70) as a plurality of image data into a plurality of repeated image data (70A1, 70B1) in a recording shooting mode, and divides the captured image data (70) into line units in a display animation shooting mode.

[0349] The plurality of overlapping image data are image data representing images having overlapping regions (71) between adjacent images among the plurality of images.

Claims

1. An imaging device comprising an image sensor, wherein the imaging device include: a memory storing photographic image data obtained by photographing a subject with the image sensor, and built in the image sensor; A processor built into the image sensor; and A plurality of signal processing processors are arranged outside the image sensor, The processor has an output circuit built in. The output circuit has a plurality of output lines respectively provided corresponding to the plurality of signal processing processors, and outputs a plurality of image data obtained by dividing the camera image data stored in the memory to corresponding signal processing processors among the plurality of signal processing processors from the plurality of output lines. Any one of the plurality of signal processing processors synthesizes the plurality of image data, The camera image data stored in the memory is image data digitized by the processor. The image sensor is an image sensor in which at least a photoelectric conversion element and the memory are integrated into a single chip. The image sensor is a stacked type image sensor in which the memory is stacked on the photoelectric conversion element. The method of dividing the captured image data is different between the recording shooting mode and the video display shooting mode. In the recording imaging mode, the captured image data is divided into a plurality of repeated image data as the plurality of image data, and in the display moving image imaging mode, the captured image data is divided in line units.

2. The imaging device according to claim 1, in, The plurality of image data respectively represent images having overlapping regions between adjacent images among the images based on the plurality of image data.

3. The imaging device according to claim 1 or 2, in, A plurality of images are divided into a designated image and an image different from the designated image.

4. The imaging device according to claim 3, further comprising: include: a detection processor that detects facial image data representing a facial image from the camera image data, The designated image includes a face image represented by the face image data detected by the detection processor, among the captured images represented by the captured image data.

5. The imaging device according to claim 1, in, The plurality of overlapping image data respectively represent images having overlapping areas between adjacent images among the plurality of images.

6. The imaging device according to claim 1, in, The recording imaging mode is an operation mode in which the image sensor is caused to perform still image imaging.

7. The imaging device according to any one of claims 1, 2, 4, 5 and 6, further comprising: include: A control processor performs control to cause a display to display an image based on the plurality of image data output by the processor.

8. A method for processing image data of a camera device, wherein the camera device include: Image sensor; a memory storing photographic image data obtained by photographing a subject with the image sensor, and built in the image sensor; A processor built into the image sensor; and a plurality of signal processing processors disposed outside the image sensor, the image data processing method of the camera device comprising the following processing: outputting a plurality of image data obtained by dividing the camera image data stored in the memory to corresponding signal processing processors among the plurality of signal processing processors from a plurality of output lines provided in the processor corresponding to each of the plurality of signal processing processors, respectively; Any one of the plurality of signal processing processors synthesizes the plurality of image data, The processor has an output circuit built in. The output circuit has the plurality of output lines, The camera image data stored in the memory is image data digitized by the processor. The image sensor is an image sensor in which at least a photoelectric conversion element and the memory are integrated into a single chip. The image sensor is a stacked type image sensor in which the memory is stacked on the photoelectric conversion element. The method of dividing the captured image data is different between the recording shooting mode and the video display shooting mode. In the recording imaging mode, the captured image data is divided into a plurality of repeated image data as the plurality of image data, and in the display moving image imaging mode, the captured image data is divided in line units.

9. A computer-readable storage medium storing a program for causing a computer used in an image pickup device to execute the following processing, wherein the image pickup device include: Image sensor; a memory storing photographic image data obtained by photographing a subject with the image sensor, and built in the image sensor; a processor built in the image sensor; and a plurality of signal processing processors disposed outside the image sensor, the processor having an output circuit built in, the output circuit having a plurality of output lines disposed in the processor corresponding to each of the plurality of signal processing processors, the captured image data stored in the memory being image data digitized by the processor, the processing comprising: outputting a plurality of image data obtained by dividing the camera image data stored in the memory to corresponding signal processing processors among the plurality of signal processing processors from a plurality of output lines provided in the processor corresponding to each of the plurality of signal processing processors, respectively; Any one of the plurality of signal processing processors synthesizes the plurality of image data, The image sensor is an image sensor in which at least a photoelectric conversion element and the memory are integrated into a single chip. The image sensor is a stacked type image sensor in which the memory is stacked on the photoelectric conversion element. The method of dividing the captured image data is different between the recording shooting mode and the video display shooting mode. In the recording imaging mode, the captured image data is divided into a plurality of repeated image data as the plurality of image data, and in the display moving image imaging mode, the captured image data is divided in line units.

Images