Imaging element, camera device, working method of imaging element and storage medium
By building a storage and control unit in the imaging element, and stacking the photoelectric conversion element, processing circuit and memory, the problem of low image data acquisition and management efficiency in the prior art is solved, and efficient image data transmission and processing are realized.
Patent Information
- Application Number
- CN202080052874.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-07-26
- Filing Date
- 2020-07-01
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-07-01
AI Technical Summary
When the existing imaging elements acquire and process image data, it is difficult to selectively acquire the required image data, and there are challenges in data transmission efficiency and storage management.
An imaging element is designed, with a built-in storage unit and a control unit, which can selectively store and output attribute information of image data according to external instructions, and can single-chip the photoelectric conversion element, processing circuit and memory through a stacked structure to improve data transmission. Speed and storage management efficiency.
Efficient selective acquisition and management of image data is realized, the power consumption required for output is reduced, the portability and design freedom of imaging elements are improved, and the transmission and processing of image data are accelerated.
Smart Images

Figure CN114208159B_ABST
Abstract
Description
Technical Field
[0001] The technology of the present invention relates to an imaging element, a camera device, a working method of the imaging element and a storage medium. Background Art
[0002] International Publication No. 2014 / 007004 discloses a solid-state imaging element, which comprises: a signal processing unit, including an AD converter that digitizes analog pixel signals read from each pixel of a pixel array unit to a signal line, and transmits the digitized pixel data at a first speed faster than a frame rate; a memory unit that holds the pixel data transmitted from the signal processing unit; a data processing unit that reads the pixel data from the memory unit at a second speed slower than the first speed; and a control unit that controls the operation of a current source connected to the signal line and the operation of at least the AD converter of the signal processing unit to stop when the pixel data is read from the memory unit.
[0003] In the solid-state imaging element described in International Publication No. 2014 / 007004, the signal processing unit, the memory unit, the data processing unit, and the control unit have a structure in which the chip formed on the pixel array unit is different from the chip formed on the pixel array unit, and the chip formed on the pixel array unit and the other at least one chip are stacked. In addition, in the solid-state imaging element described in International Publication No. 2014 / 007004, the data processing unit has a decoder that specifies a column address to the memory unit and a readout amplifier that reads out pixel data of the specified address, and the pixel data is read out from the memory unit through the readout amplifier and the decoder.
[0004] International Publication No. 2013 / 145765 discloses a camera unit comprising: an image pickup section having a first group including one or more pixels and a second group including one or more pixels different from the pixels constituting the first group; and a control section which, while causing the first group to perform one charge accumulation, causes the second group to perform a different number of charge accumulations from the first group and outputs respective pixel signals.
[0005] In the camera unit described in International Publication No. 2013 / 145765, an imaging chip including a camera section and a signal processing chip including a processing circuit for processing pixel signals are electrically connected through a stacked structure. Furthermore, a memory chip including a pixel memory for storing pixel signals is also electrically connected through a stacked structure. Furthermore, in the camera unit described in International Publication No. 2013 / 145765, a control unit reads out pixel signals of a specified group from the pixel memory and delivers them to an image processing unit in accordance with a delivery request from an external circuit for a specified group in a group including a second group. Furthermore, the camera unit described in International Publication No. 2013 / 145765 is provided with a data transmission interface for transmitting pixel signals in accordance with a delivery request. The transmission of pixel signals using the data transmission interface adopts at least one of a double data rate method, an address designation method, a short pulse transmission method, a bus method, and a serial method. In the imaging unit described in International Publication No. 2013 / 145765, when each pixel signal for a plurality of charge accumulations is stored in the pixel memory, the control unit delivers each pixel signal to the image processing unit. Summary of the invention
[0006] One embodiment of the technology according to the present invention provides an imaging element, an imaging device, an operating method of the imaging element, and a storage medium capable of selectively acquiring necessary image data from the imaging element.
[0007] Means for solving technical problems
[0008] The first method involved in the technology of the present invention is an imaging element, which includes: a storage unit, which stores image data obtained by shooting and is built into the imaging element; a control unit, which controls the storage of the image data in the storage unit, stores attribute information of the image data in the storage unit, and is built into the imaging element; an output unit, which outputs the image data stored in the storage unit and is built into the imaging element; and a receiving unit, which receives instructions related to the attribute information, and the output unit outputs the attribute information corresponding to the instructions received by the receiving unit.
[0009] A second aspect according to the technology of the present invention is the imaging element according to the first aspect, wherein the output unit outputs the attribute information at a timing when the instruction is received by the reception unit.
[0010] A third aspect according to the technology of the present invention is the imaging element according to the first aspect or the second aspect, wherein the instruction is a frame synchronization signal from the outside.
[0011] A fourth aspect according to the technology of the present invention is the imaging element according to any one of the first aspect to the third aspect, wherein the output unit outputs attribute information on the latest image data among the image data stored in the storage unit.
[0012] A fifth aspect according to the technology of the present invention is the imaging element according to the fourth aspect, wherein the output unit outputs the attribute information on the latest image data at a timing when the receiving unit receives an instruction.
[0013] A sixth aspect of the technology of the present invention is the imaging element according to any one of the first to fifth aspects, wherein the output unit is capable of outputting attribute information of each of the plurality of image data, and the attribute information is output by the output unit in the order of imaging.
[0014] A seventh aspect according to the technology of the present invention is the imaging element according to any one of the first to sixth aspects, wherein the attribute information is information including at least one of an address, an image size, an imaging time, and an imaging condition.
[0015] The 8th mode involved in the technology of the present invention is an imaging element described in any one of the 1st to 7th modes, wherein, when the image data is erased from the storage unit accompanying the next camera shot, the attribute information output from the output unit is information including erasure information indicating that the image data is erased from the storage unit accompanying the next camera shot.
[0016] The 9th method involved in the technology of the present invention is the imaging element involved in the 8th method, wherein, when erasing image data from the storage unit, the control unit erases attribute information related to the erasure object image data which is the image data to be erased in the storage unit, and erases the erasure object image data from the storage unit.
[0017] The 10th method involved in the technology of the present invention is an imaging element described in any one of the 1st to 9th methods, which includes a compression circuit for compressing image data, a control unit stores image data of a specified number of frames in a storage unit, stores compressed image data obtained by compressing the image data by the compression circuit in the storage unit, and establishes a corresponding association between attribute information related to the compressed image data and compression determination information that can determine that the image data is compressed and the compressed image data.
[0018] The 11th mode involved in the technology of the present invention is the imaging element involved in the 10th mode, wherein, when compressed image data is stored in the storage unit, the attribute information output from the output unit is information including compression scheduled information indicating that the compressed image data is scheduled to be stored in the storage unit.
[0019] A twelfth aspect according to the technology of the present invention is the imaging element according to the tenth aspect or the eleventh aspect, wherein the attribute information output from the output unit is information including information capable of specifying a compression format of the compressed image data.
[0020] A thirteenth aspect according to the technology of the present invention is the imaging element according to any one of the first aspect to the twelfth aspect, wherein the instruction includes an output amount based on the attribute information of the output unit, and the output unit outputs the attribute information at the output amount.
[0021] A fourteenth aspect according to the technology of the present invention is the imaging element according to the thirteenth aspect, wherein the output amount is defined by the number of frames of image data.
[0022] The 15th method involved in the technology of the present invention is an imaging element described in any one of the 1st to 14th methods, which also has a receiving unit, which receives attribute information sent from a subsequent circuit located at the subsequent stage of the imaging element, and the control unit obtains image data from the storage unit based on the attribute information received by the receiving unit, and causes the output unit to output the acquired image data to the subsequent circuit.
[0023] A 16th aspect involved in the technology of the present invention is an imaging element according to any one of the 1st to 15th aspects, wherein the output unit has a first output unit and a second output unit, the first output unit outputs image data, and the second output unit outputs attribute information.
[0024] A seventeenth aspect according to the technology of the present invention is the imaging element according to any one of the first aspect to the sixteenth aspect, wherein at least the photoelectric conversion element and the storage unit are integrated into a single chip.
[0025] An eighteenth aspect according to the technology of the present invention is the imaging element according to the seventeenth aspect, wherein the imaging element is a stacked-type imaging element in which a storage unit is stacked on a photoelectric conversion element.
[0026] The 19th method involved in the technology of the present invention is a camera device, which includes: an imaging element described in any one of the 1st to 18th methods; and a control device, which performs at least one of controlling a display unit to display an image based on image data output by an output unit and controlling the image data output by the output unit to store in a storage device.
[0027] The 20th method involved in the technology of the present invention is a working method of an imaging element, which has a built-in storage unit for storing image data obtained by shooting, and the working method of the imaging element includes: controlling the storage of image data in the storage unit; storing attribute information of the image data in the storage unit; outputting the image data stored in the storage unit; receiving instructions related to the attribute information; and outputting the attribute information corresponding to the received instructions.
[0028] The 21st method involved in the technology of the present invention is a program for causing a computer to perform processing, and the computer is suitable for an imaging element having a built-in storage unit for storing image data obtained by shooting, and the processing includes: controlling the storage of image data in the storage unit; storing attribute information of the image data in the storage unit; outputting the image data stored in the storage unit; receiving instructions related to the attribute information; and outputting the attribute information corresponding to the received instructions. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is a perspective view showing an example of the appearance of the imaging device according to the embodiment.
[0030] Figure 2 Yes means Figure 1 The figure is a rear view showing an example of the appearance of the rear side of the imaging device shown.
[0031] Figure 3 This is a block diagram showing an example of the configuration of an imaging device according to an embodiment.
[0032] Figure 4 This is a block diagram showing an example of a hardware configuration of an electrical system of an imaging device main body included in the imaging device according to the embodiment.
[0033] Figure 5 This is a conceptual diagram for explaining the frame rate of an imaging element included in the imaging device according to the embodiment.
[0034] Figure 6 This is a conceptual diagram showing an example of a stacked structure of an imaging element according to an embodiment.
[0035] Figure 7 This is a block diagram showing an example of the structure of a memory included in the imaging element according to the embodiment.
[0036] Figure 8 This is a block diagram showing an example of a hardware configuration of an electrical system of an imaging element included in an imaging device according to an embodiment, and is a block diagram for explaining an example of a method of capturing an image with the imaging element and outputting attribute information from the imaging element.
[0037] Fig. 9 This is a conceptual diagram showing an example of the contents of an imaging process and an output process performed by the imaging element according to the embodiment.
[0038] Fig.10 This is a timing chart showing an example of processing content in an imaging element included in the imaging device according to the embodiment.
[0039] Fig.11AThis is a conceptual diagram showing an example of the content of attribute information output from the output I / F of the imaging element in the first to fourth frames.
[0040] Fig. 11B This is a conceptual diagram showing an example of the content of attribute information output from the output I / F of the imaging element in the fifth and sixth frames.
[0041] Fig.12 This is a block diagram showing an example of a hardware configuration of an electrical system of an imaging element included in the imaging device according to the embodiment, and is a block diagram for explaining an example of a method of outputting digital image data from the imaging element.
[0042] Fig.13 This is a flowchart showing an example of the flow of attribute information output processing involved in the embodiment.
[0043] Fig.14 This is a flowchart showing an example of the flow of data erasing processing according to the embodiment.
[0044] Fig.15 This is a timing chart showing an example of the processing contents of storing digital image data and compressed image data in a memory when compression processing is performed on digital image data.
[0045] Fig.16A This is a conceptual diagram showing an example of the content of attribute information output from the output I / F of the imaging element for the first to fourth frames when compression processing is performed on digital image data of the fourth frame and thereafter.
[0046] Fig. 16B This is a conceptual diagram showing an example of the content of attribute information output from the output I / F of the imaging element for the fifth and sixth frames when compression processing is performed on digital image data of the fourth and subsequent frames.
[0047] Fig.17A This is a conceptual diagram showing an example of the content of attribute information including a compression reservation flag indicating whether one frame of digital image data is scheduled to be compressed in the next imaging, that is, attribute information output from the output I / F of the imaging element in the first to fourth frames.
[0048] Fig. 17B This is a conceptual diagram showing an example of the content of attribute information including a compression reservation flag indicating whether one frame of digital image data is scheduled to be compressed in the next imaging, that is, attribute information output from the output I / F of the imaging element in the fifth and sixth frames.
[0049] Fig.18AThis is a conceptual diagram showing an example of the content of attribute information including a compression reservation flag indicating whether one frame of digital image data is scheduled to be compressed accompanying the next camera shooting and information on determining the compression format, that is, attribute information output from the output I / F of the imaging element in the 4th frame to the 5th frame.
[0050] Fig.18B This is a conceptual diagram showing an example of the content of attribute information including a compression reservation flag indicating whether it is scheduled to compress one frame of digital image data accompanying the next camera shooting and information on determining the compression format, that is, attribute information output from the output I / F of the imaging element in the sixth frame.
[0051] Fig.19 This is a timing chart showing an example of processing content in the imaging element when the output I / F of the imaging element outputs attribute information at the timing when the receiving I / F of the imaging element receives an instruction (for example, a vertical synchronization signal) related to attribute information.
[0052] Fig. 20 This is a conceptual diagram showing an example of the content of attribute information output in the first to sixth frames when the attribute information output for each frame is attribute information related to one frame of digital image data.
[0053] Fig.21 This is a conceptual diagram showing an example of the content of attribute information output in the first to sixth frames when the attribute information output for each frame is attribute information related to one frame of digital image data, that is, attribute information not including an erase flag and a compression schedule flag.
[0054] Fig. 22 This is a flowchart showing a modified example of the flow of the attribute information output process according to the embodiment.
[0055] Fig.23 This is a block diagram showing a modified example of the hardware configuration of the electrical system of the imaging element included in the imaging device according to the embodiment.
[0056] Fig.24 This is a rear perspective view showing an example of the appearance of the rear side of the smart device according to the embodiment.
[0057] Fig.25 Yes means Fig.24 A front perspective view showing an example of the appearance of the front side of the smart device shown.
[0058] Fig.26 This is a conceptual diagram showing an example of a method of installing the imaging element side program from a storage medium storing the imaging element side program according to the embodiment into a computer in the imaging element. DETAILED DESCRIPTION
[0059] Hereinafter, an example of an embodiment of an imaging device according to the technology of the present invention will be described with reference to the drawings.
[0060] First, the terms used in the following description are explained.
[0061] CPU is short for "Central Processing Unit". GPU is short for "Graphics Processing Unit". RAM is short for "Random Access Memory". ROM is short for "Read Only Memory". DRAM is short for "Dynamic Random Access Memory". SRAM is short for "Static Random Access Memory". LSI is short for "Large-Scale Integrated Circuit". ASIC is short for "Application Specific Integrated Circuit". PLD is short for "Programmable Logic Device". FPGA is short for "Field-Programmable Gate Array". SoC is short for "System-on-a-chip". SSD is short for "Solid State Drive". USB is short for "Universal Serial Bus". HDD is the abbreviation of “Hard Disk Drive”. EEPROM is the abbreviation of “Electrically Erasable and Programmable Read Only Memory”. CCD is the abbreviation of “Charge Coupled Device”. CMOS is the abbreviation of “Complementary Metal Oxide Semiconductor”. EL is the abbreviation of “Electro-Luminescence”. A / D is the abbreviation of “Analog / Digital”. I / F is the abbreviation of “Interface”. UI is the abbreviation of “User Interface”. LVDS is the abbreviation of “Low Voltage Differential Signaling”.PCI-e is the abbreviation of "Peripheral Component Interconnect Express". SATA is the abbreviation of "Serial Advanced Technology Attachment". SLVS-EC is the abbreviation of "Scalable Low Signaling with Embedded Clock". MIPI is the abbreviation of "Mobile Industry Processor Interface". fps is the abbreviation of "frame per second". FIFO is the abbreviation of "First In, First Out". MPEG is the abbreviation of "Moving Picture Experts Group". JPEG is the abbreviation of "Joint Photographic Experts Group". TIFF is the abbreviation of "Tagged Image File Format". BMP is the abbreviation of "Bitmap". PNG is the abbreviation of "Portable Network Graphics". GIF is the abbreviation of "Graphics Interchange Format".
[0062] As an example, Figure 1 As shown, the imaging device 10 is a digital camera with interchangeable lenses and no reflective mirror. The imaging device 10 includes an imaging device body 12 and an interchangeable lens 14 replaceably mounted on the imaging device body 12. In addition, here, as an example of the imaging device 10, a digital camera with interchangeable lenses and no reflective mirror is cited, but the technology of the present invention is not limited to this, and the imaging device 10 may be other types of digital cameras such as a fixed lens type.
[0063] The imaging device body 12 is provided with an imaging element 38. When the interchangeable lens 14 is mounted on the imaging device body 12, subject light representing the subject is transmitted through the interchangeable lens 14 to form an image on the imaging element 38, and the imaging element 38 generates image data representing an image of the subject (for example, refer to Figure 4 and Figure 5 ).
[0064] A hybrid viewfinder (registered trademark) 16 is provided on the camera body 12. The hybrid viewfinder 16 mentioned here refers to a viewfinder that selectively uses an optical viewfinder (hereinafter referred to as "OVF") and an electronic viewfinder (hereinafter referred to as "EVF"), for example. In addition, OVF is an abbreviation of "optical viewfinder". And EVF is an abbreviation of "electronic viewfinder".
[0065] A viewfinder switching lever 18 is provided on the front surface of the camera body 12. By rotating the viewfinder switching lever 18 in the direction of arrow SW, the optical image that can be visually recognized by the OVF and the electronic image that can be visually recognized by the EVF, i.e., the instant preview image, are switched. The "instant preview image" mentioned here refers to a dynamic image for display based on the image data obtained by shooting by the imaging element 38. The instant preview image is also generally referred to as a real-time view image. A release button 20 and a dial 23 are provided on the upper surface of the camera body 12. When setting the operation mode of the camera system and the operation mode of the playback system, etc., the dial 23 is operated, thereby selectively setting the shooting mode and the playback mode as the operation mode in the camera 10.
[0066] The release button 20 functions as a shooting preparation indication unit and a shooting indication unit, and can detect pressing operations in two stages: a shooting preparation indication state and a shooting indication state. The shooting preparation indication state refers to, for example, a state in which the button is pressed from the standby position to an intermediate position (half-pressed position), and the shooting indication state refers to a state in which the button is pressed to a final pressed position (full-pressed position) beyond the intermediate position. In addition, hereinafter, "a state in which the button is pressed from the standby position to a half-pressed position" is referred to as "a half-pressed state", and "a state in which the button is pressed from the standby position to a full-pressed position" is referred to as "a full-pressed state".
[0067] As an example, Figure 2 As shown in FIG. 1 , a touch panel display 24 , instruction keys 27 , and a viewfinder eyepiece portion 30 are provided on the back surface of the imaging device body 12 .
[0068] The touch panel display 24 includes a display 26 and a touch panel 28 (see also Figure 4 As an example of the display 26, a liquid crystal display can be cited. The display 26 may not be a liquid crystal display, but may be another type of display such as an organic EL display or an inorganic EL display. In addition, the display 26 and the EVF are examples of the "display unit (display)" involved in the technology of the present invention. As the display of the EVF is equivalent to the display of the display 26, the subsequent description is omitted, and in this specification, the display of the display 26 can be replaced by the display of the EVF.
[0069] The display 26 displays images and character information, etc. The display 26 is used to display a live preview image obtained by continuous shooting when the camera device 10 is in shooting mode. In addition, the display 26 is also used to display a still image obtained by shooting when an instruction for shooting a still image is given. In addition, the display 26 is also used to display a playback image and a menu screen, etc. when the camera device 10 is in playback mode.
[0070] The touch panel 28 is a transmissive touch panel, and overlaps the surface of the display area of the display 26. The touch panel 28 receives instructions from the user by detecting contact with an instruction body such as a finger or a stylus pen.
[0071] The instruction key 27 receives various instructions. The "various instructions" mentioned here include, for example, instructions for displaying a menu screen that can select various menus, instructions for selecting one or more menus, instructions for confirming the selected content, instructions for erasing the selected content, and various instructions such as zooming in, zooming out, and frame transfer.
[0072] As an example, Figure 3 As shown, the interchangeable lens 14 has an imaging lens 40. The imaging lens 40 includes an objective lens 40A, a focusing lens 40B, and an aperture 40C. The objective lens 40A, the focusing lens 40B, and the aperture 40C are arranged in the order of the objective lens 40A, the focusing lens 40B, and the aperture 40C from the subject side (object side) to the camera body 12 side (image side) along the optical axis L1. The focusing lens 40B and the aperture 40C operate by receiving power from a driving source such as a motor (not shown). That is, the focusing lens 40B and the aperture 40C move along the optical axis L1 according to the applied power. And the aperture 40C adjusts the exposure by operating according to the applied power.
[0073] The camera body 12 includes a post-stage circuit 13, a UI system device 17, a mechanical shutter 41, and an imaging element 38. The post-stage circuit 13 is a circuit located after the imaging element 38. The post-stage circuit 13 includes a controller 15 and a signal processing circuit 34. The controller 15 is connected to the UI system device 17, the signal processing circuit 34, and the imaging element 38, and controls the entire electrical system of the camera 10.
[0074] The imaging element 38 includes a photoelectric conversion element 42 having a light receiving surface 42A. In the present embodiment, the imaging element 38 is a CMOS image sensor. Here, a CMOS image sensor is exemplified as the imaging element 38, but the technology of the present invention is not limited thereto. For example, even if the imaging element 38 is another type of image sensor such as a CCD image sensor, the technology of the present invention is also applicable.
[0075] The mechanical shutter 41 is operated by power from a driving source such as a motor (not shown). When the interchangeable lens 14 is mounted on the imaging device body 12 , subject light representing the subject passes through the imaging lens 40 and is imaged on the light receiving surface 42A via the mechanical shutter 41 .
[0076] The UI system device 17 is a device that presents information to the user or receives instructions from the user. The controller 15 acquires various information from the UI system device 17 and controls the UI system device 17 .
[0077] The imaging element 38 is connected to the controller 15 , and generates image data representing an image of the subject by capturing the subject under the control of the controller 15 .
[0078] The imaging element 38 is connected to the signal processing circuit 34. The signal processing circuit 34 is an LSI, specifically, a device including an ASIC and an FPGA. The controller 15 obtains various information from the signal processing circuit 34 and controls the imaging element 38. The imaging element 38 outputs the image data generated by the photoelectric conversion element 42 to the signal processing circuit 34 under the control of the controller 15.
[0079] The signal processing circuit 34 performs various signal processing on the image data input from the imaging element 38. The various signal processing performed by the signal processing circuit 34 includes, for example, well-known signal processing such as white balance adjustment, sharpness adjustment, gamma correction, color space conversion processing, and color difference correction.
[0080] The various signal processing performed by the signal processing circuit 34 may be distributed between the signal processing circuit 34 and the imaging element 38. That is, the processing circuit 110 of the imaging element 38 may be responsible for at least part of the various signal processing performed by the signal processing circuit 34.
[0081] In addition, in the present embodiment, a device including an ASIC and an FPGA is exemplified as the signal processing circuit 34 , but the technology of the present invention is not limited thereto, and the signal processing circuit 34 may also be a device including an ASIC, an FPGA, and / or a PLD.
[0082] Furthermore, the signal processing circuit 34 may also be a computer including a CPU, storage and memory. The "storage" mentioned here refers to a non-volatile storage device such as an SSD or HDD, and the "memory" mentioned here refers to a volatile storage device such as a DRAM or SRAM. The CPU included in the computer may be single or multiple. Furthermore, a GPU may be used instead of a CPU. Furthermore, the signal processing circuit 34 may be implemented by a combination of a hardware structure and a software structure.
[0083] As an example, Figure 4 As shown in FIG. 1 , the controller 15 includes a CPU 15A, a storage 15B, a memory 15C, an output I / F 15D, and a receiving I / F 15E. The CPU 15A, the storage 15B, the memory 15C, the output I / F 15D, and the receiving I / F 15E are connected via a bus 100. Figure 4 In the example shown, for the sake of convenience, one bus is shown as the bus 100 , but the bus 100 includes a data bus, an address bus, a control bus, and the like.
[0084] The memory 15B stores various parameters and various programs. The memory 15B is a non-volatile storage device. Here, as an example of the memory 15B, an EEPROM is used, but it is not limited to this, and it can also be a mask ROM, HDD or SSD, etc. The memory 15C is a volatile storage device. Various information is temporarily stored in the memory 15C. The memory 15C is used as a working memory by the CPU 15A. Here, as an example of the memory 15C, a DRAM is used, but it is not limited to this, and it can also be other types of volatile storage devices such as SRAM. In addition, the CPU 15A is an example of a "control device" involved in the technology of the present invention, and the memory 15B is an example of a "storage device" involved in the technology of the present invention.
[0085] The memory 15B stores various programs. The CPU 15A reads various programs from the memory 15B and expands the read various programs into the memory 15C. The CPU 15A controls the entire imaging device 10 according to the various programs expanded into the memory 15C.
[0086] The output I / F 15D is connected to the imaging element 38. The CPU 15A controls the imaging element 38 via the output I / F 15D. For example, the CPU 15A controls the imaging timing of the imaging element 38 by supplying an imaging timing signal that specifies the timing of imaging to the imaging element 38 via the output I / F 15D.
[0087] The reception I / F 15E is connected to the signal processing circuit 34. The CPU 15A exchanges various information with the signal processing circuit 34 via the reception I / F 15E.
[0088] Image data is input from the imaging element 38 to the signal processing circuit 34. The signal processing circuit 34 performs various signal processing (details will be described later) on the image data input from the imaging element 38. The signal processing circuit 34 outputs the image data subjected to various signal processing to the receiving I / F 15E. The receiving I / F 15E receives the image data from the signal processing circuit 34 and transmits the received image data to the CPU 15A.
[0089] The external I / F 104 is connected to the bus 100. The external I / F 104 is a communication device composed of a circuit. In addition, here, a device composed of a circuit is used as the external I / F 104, but this is only an example. The external I / F 104 may also be a device including an ASIC, an FPGA and / or a PLD. In addition, the external I / F 104 may be implemented by a combination of a hardware structure and a software structure.
[0090] An example of the external I / F 104 is a USB interface, to which an external device such as a memory card controller, a smart device, a personal computer, a server, a USB memory, and / or a memory card (not shown) can be connected. The external I / F 104 controls the exchange of various information between the CPU 15A and the external device. In addition, an external device directly or indirectly connected to the external I / F 104, that is, an external device such as a smart device, a personal computer, a server, a USB memory, and / or a memory card is an example of a "storage device" involved in the technology of the present invention.
[0091] The UI system device 17 includes a touch panel display 24 and a receiving device 84. The display 26 and the touch panel 28 are connected to the bus 100. Therefore, the CPU 15A causes the display 26 to display various information and operates according to various instructions received from the touch panel 28.
[0092] The receiving device 84 includes a hard key unit 25. The hard key unit 25 is a plurality of hard keys, including a release button 20 (see Figure 1 )、Turntable 23 (reference Figure 1 and Figure 2 ) and indication key 27 (reference Figure 2 ). The hard key unit 25 is connected to the bus 100, and the CPU 15A acquires instructions received by the hard key unit 25 and operates according to the acquired instructions.
[0093] As an example, Figure 5 As shown, the imaging element 38 is input with an imaging timing signal from the controller 15. The imaging timing signal includes a vertical synchronization signal and a horizontal synchronization signal. The vertical synchronization signal is a synchronization signal that specifies the timing for starting the reading of image data of each frame from the photoelectric conversion element 42. The horizontal synchronization signal is a synchronization signal that specifies the timing for starting the reading of image data of each horizontal line from the photoelectric conversion element 42. The imaging element 38 reads out image data from the photoelectric conversion element 42 at a frame rate determined by the vertical synchronization signal input from the controller 15. In addition, the vertical synchronization signal is an example of "an indication related to attribute information" and a "frame synchronization signal from the outside" involved in the technology of the present invention, and the controller 15 is an example of "external" involved in the technology of the present invention.
[0094] exist Figure 5In the example shown, as the frame rate of the imaging element 38, a frame rate of eight frames read out from the photoelectric conversion element 42 within the period T is shown. As an example of a specific frame rate, 120 fps can be cited, but the present invention is not limited thereto, and a frame rate exceeding 120 fps (for example, 240 fps) or a frame rate less than 120 fps (for example, 60 fps) may be used.
[0095] As an example, Figure 6 As shown, the imaging element 38 has a built-in photoelectric conversion element 42, a processing circuit 110, and a memory 112. The imaging element 38 is an imaging element in which the photoelectric conversion element 42, the processing circuit 110, and the memory 112 are integrated into a single chip. That is, the photoelectric conversion element 42, the processing circuit 110, and the memory 112 are packaged together. In the imaging element 38, the processing circuit 110 and the memory 112 are stacked on the photoelectric conversion element 42. Specifically, the photoelectric conversion element 42 and the processing circuit 110 are electrically connected to each other through conductive bumps such as copper (not shown in the figure), and the processing circuit 110 and the memory 112 are also electrically connected to each other through conductive bumps such as copper (not shown in the figure).
[0096] The processing circuit 110 is, for example, an LSI. The memory 112 is a memory in which writing timing and reading timing are different. Here, as an example of the memory 112, a DRAM is used.
[0097] The processing circuit 110 is a device including an ASIC and an FPGA, and controls the entire imaging element 38 according to the instructions of the above-mentioned controller 15. In addition, here, an example is given in which the processing circuit 110 is implemented by a device including an ASIC and an FPGA, but the technology of the present invention is not limited to this. For example, it can also be a device including an ASIC, an FPGA and / or a PLD. In addition, as the processing circuit 110, a computer including a CPU, a storage device such as an EEPROM as a non-volatile storage device, and a memory such as a RAM as a volatile storage device can also be used. The CPU included in the computer can be single or multiple. A GPU is used instead of a CPU. In addition, the processing circuit 110 can be implemented by a combination of a hardware structure and a software structure.
[0098] The photoelectric conversion element 42 has a plurality of photodiodes arranged in a matrix. An example of the plurality of photodiodes is photodiodes for "4896×3265" pixels.
[0099] Color filters are arranged in each photodiode included in the photoelectric conversion element 42. The color filters include a G filter corresponding to G (green), an R filter corresponding to R (red), and a B filter corresponding to B (blue) which are most useful for obtaining a brightness signal.
[0100] The photoelectric conversion element 42 has R pixels, G pixels, and B pixels. R pixels are pixels corresponding to photodiodes equipped with R color filters, G pixels are pixels corresponding to photodiodes equipped with G color filters, and B pixels are pixels corresponding to photodiodes equipped with B color filters. R pixels, G pixels, and B pixels are arranged in a prescribed periodicity in the row direction (horizontal direction) and the column direction (vertical direction), respectively. In the present embodiment, R pixels, G pixels, and B pixels are arranged in a periodicity corresponding to the X-Trans (registered trademark) arrangement. In addition, the X-Trans arrangement is illustrated here, but the technology of the present invention is not limited thereto, and the arrangement of R pixels, G pixels, and B pixels may also be a Bayer arrangement or a honeycomb arrangement, etc.
[0101] The imaging element 38 has a so-called electronic shutter function, and the charge accumulation time of each photodiode in the photoelectric conversion element 42 is controlled by activating the electronic shutter function under the control of the controller 15. The charge accumulation time is a so-called shutter speed.
[0102] In the imaging element 38, still image shooting and live preview image shooting are selectively performed in a rolling shutter method. Still image shooting is achieved by activating the electronic shutter function and operating the mechanical shutter (not shown), and live preview image shooting is achieved by not operating the mechanical shutter but activating the electronic shutter function. In addition, the rolling shutter method is exemplified here, but the technology of the present invention is not limited to this, and a global shutter method can be applied instead of the rolling shutter method.
[0103] In addition, the memory 112 is an example of a "storage unit (memory)" involved in the technology of the present invention. In this embodiment, a DRAM is used as the memory 112, but even if the memory 112 is another type of memory, the technology of the present invention is also established. In addition, the imaging element 38 is an example of a "stacked imaging element" involved in the technology of the present invention.
[0104] As an example, Figure 7 As shown in FIG. 1 , the memory 112 includes a first storage area 112A, a second storage area 112B, a third storage area 112C, a fourth storage area 112D, a fifth storage area 112E, and a sixth storage area 112F. In addition, six storage areas are illustrated here for convenience of description, but this is only an example, and the memory 112 only needs to have a plurality of storage areas capable of storing a plurality of frames of image data.
[0105] exist Figure 7In the example shown, the first storage area 112A, the second storage area 112B, the third storage area 112C, the fourth storage area 112D, the fifth storage area 112E, and the sixth storage area 112F are respectively assigned addresses in the memory 112. The first storage area 112A is assigned an address "0X00000000". The second storage area 112B is assigned an address "0X00100000". The third storage area 112C is assigned an address "0X00200000". The fourth storage area 112D is assigned an address "0X00300000". The fifth storage area 112E is assigned an address "0X00400000". The sixth storage area 112F is assigned an address "0X00500000".
[0106] As an example, Figure 8 As shown, the processing circuit 110 includes a receiving I / F 110D1 and an output I / F 110D2. Here, the receiving I / F 110D1 is an example of a "receiving unit" and an example of a "receiving unit" involved in the technology of the present invention, and the output I / F 110D2 is an example of an "output unit (output interface)" involved in the technology of the present invention. In addition, the receiving unit, for example, represents an acceptor, and the receiving unit, for example, represents a receiver.
[0107] The output I / F 15D of the controller 15 is connected to the receiving I / F 110D1 of the processing circuit 110, and outputs the imaging timing signal to the receiving I / F 110D1. The receiving I / F 110D1 receives the imaging timing signal output from the output I / F 15D.
[0108] The signal processing circuit 34 includes a receiving I / F 34A and an output I / F 34B. The receiving I / F 34A is connected to the output I / F 110D2 of the imaging element 38. The output I / F 110D2 of the processing circuit 110 outputs various information such as image data (hereinafter, also referred to as "various information") to the receiving I / F 34A of the signal processing circuit 34, and the receiving I / F 34A receives the various information output from the output I / F 110D2. The signal processing circuit 34 performs signal processing on the various information received by the receiving I / F 34A as needed. The output I / F 34B is connected to the receiving I / F 15E of the controller 15, and outputs various information to the receiving I / F 15E of the controller 15. The receiving I / F 15E receives the various information output from the output I / F 34B.
[0109] In the imaging element 38, the processing circuit 110 includes a readout circuit 110A, a digital processing circuit 110B, a control circuit 110C, and an image processing circuit 110E in addition to the receiving I / F 110D1 and the output I / F 110D2. The control circuit 110C is an example of a "control unit (control circuit)" involved in the technology of the present invention.
[0110] The readout circuit 110A is connected to the photoelectric conversion element 42, the digital processing circuit 110B, and the control circuit 110C. The digital processing circuit 110B is connected to the control circuit 110C. The control circuit 110C is connected to the memory 112, the reception I / F 110D1, the output I / F 110D2, and the image processing circuit 110E.
[0111] As an example, Figure 8 As shown, the image data is roughly divided into analog image data 70A and digital image data 70B. In the following, for the sake of convenience, when it is not necessary to distinguish between the analog image data 70A and the digital image data 70B, they are referred to as "image data" without any reference numerals.
[0112] The receiving I / F 110D1 and the output I / F 110D2 of the processing circuit 110 are communication devices each having an FPGA. Furthermore, the output I / F 15D and the receiving I / F 15E of the controller 15 are also communication devices each having an FPGA. Furthermore, the receiving I / F 34A and the output I / F 34B of the signal processing circuit 34 are also communication devices each having an FPGA.
[0113] The receiving I / F 110D1 of the processing circuit 110 and the output I / F 15D of the controller 15 are connected according to the connection standard of PCI-e. Furthermore, the output I / F 110D2 of the processing circuit 110 and the receiving I / F 34A of the signal processing circuit 34 are also connected according to the connection standard of PCI-e. Furthermore, the output I / F 34B of the signal processing circuit 34 and the receiving I / F 15E of the controller 15 are also connected according to the connection standard of PCI-e. In addition, in the following, when it is not necessary to distinguish and explain the receiving I / F 110D1, the output I / F 110D2, the receiving I / F 34A, the output I / F 34B, the receiving I / F 15E and the output I / F 15D, they are not marked with symbols and are referred to as "communication I / F".
[0114] Here, a communication device composed of a circuit (ASIC, FPGA and / or PLD, etc.) is used as the communication I / F, but this is only an example. The communication I / F may also be a computer including a CPU, a storage device such as an EEPROM, and a memory device such as a RAM. In this case, the CPU included in the computer may be single or multiple. A GPU is used instead of a CPU. Furthermore, the communication I / F may be implemented by a combination of a hardware structure and a software structure.
[0115] The reception I / F 110D1 receives the imaging timing signal output from the output I / F 15D of the controller 15 , and transmits the received imaging timing signal to the control circuit 110C.
[0116] The readout circuit 110A controls the photoelectric conversion element 42 under the control of the control circuit 110C, and reads out the analog image data 70A from the photoelectric conversion element 42. The analog image data 70A is read out from the photoelectric conversion element 42 in accordance with the imaging timing signal input from the controller 15 to the processing circuit 110.
[0117] Specifically, first, the receiving I / F 110D1 receives the imaging timing signal from the controller 15, and transmits the received imaging timing signal to the control circuit 110C. Next, the control circuit 110C transmits the imaging timing signal transmitted from the receiving I / F 110D1 to the readout circuit 110A. That is, the vertical synchronization signal and the horizontal synchronization signal are transmitted to the readout circuit 110A. And, the readout circuit 110A starts to read out the analog image data 70A in units of frames from the photoelectric conversion element 42 according to the vertical synchronization signal transmitted from the control circuit 110C. And, the readout circuit 110A starts to read out the analog image data 70A in units of horizontal lines according to the horizontal synchronization signal transmitted from the control circuit 110C.
[0118] The readout circuit 110A performs analog signal processing on the analog image data 70A read out from the photoelectric conversion element 42. The analog signal processing includes well-known processing such as noise elimination processing and analog gain processing. The noise elimination processing is a processing for eliminating noise caused by the deviation of characteristics between pixels included in the photoelectric conversion element 42. The analog gain processing is a processing for applying gain to the analog image data 70A. In addition, the readout circuit 110A performs correlated double sampling on the analog image data 70A. After the analog image data 70A is correlated double sampled by the readout circuit 110A, the analog image data 70A is output to the digital processing circuit 110B.
[0119] The digital processing circuit 110B includes an A / D converter 110B1. The A / D converter 110B1 performs A / D conversion on the analog image data 70A.
[0120] The digital processing circuit 110B performs digital signal processing on the analog image data 70A input from the readout circuit 110A. The digital signal processing includes, for example, correlated double sampling, A / D conversion by the A / D converter 110B1, and digital gain processing.
[0121] The analog image data 70A input from the readout circuit 110A is A / D converted by the A / D converter 110B1, whereby the analog image data 70A is digitized and digital image data 70B is obtained as RAW data. Furthermore, the digital image data 70B is subjected to digital gain processing by the digital processing circuit 110B. The digital gain processing is a processing for applying a gain to the digital image data 70B. The digital image data 70B obtained by the digital signal processing is output by the digital processing circuit 110B to the control circuit 110C.
[0122] The control circuit 110C outputs the digital image data 70B input from the digital processing circuit 110B to the image processing circuit 110E. The image processing circuit 110E performs image processing on the digital image data 70B input from the control circuit 110C, and outputs the processed digital image data 70B to the control circuit 110C. The "image processing" mentioned here may include, for example, de-mosaic processing and / or digital thinning processing.
[0123] De-mosaicing is a process of calculating all color information for each pixel based on a mosaic image corresponding to the arrangement of color filters. For example, if the imaging element 38 is an imaging element that uses RGB3 color filters, all RGB color information is calculated for each pixel based on the RGB mosaic image. Digital thinning is a process of thinning out pixels included in image data in line units. Line units refer to, for example, horizontal line units and / or vertical line units.
[0124] The memory 112 is a memory capable of storing digital image data of a plurality of frames. The memory 112 has a storage area in pixel units (see Figure 7 ), the digital image data 70B is stored in a corresponding storage area in the memory 112 in units of pixels by the control circuit 110C. The control circuit 110C stores the digital image data 70B input from the image processing circuit 110E in the memory 112.
[0125] The control circuit 110C can randomly access the memory 112. The control circuit 110C acquires the attribute information of the digital image data 70B stored in the memory 112 according to the instruction from the controller 15. The attribute information is information indicating the attribute of the digital image data 70B stored in the memory 112. Here, the attribute information refers to information including, for example, address, image size, imaging time, exposure time, imaging element sensitivity, and erase flag.
[0126] The address refers to information (recording address) that specifies the storage location of the digital image data 70B in the memory 112 in units of frames. The image size refers to the size of the horizontal line direction (horizontal size) of the digital image data 70B for one frame and the size of the vertical line direction (vertical size) of the digital image data 70B for one frame. The shooting time refers to the time when the imaging element 38 performs shooting (for example, year, month, day, hour, minute, and second). The "time of shooting" mentioned here refers to, for example, the time when the digital image data 70B for one frame is stored in the memory 112. However, this is only an example, and the time of shooting may also be the time when the exposure of one frame is completed or the time when the A / D conversion of one frame is completed.
[0127] The exposure time refers to the exposure time required to obtain one frame of digital image data 70B. The imaging element sensitivity refers to, for example, the sensitivity (sensor gain) of the photoelectric conversion element 42. The erase flag refers to a flag indicating whether the digital image data 70B is erased from the memory 112 with the next image capture. When the erase flag is off, the digital image data 70B is not erased from the memory 112 with the next image capture, and when the erase flag is on, the digital image data 70B is erased from the memory 112 with the next image capture. As an example of the digital image data 70B erased from the memory 112, one frame of digital image data 70B can be cited.
[0128] In addition, the exposure time and the imaging element sensitivity are examples of the “imaging conditions” of the technology of the present invention, and the erase flag in the on state is an example of the “erasure information” involved in the technology of the present invention.
[0129] Here, exposure time and imaging element sensitivity are cited as an example of the “imaging conditions” involved in the technology of the present invention, but the technology of the present invention is not limited to this. Instead of or in addition to the exposure time and / or imaging element sensitivity, the type of imaging lens 40, the subject distance, focal length, viewing angle and / or the presence or absence of shake correction may also be used.
[0130] The control circuit 110C generates attribute information according to the instruction related to attribute information from the controller 15, and outputs the generated attribute information to the output I / F 110D2. The instruction related to attribute information is, for example, an instruction from the controller 15 requesting the processing circuit 110 to output attribute information. In this embodiment, as an example of the instruction related to attribute information, a vertical synchronization signal is used. The output I / F 110D2 outputs the attribute information input from the control circuit 110C to the signal processing circuit 34.
[0131] Specifically, first, the receiving I / F 110D1 receives a vertical synchronization signal as an instruction related to the attribute information from the controller 15. Next, the control circuit 110C generates attribute information related to the digital image data 70B stored earliest among the digital image data 70B stored in the memory 112 as attribute information corresponding to the vertical synchronization signal received by the receiving I / F 110D1. And, the output I / F 110D2 outputs the attribute information generated by the control circuit 110C to the signal processing circuit 34.
[0132] As an example, Fig. 9 As shown, the process performed by the imaging element 38 includes an image pickup process and an output process.
[0133] In the imaging process, exposure, reading of analog image data 70A, resetting of the photoelectric conversion element 42, analog signal processing, digital signal processing, first storage, acquisition of digital image data 70B, image processing, and second storage are performed in sequence.
[0134] The photoelectric conversion element 42 is exposed. The readout circuit 110A reads out the analog image data 70A, resets the photoelectric conversion element 42, and performs analog signal processing. The readout of the analog image data 70A is started on the condition that the vertical synchronization signal is received by the receiving I / F 110D1. The period during which the photoelectric conversion element 42 is exposed is a period during which the analog image data 70A is not read out and the photoelectric conversion element 42 is not reset. Resetting the photoelectric conversion element 42 refers to an operation of eliminating the residual charge of each pixel in the photoelectric conversion element 42. The exposure by the photoelectric conversion element 42 is performed from the last reset of the photoelectric conversion element 42 by the readout circuit 110A to the reading.
[0135] The digital processing circuit 110B performs digital signal processing. The first storage refers to the storage of the digital image data 70B obtained by the digital signal processing in the memory 112. The acquisition of the digital image data 70B refers to the acquisition of the digital image data 70B from the memory 112. The control circuit 110C performs the first storage and the acquisition of the digital image data 70B. The image processing circuit 110E performs image processing on the digital image data 70B acquired by the control circuit 110C. The second storage refers to the storage of the digital image data 70B that has been subjected to image processing in the memory 112. The control circuit 110C performs the second storage.
[0136] In the output process, attribute information is generated and output. The generation of attribute information refers to the generation of attribute information for each frame related to the digital image data 70B stored in the memory 112. The control circuit 110C generates the attribute information. The output of attribute information refers to the output of the attribute information generated by the control circuit 110C using the output I / F 110D2.
[0137] Fig.10 An example of the flow from reading out the image data of the first frame (1F) to the sixth frame (6F) to storing them in the memory 112 and the output timing of the attribute information of each frame are shown. Fig.10 The "F" in 1F to 6F shown in the figure is the abbreviation of "Frame". Fig.10 In the example shown, for ease of explanation, a method of storing one frame of digital image data 70B in the first storage area 112A, the second storage area 112B, the third storage area 112C, the fourth storage area 112D, the fifth storage area 112E and the sixth storage area 112F of the memory 112 is shown.
[0138] As an example, Fig.10 As shown, each time the receiving I / F 110D1 receives a vertical synchronization signal, one frame of analog image data 70A starts to be read out from the photoelectric conversion element 42. Fig.10 In the example shown, the reception I / F 110D1 sequentially receives the first to sixth vertical synchronization signals, and starts reading out the first to sixth frames of analog image data 70A.
[0139] The analog image data 70A is read out for each line from the horizontal line of the first row to the horizontal line of the last row of the photoelectric conversion element 42 according to the horizontal synchronization signal. When the reading of each line is completed, each pixel of the horizontal line that has been completed is reset. The analog image data 70A is converted into digital image data 70B, and the digital image data 70B is stored in the memory 112 in a distinguishable 1-frame unit and FIFO manner according to the order of the frames of the read analog image data 70A (imaging order). Specifically, the latest digital image data 70B is sequentially overwritten and stored in the first storage area 112A, the second storage area 112B, the third storage area 112C, the fourth storage area 112D, the fifth storage area 112E, and the sixth storage area 112F in the unit of 1 frame. The overwriting and saving is repeatedly performed in the order of the first storage area 112A, the second storage area 112B, the third storage area 112C, the fourth storage area 112D, the fifth storage area 112E, and the sixth storage area 112F.
[0140] The output I / F 110D2 outputs the attribute information about the latest digital image data 70B among the digital image data 70B stored in the memory 112 to the signal processing circuit 34. In this case, first, the control circuit 110C generates the attribute information about the latest digital image data 70B among the digital image data 70B stored in the memory 112, every time the storage of the latest digital image data 70B for one frame in the memory 112 is completed at the timing when the vertical synchronization signal is received by the reception I / F 110D1. Then, the output I / F 110D2 outputs the attribute information generated by the control circuit 110C to the signal processing circuit 34.
[0141] As an example, Fig.11A and Fig. 11B As shown, each frame (in Fig.11A and Fig. 11B In the example shown, the attribute information of the first frame to the sixth frame is output from the output I / F 110D2 to the signal processing circuit 34. Then, the signal processing circuit 34 outputs the input attribute information to the controller 15.
[0142] As an example, Fig.11A As shown in FIG. 1 , the attribute information output in the first frame is the attribute information related to the digital image data 70B stored in the first storage area 112A. Fig.11A In the example shown, as an example of the attribute information related to the digital image data 70B stored in the first storage area 112A, there can be cited information including the address of the first storage area 112A, the horizontal size of the digital image data 70B in the first storage area 112A, the vertical size of the digital image data 70B in the first storage area 112A, the image capturing time related to the digital image data 70B in the first storage area 112A, the exposure time related to the digital image data 70B in the first storage area 112A, the imaging element sensitivity related to the digital image data 70B in the first storage area 112A, and the erase flag_off. Here, the “erase flag_off” refers to information indicating that the erase flag is in the off state.
[0143] The attribute information output in the second frame is attribute information related to the digital image data 70B stored in the first storage area 112A and attribute information related to the digital image data 70B stored in the second storage area 112B. Fig.11AIn the example shown, as an example of attribute information related to the digital image data 70B stored in the second storage area 112B, there can be cited information including the address of the second storage area 112B, the horizontal size of the digital image data 70B in the second storage area 112B, the vertical size of the digital image data 70B in the second storage area 112B, the shooting time related to the digital image data 70B in the second storage area 112B, the exposure time related to the digital image data 70B in the second storage area 112B, the imaging element sensitivity related to the digital image data 70B in the second storage area 112B, and the erase flag_off information.
[0144] The attribute information output in the third frame is attribute information related to the digital image data 70B stored in the first storage area 112A, attribute information related to the digital image data 70B stored in the second storage area 112B, and attribute information related to the digital image data 70B stored in the third storage area 112C. Fig.11A In the example shown, as an example of attribute information related to the digital image data 70B stored in the third storage area 112C, there can be cited information including the address of the third storage area 112C, the horizontal size of the digital image data 70B in the third storage area 112C, the vertical size of the digital image data 70B in the third storage area 112C, the shooting time related to the digital image data 70B in the third storage area 112C, the exposure time related to the digital image data 70B in the third storage area 112C, the imaging element sensitivity related to the digital image data 70B in the third storage area 112C, and the erase flag_off information.
[0145] The attribute information output in the fourth frame is attribute information related to the digital image data 70B stored in the first storage area 112A, attribute information related to the digital image data 70B stored in the second storage area 112B, attribute information related to the digital image data 70B stored in the third storage area 112C, and attribute information related to the digital image data 70B stored in the fourth storage area 112D. Fig.11AIn the example shown, as an example of the attribute information related to the digital image data 70B stored in the fourth storage area 112D, there can be cited information including the address of the fourth storage area 112D, the horizontal size of the digital image data 70B in the fourth storage area 112D, the vertical size of the digital image data 70B in the fourth storage area 112D, the shooting time related to the digital image data 70B in the fourth storage area 112D, the exposure time related to the digital image data 70B in the fourth storage area 112D, the sensitivity of the imaging element related to the digital image data 70B in the fourth storage area 112D, and the information of the erase flag_off. Here, the "erase flag_off" in the attribute information related to the digital image data 70B stored in the first storage area 112A included in the attribute information output in the fourth frame is changed to "erase flag_on" by the control circuit 110C. "Erase flag_on" refers to information indicating that the erase flag is in the on state.
[0146] Thus, the "erase flag_off" in the attribute information related to the digital image data 70B stored in the first storage area 112A is changed to "erase flag_on" by the control circuit 110C. This indicates that the attribute information output from the output I / F 110D2 in the fourth frame includes information indicating that the digital image data 70B in the first storage area 112A is erased from the memory 112 with the next imaging.
[0147] When the attribute information outputted in the fourth frame includes "erase flag_on", the attribute information related to the erasing target image data, that is, the attribute information related to the first storage area 112A, which is the digital image data 70B to be erased in the memory 112, is erased by the control circuit 110C with the next imaging (for example, at the time when the fifth frame imaging starts or during the period from the current time to the start of the fifth frame imaging). Furthermore, the digital image data 70B is also erased from the first storage area 112A by the control circuit 110C. In this case, for example, with the next imaging, the attribute information related to the first storage area 112A is first erased by the control circuit 110C, and then the digital image data 70B is erased from the first storage area 112A by the control circuit 110C. In addition, an example of erasing the digital image data 70B and attribute information is given here, but the technology of the present invention is not limited to this. In fact, "erasing" not only erases the digital image data 70B and attribute information, but also includes releasing the area and overwriting it.
[0148] As an example, Fig. 11BAs shown, the attribute information output in the fifth frame is the attribute information related to the digital image data 70B stored in the second storage area 112B, the attribute information related to the digital image data 70B stored in the third storage area 112C, the attribute information related to the digital image data 70B stored in the fourth storage area 112D, and the attribute information related to the digital image data 70B stored in the fifth storage area 112E. Fig. 11B In the example shown, as an example of the attribute information related to the digital image data 70B stored in the fifth storage area 112E, there can be cited information including the address of the fifth storage area 112E, the horizontal size of the digital image data 70B in the fifth storage area 112E, the vertical size of the digital image data 70B in the fifth storage area 112E, the image capturing time related to the digital image data 70B in the fifth storage area 112E, the exposure time related to the digital image data 70B in the fifth storage area 112E, the imaging element sensitivity related to the digital image data 70B in the fifth storage area 112E, and the erase flag_off. Here, the "erase flag_off" in the attribute information related to the digital image data 70B stored in the second storage area 112B included in the attribute information output in the fifth frame is changed to "erase flag_on" by the control circuit 110C.
[0149] Thus, when the "erase flag_off" in the attribute information related to the digital image data 70B stored in the second storage area 112B is changed to "erase flag_on" by the control circuit 110C, the attribute information related to the erasure target image data, that is, the attribute information related to the second storage area 112B, which is the digital image data 70B to be erased in the memory 112, is erased by the control circuit 110C with the next imaging (for example, at the time of starting the imaging of the sixth frame or during the period from the current time to the start of the imaging of the sixth frame). Furthermore, the digital image data 70B is also erased from the second storage area 112B by the control circuit 110C. In this case, for example, with the next imaging, the attribute information related to the second storage area 112B is first erased by the control circuit 110C, and then the digital image data 70B is erased from the second storage area 112B by the control circuit 110C.
[0150] The attribute information output in the sixth frame is attribute information related to the digital image data 70B stored in the third storage area 112C, attribute information related to the digital image data 70B stored in the fourth storage area 112D, attribute information related to the digital image data 70B stored in the fifth storage area 112E, and attribute information related to the digital image data 70B stored in the sixth storage area 112F. Fig. 11BIn the example shown, as an example of the attribute information related to the digital image data 70B stored in the sixth storage area 112F, there can be cited information including the address of the sixth storage area 112F, the horizontal size of the digital image data 70B in the sixth storage area 112F, the vertical size of the digital image data 70B in the sixth storage area 112F, the image capturing time related to the digital image data 70B in the sixth storage area 112F, the exposure time related to the digital image data 70B in the sixth storage area 112F, the imaging element sensitivity related to the digital image data 70B in the sixth storage area 112F, and the erase flag_off. Here, the "erase flag_off" in the attribute information related to the digital image data 70B stored in the third storage area 112C included in the attribute information output in the sixth frame is changed to "erase flag_on" by the control circuit 110C.
[0151] Thus, when the "erase flag_off" in the attribute information related to the digital image data 70B stored in the third storage area 112C is changed to "erase flag_on" by the control circuit 110C, the attribute information related to the erasure target image data, that is, the attribute information related to the third storage area 112C, which is the digital image data 70B to be erased in the memory 112, is erased by the control circuit 110C with the next imaging (for example, at the time of starting the imaging of the seventh frame or during the period from the current time to the start of the imaging of the seventh frame). Furthermore, the digital image data 70B is also erased from the third storage area 112C by the control circuit 110C. In this case, for example, with the next imaging, the attribute information related to the third storage area 112C is first erased by the control circuit 110C, and then the digital image data 70B is erased from the third storage area 112C by the control circuit 110C.
[0152] In this way, when the attribute information is sent from the output I / F 110D2 to the signal processing circuit 34 for each frame, the signal processing circuit 34 transmits the attribute information to the controller 15. The attribute information transmitted from the signal processing circuit 34 to the controller 15 is received by the receiving I / F 15E of the controller 15. The controller 15 obtains an address from the attribute information received by the receiving I / F 15E as needed.
[0153] As an example, Fig.12As shown, the controller 15 outputs the address acquired from the attribute information to the receiving I / F 110D1 of the imaging element 38 via the output I / F 15D. The receiving I / F 110D1 of the imaging element 38 receives the address from the controller 15. The control circuit 110C acquires the digital image data 70B from the memory 112 according to the address received by the receiving I / F 110D1, and causes the output I / F 110D2 to output the acquired digital image data 70B to the signal processing circuit 34.
[0154] In addition, although the example of outputting the address from the output I / F 15D of the controller 15 to the imaging element 38 is described here, the technology of the present invention is not limited to this. For example, the controller 15 may also have a transmission I / F, and the address may be sent to the imaging element 38 from the transmission I / F.
[0155] Here, the receiving I / F 110D1 receives the address, but the technology of the present invention is not limited to this. For example, the imaging element 38 may also include a receiving I / F, and the receiving I / F of the imaging element 38 may receive the address sent from the controller 15.
[0156] Next, the operation of the imaging device 10 will be described.
[0157] First, refer to Fig.13 The flow of the attribute information output process executed by the processing circuit 110 of the imaging element 38 will be described.
[0158] exist Fig.13 In the attribute information output process shown, first, in step ST10, the control circuit 110C determines whether the vertical synchronization signal is received by the receiving I / F 110D1. In step ST10, if the receiving I / F 110D1 does not receive the vertical synchronization signal, the determination is "No", and the attribute information output process moves to step ST22. In step ST10, if the receiving I / F 110D1 receives the vertical synchronization signal, the determination is "Yes", and the attribute information output process moves to step ST12.
[0159] In step ST12 , the control circuit 110C stores the digital image data 70B obtained by photographing in the memory 112 , and then the attribute information output process proceeds to step ST14 .
[0160] In step ST14 , the control circuit 110C generates attribute information about the digital image data 70B stored in the memory 112 in step ST12 , holds the generated attribute information, and then the attribute information output process shifts to step ST16 .
[0161] In step ST16, the control circuit 110C determines whether the number of frames of the digital image data 70B that can be stored in the memory 112 by performing the next imaging has reached the upper limit. In the present embodiment, the memory 112 has six storage areas, namely, the first storage area 112A, the second storage area 112B, the third storage area 112C, the fourth storage area 112D, the fifth storage area 112E, and the sixth storage area 112F, so the "upper limit" mentioned here is "6". In addition, the upper limit is not limited to this, as long as it is the number of frames that can be stored in the memory 112.
[0162] In step ST16, if the number of frames of the digital image data 70B that can be stored in the memory 112 has not reached the upper limit by performing the next image capture, the determination is "No", and the attribute information output process moves to step ST20. In step ST16, if the number of frames of the digital image data 70B that can be stored in the memory 112 has reached the upper limit by performing the next image capture, the determination is "Yes", and the attribute information output process moves to step ST18.
[0163] In step ST18, the control circuit 110C turns on the erase flag contained in the attribute information of the earliest 1 frame of digital image data 70B stored in the memory 112 among all frames of digital image data 70B stored in the memory 112, and then the attribute information output processing is transferred to step ST20.
[0164] In step ST20 , the control circuit 110C causes the output I / F 110D2 to output the attribute information currently held to the signal processing circuit 34 , and then the attribute information output process shifts to step ST22 .
[0165] In step ST22, the control circuit 110C determines whether a condition for terminating the attribute information output process (hereinafter referred to as "attribute information output process termination condition") is satisfied. An example of the attribute information output process termination condition is a condition for terminating the attribute information output process. Figure 4 ) receives the condition of the instruction to end the attribute information output processing. In step ST22, if the attribute information output processing end condition is not satisfied, it is determined as "no", and the attribute information output processing is transferred to step ST10. In step ST22, if the attribute information output processing end condition is satisfied, it is determined as "yes", and the attribute information output processing is ended.
[0166] Next, refer to Fig.14 The flow of the data erasing process executed by the processing circuit 110 of the imaging element 38 will be described.
[0167] exist Fig.14In the data erasing process shown, first, in step ST50, the control circuit 110C determines whether the attribute information is output by the output I / F 110D2. In step ST50, if the output I / F 110D2 does not output the attribute information, the determination is "No", and the data erasing process moves to step ST60. In step ST50, if the output I / F 110D2 outputs the attribute information, the determination is "Yes", and the data erasing process moves to step ST52.
[0168] In step ST52, the control circuit 110C determines whether the vertical synchronization signal is received by the receiving I / F 110D1. In step ST52, if the receiving I / F 110D1 does not receive the vertical synchronization signal, the determination is "No", and the determination of step ST52 is performed again. In step ST52, if the receiving I / F 110D1 receives the vertical synchronization signal, the determination is "Yes", and the data erasing process is transferred to step ST54.
[0169] In step ST54, the control circuit 110C determines whether the attribute information of the erase flag being turned on is maintained. In step ST54, if the attribute information of the erase flag being turned on is not maintained, the determination is "No", and the data erase process is transferred to step ST60. In step ST54, if the attribute information of the erase flag being turned on is maintained, the determination is "Yes", and the data erase process is transferred to step ST56.
[0170] In step ST56, the control circuit 110C first specifies the digital image data 70B based on the address included in the attribute information whose erase flag is turned on among all the attribute information currently held. Next, the control circuit 110C erases the attribute information whose erase flag is turned on among all the attribute information currently held. Furthermore, the control circuit 110C erases the digital image data 70B specified by the address from the memory 112, and then the data erase process is transferred to step ST60.
[0171] In step ST60, the control circuit 110C determines whether a condition for terminating the data erasing process (hereinafter referred to as "data erasing process terminating condition") is satisfied. An example of the data erasing process terminating condition is a condition that the receiving device 84 (see Figure 4 ) receives the condition of the instruction to end the data erasing process. In step ST60, if the data erasing process end condition is not satisfied, it is determined as "no", and the data erasing process is transferred to step ST50. In step ST60, if the data erasing process end condition is satisfied, it is determined as "yes", and the data erasing process is terminated.
[0172] As described above, in the imaging device 10, the receiving I / F 110D1 receives the instruction related to the attribute information, and the output I / F 110D2 outputs the attribute information corresponding to the received instruction to the subsequent circuit 13. Thus, the subsequent circuit 13 can request the imaging element 38 to provide the digital image data 70B by using the attribute information. Therefore, the subsequent circuit 13 can selectively obtain the required digital image data 70B from the imaging element 38. In addition, compared with the case where both the digital image data 70B and at least a part of the attribute information are output at the same time, the power consumption required for the output of the imaging element 38 can be reduced.
[0173] Furthermore, in the imaging device 10, the control circuit 110C generates attribute information corresponding to the vertical synchronization signal from the controller 15, and the output I / F 110D2 outputs the generated attribute information to the signal processing circuit 34. Therefore, the subsequent circuit 13 can obtain the attribute information from the imaging element 38 without newly generating a signal different from the vertical synchronization signal for use.
[0174] In the imaging device 10, the output I / F 110D2 outputs attribute information about the latest digital image data 70B among all the digital image data 70B stored in the memory 112. Therefore, the subsequent circuit 13 can acquire attribute information about the latest digital image data 70B.
[0175] Furthermore, in the image pickup device 10, the output I / F 110D2 outputs the attribute information about the latest digital image data 70B at the timing when the vertical synchronization signal is received by the receiving I / F 110D1. Therefore, the subsequent circuit 13 can obtain the attribute information about the latest digital image data 70B at the timing when the vertical synchronization signal is received by the receiving I / F 110D1.
[0176] In the imaging device 10, the output I / F 110D2 can output attribute information of each of the digital image data 70B of a plurality of frames, and the attribute information is outputted by the output I / F 110D2 in the imaging sequence. Therefore, the subsequent circuit 13 can acquire the digital image data 70B from the imaging element 38 in the imaging sequence.
[0177] Furthermore, in the imaging device 10, information including address, image size, imaging time, and imaging conditions is used as attribute information. Therefore, compared with a case where the attribute information does not include any of address, image size, imaging time, and imaging conditions, the range of uses of the digital image data 70B acquired by the subsequent circuit 13 from the imaging element 38 can be expanded.
[0178] Furthermore, in the imaging device 10, when the digital image data 70B is erased from the memory 112 with the next imaging, the attribute information outputted from the output I / F 110D2 includes an erase flag with the flag turned on. Therefore, it is possible to prevent the subsequent circuit 13 from instructing the output of the digital image data 70B not stored in the memory 112.
[0179] Furthermore, in the imaging device 10, when the digital image data 70B is erased from the memory 112 in conjunction with the next imaging, the attribute information whose erase flag is turned on is erased from the memory 112 by the control circuit 110C, and the digital image data 70B specified by the address included in the attribute information whose erase flag is turned on is erased from the memory 112 by the control circuit 110C. Therefore, both the attribute information and the digital image data 70B related to the attribute information can be erased at different timings.
[0180] Furthermore, in the image pickup device 10, the receiving I / F 110D1 receives the attribute information transmitted from the controller 15. In the control circuit 110C, the digital image data 70B is acquired from the memory 112 according to the attribute information received by the receiving I / F 110D1, and the acquired digital image data 70B is output to the signal processing circuit 34 by the output I / F 110D2. Therefore, the imaging element 38 can output the digital image data 70B requested by the subsequent circuit 13 to the subsequent circuit 13.
[0181] Furthermore, in the imaging device 10, an imaging device in which the photoelectric conversion element 42, the processing circuit 110, and the memory 112 are integrated into a single chip is used as the imaging device 38. As a result, the portability of the imaging device 38 is improved compared to an imaging device in which the photoelectric conversion element 42, the processing circuit 110, and the memory 112 are not integrated into a single chip. Furthermore, the degree of freedom in design can be improved compared to an imaging device in which the photoelectric conversion element 42, the processing circuit 110, and the memory 112 are not integrated into a single chip. In addition, compared to an imaging device in which the photoelectric conversion element 42, the processing circuit 110, and the memory 112 are not integrated into a single chip, it can also contribute to the miniaturization of the imaging device 10.
[0182] And, if Figure 6As shown in FIG. 1 , as the imaging element 38, a stacked imaging element in which a memory 112 is stacked on a photoelectric conversion element 42 is used. As a result, compared with a case where the photoelectric conversion element 42 and the memory 112 are not stacked, the transmission speed of image data from the photoelectric conversion element 42 to the memory 112 can be increased. The increase in the transmission speed also contributes to the speed-up of the processing in the entire processing circuit 110. In addition, compared with a case where the photoelectric conversion element 42 and the memory 112 are not stacked, the degree of freedom in design can also be increased. In addition, compared with a case where the photoelectric conversion element 42 and the memory 112 are not stacked, it can also contribute to the miniaturization of the imaging device 10.
[0183] In the imaging device 10, the CPU 15A displays a live preview image based on the digital image data 70B outputted from the output I / F 110D2 on the display 26. This allows the user to visually recognize the live preview image based on the digital image data 70B outputted from the output I / F 110D2.
[0184] In the imaging device 10, the digital image data 70B outputted from the output I / F 110D2 is stored by the CPU 15A in the storage 15B, a smart device, a personal computer, a server, a USB memory, and / or a memory card, etc. Thus, the digital image data 70B outputted from the output I / F 110D2 can be managed.
[0185] In addition, in the above-mentioned embodiment, an example of a method in which multiple frames of digital image data 70B are stored in the memory 112 is given for explanation, but the technology of the present invention is not limited to this. For example, in addition to multiple frames of digital image data 70B, the memory 112 can also store compressed image data. The compressed image data can be obtained by performing a compression process by the image processing circuit 110E. The compression process refers to the process of compressing the digital image data 70B. That is, in addition to multiple frames of digital image data 70B, the memory 112 can also store compressed image data obtained by compressing the digital image data 70B by the image processing circuit 110E, which is an example of a "compression circuit" involved in the technology of the present invention.
[0186] Specifically, the control circuit 110C sets the specified number of frames (in Fig.15 In the example shown, digital image data 70B (4 frames) is stored in the memory 112. Furthermore, the control circuit 110C stores compressed image data obtained by compressing the digital image data by the image processing circuit 110E in the memory 112. In addition, the control circuit 110C associates attribute information related to the compressed image data and compression determination information capable of determining that the image data is compressed with the compressed image data.
[0187] In this case, as an example, Fig.15 As shown, the memory 112 has addresses "0X00540000", "0X00580000", and "0X005C0000" in addition to the addresses described in the above embodiment, and has a storage area for one frame for each address. Fig.15 In the example shown, when the fourth frame digital image data 70B is stored in the fourth storage area 112D (the storage area at the address "0X00300000"), the fourth frame digital image data 70B is compressed by the image processing circuit 110E. The control circuit 110C compresses the fourth frame digital image data 70B to obtain the compressed image data ( Fig.15 The "compressed data" in the example shown) is stored in the fifth storage area 112E (the storage area at the address "0X00400000").
[0188] And, in Fig.15 In the example shown, with the completion of the fourth frame of imaging, the control circuit 110C erases the first frame of digital image data 70B from the first storage area 112A (the storage area at the address "0X00000000"), and the control circuit 110C stores the fifth frame of digital image data 70B in the first storage area 112A. Furthermore, when the fifth frame of digital image data 70B is stored in the first storage area 112A, the image processing circuit 110E compresses the fifth frame of digital image data 70B. The control circuit 110C stores the compressed image data obtained by compressing the fifth frame of digital image data 70B in the sixth storage area 112F (the storage area at the address "0X00500000").
[0189] In addition, Fig.15 In the example shown, with the completion of the fifth frame of imaging, the control circuit 110C erases the second frame of digital image data 70B from the second storage area 112B (the storage area at the address "0X00100000"), and the control circuit 110C stores the sixth frame of digital image data 70B in the second storage area 112B. Furthermore, when the sixth frame of digital image data 70B is stored in the second storage area 112B, the image processing circuit 110E compresses the sixth frame of digital image data 70B. The control circuit 110C stores the compressed image data obtained by compressing the sixth frame of digital image data 70B in the seventh storage area 112G (the storage area at the address "0X00540000").
[0190] In this way, the digital image data 70B of the fourth frame and later are compressed by the image processing circuit 110E, and the compressed image data obtained by the compression is stored in the free storage area of the memory 112 by the control circuit 110C.
[0191] Furthermore, when the digital image data 70B after the fourth frame is compressed and stored in the memory 112 as compressed image data, as an example, Fig.16A and Fig. 16B As shown, the attribute information is generated by the control circuit 110C.
[0192] As an example, Fig.16A As shown, the attribute information output in the 1st to 3rd frames is Fig.11A The attribute information outputted in the 4th frame includes attribute information related to the 4th frame digital image data 70B. Fig.16A In the example shown, as attribute information related to the 4th frame digital image data 70B, the address "0X00300000", image size, shooting time, shooting conditions and erase flag_off that can determine the 4th storage area 112D are shown. In addition, the erase flag of the 1st frame attribute information included in the attribute information output in the 4th frame is turned on. As a result, the 1st frame digital image data 70B is erased from the 1st storage area 112A as the 5th frame is shot. In addition, the attribute information output in the 4th frame also includes attribute information related to the compressed image data of the 5th frame digital image data 70B. Fig.16A In the example shown, as attribute information related to the compressed image data of the fifth frame digital image data 70B, the address "0X00400000" that can identify the fifth storage area 112E, the image size, the shooting time, the shooting conditions and the compression determination information are shown.
[0193] As an example, Fig. 16B As shown, the attribute information outputted in the fifth frame includes attribute information related to the fifth frame digital image data 70B. Fig. 16B In the example shown, as attribute information related to the fifth frame digital image data 70B, the address "0X00000000" that can determine the first storage area 112A, the image size, the shooting time, the shooting conditions, and the erase flag_off are shown. Furthermore, the erase flag of the second frame attribute information included in the attribute information output in the fifth frame is turned on. As a result, the second frame digital image data 70B is erased from the second storage area 112B as the sixth frame is shot. Furthermore, the attribute information output in the fifth frame also includes attribute information related to the compressed image data of the fifth frame digital image data 70B and attribute information related to the compressed image data of the sixth frame digital image data 70B. Fig. 16BIn the example shown, as attribute information related to the compressed image data of the sixth frame digital image data 70B, the address "0X00500000" that can identify the sixth storage area 112F, the image size, the shooting time, the shooting conditions and the compression determination information are shown.
[0194] As an example, Fig. 16B As shown, the attribute information outputted in the sixth frame includes attribute information related to the sixth frame digital image data 70B. Fig. 16B In the example shown, as attribute information related to the 6th frame digital image data 70B, the address "0X01000000", image size, shooting time, shooting conditions, and erase flag_off that can determine the 2nd storage area 112B are shown. Furthermore, the erase flag of the 3rd frame attribute information included in the attribute information output in the 6th frame is turned on. As a result, the 3rd frame digital image data 70B is erased from the 3rd storage area 112C as the 7th frame is shot. Furthermore, the attribute information output in the 5th frame also includes attribute information related to the compressed image data of the 5th frame digital image data 70B, attribute information related to the compressed image data of the 6th frame digital image data 70B, and attribute information related to the compressed image data of the 7th frame digital image data 70B. Fig. 16B In the example shown, as attribute information related to the compressed image data of the seventh frame digital image data 70B, an address of "0X00540000", an image size, an imaging time, an imaging condition, and compression determination information are shown.
[0195] In this way, the control circuit 110C stores the digital image data 70B of the specified number of frames in the memory 112, the control circuit 110C also stores the compressed image data in the memory 112, and the control circuit 110C establishes a corresponding association between the attribute information related to the compressed image data and the compression determination information (in Fig.16A and Fig. 16B In the example shown, the attribute information includes compression identification information. Therefore, even when the digital image data 70B and the compressed image data are stored in the memory 112, the digital image data 70B and the compressed image data can be distinguishably identified.
[0196] exist Fig.16A and Fig. 16B In the example shown, the attribute information related to the digital image data 70B includes an address, an image size, a shooting time, a shooting condition, and an erase flag, but the technology of the present invention is not limited to this. For example, when the compressed image data is stored in the memory 112, as an example, Fig.17A and Fig. 17BAs shown, the attribute information related to the digital image data 70B may include, in addition to the address, image size, shooting time, shooting conditions, and erase flag, a compression reservation flag indicating whether the compressed image data is scheduled to be stored in the memory 112. The compression reservation flag that is turned on is an example of the "compression reservation information" involved in the technology of the present invention, and indicates that the compressed image data is scheduled to be stored in the memory 112 with the next shooting. The compression reservation flag that is turned off indicates that the compressed image data is not scheduled to be stored in the memory 112 with the next shooting.
[0197] When the compression reservation flag is turned on, the digital image data 70B related to the attribute information for which the compression reservation flag is turned on is compressed by the image processing circuit 110E in the next image capture. Fig.17A As shown, the attribute information related to the first frame digital image data 70B included in the attribute information output in the third frame includes "compression reservation flag_on" indicating that the compression reservation flag is turned on. In this case, the first frame digital image data 70B is compressed by the image processing circuit 110E with the next image capture (the fourth frame image capture), and the compressed image data obtained by compression is stored in the fifth storage area 112E. In conjunction with this, the attribute information output in the fourth frame includes attribute information related to the compressed image data stored in the fifth storage area 112E (the storage area at the address "0X00400000"). The attribute information related to the compressed image data stored in the fifth storage area 112E includes compression determination information, and thus the subsequent circuit 13 can determine that the compressed image data is stored in the fifth storage area 112E.
[0198] In addition, the timing synchronization of the erasure flag and the compression reservation flag being turned on, as an example, Fig.17A and Fig. 17B As shown, when the control circuit 110C turns on the erase flag, the control circuit 110C also turns on the compression flag. Therefore, even if the digital image data 70B is erased from the memory 112, the compressed image data obtained by compressing the digital image data 70B before erasure is retained in the memory 112.
[0199] Thus, when the compressed image data is stored in the memory 112, the attribute information output from the output I / F 110D2 includes the turned-on compression reservation flag, so that the output destination of the compressed image data (for example, the subsequent circuit 13) can be given a preparation margin for receiving the compressed image data.
[0200] In addition, as a compression format when the digital image data 70B is compressed into compressed image data, for example, MPEG, JPEG, TIFF, PNG, GIF, and BMP can be cited. When the compressed image data is stored in the memory 112, as an example, Fig.18A and Fig.18B As shown in FIG. 1 , the attribute information output from the output I / F 110D2 may include compression format identification information capable of identifying the compression format of the compressed image data. Fig.18A and Fig.18B In the example shown, the compression format specifying information is included in the attribute information together with the compression specifying information. This allows the output destination of the compressed image data (for example, the subsequent circuit 13) to perform processing according to the compression format.
[0201] Furthermore, in the above-mentioned embodiment, an example is given in which the digital image data 70B of one frame is stored in the memory 112 and then the attribute information is outputted from the output I / F 110D2, but the technology of the present invention is not limited thereto. Fig.19 As shown, the output I / F 110D2 can output the attribute information at the timing when the receiving I / F 110D1 receives an instruction (eg, a vertical synchronization signal) related to the attribute information. Thus, the attribute information can be provided to the subsequent circuit 13 immediately according to the instruction related to the attribute information.
[0202] In the above-mentioned embodiment, an example is given in which the attribute information outputted for each frame includes the attribute information of a plurality of frames, but the technology of the present invention is not limited thereto. Fig. 20 As shown, the attribute information outputted for each frame may be only the attribute information related to the digital image data 70B of the corresponding frame. In this case, the amount of attribute information outputted for each frame to the signal processing circuit 34 can be reduced compared to the case where the attribute information outputted for each frame includes attribute information for a plurality of frames.
[0203] In addition, Fig. 20 In the example shown, the attribute information outputted for each frame includes an erase flag and a compression reservation flag, but the technology of the present invention is not limited to this. Fig.21 As shown, the attribute information output for each frame may not include the erase flag and the compression schedule flag. Alternatively, the attribute information output for each frame may include the compression schedule flag instead of the erase flag.
[0204] Furthermore, in the above-mentioned embodiment, an example of a method in which the output I / F110D2 outputs attribute information related to each of all frames is given for explanation, but the technology of the present invention is not limited to this. The output I / F110D2 may be instructed from the subsequent circuit 13 to the imaging element 38 to output attribute information for a certain number of frames.
[0205] In this case, for example, instead of Fig.13 The attribute information output process shown is executed by the processing circuit 110 of the imaging element 38 as an example. Fig. 22 The attribute information output processing shown.
[0206] Fig. 22 The attribute information output processing shown is the same as Fig.13 Compared with the attribute information output process shown in FIG. 1 , the difference is that step ST10A is provided in place of step ST10, step ST18A is provided in place of step ST18, and step ST20A is provided in place of step ST20. The controller 15 outputs an attribute information request instruction to the imaging element 38 via the output I / F 15D. The attribute information request instruction includes a vertical synchronization signal and output amount information. The output amount information refers to information indicating the output amount of the attribute information using the output I / F 110D2 of the imaging element 38. The "output amount" mentioned here is defined by, for example, the number of frames.
[0207] exist Fig. 22 In the attribute information output process shown, in step ST10A, the control circuit 110C determines whether the attribute information request instruction output from the controller 15 is received by the receiving I / F 110D1. In step ST10A, if the receiving I / F 110D1 does not receive the attribute information request instruction, the determination is "No", and the attribute information output process moves to step ST22. In step ST10A, if the receiving I / F 110D1 receives the attribute information request instruction, the determination is "Yes", and the attribute information output process moves to step ST12.
[0208] In step ST18A, the control circuit 110C selects attribute information to be outputted from the output I / F 110D2 to the signal processing circuit 34 from the attribute information currently held, according to the output amount indicated by the output amount information included in the attribute information request instruction. For example, if the output amount indicated by the output amount information is 1 frame, the control circuit 110C selects attribute information related to the latest digital image data 70B for 1 frame, and if the output amount indicated by the output amount information is 2 frames, the control circuit 110C selects attribute information related to the latest digital image data 70B for 2 frames.
[0209] In step ST20A, the control circuit 110C causes the output I / F 110D2 to output the attribute information selected in step ST18A to the signal processing circuit 34. This can prevent the output I / F 110D2 from outputting more attribute information than necessary. Furthermore, since the output amount is defined by the number of frames, the subsequent circuit 13 can process the attribute information in units of frames.
[0210] Furthermore, in the above-mentioned embodiment, an example is given in which the output I / F 110D2 outputs the attribute information and the digital image data 70B to the signal processing circuit 34, but the technology of the present invention is not limited to this. Fig.23 As shown, the attribute information and the digital image data 70B can be output to the signal processing circuit 34 through the output I / F 110D2 and the output I / F 110D3. Fig.23 In the example shown, the processing circuit 110 includes an output I / F 110D2 and an output I / F 110D3. The output I / F 110D3 is an example of a "first output unit" according to the technology of the present invention, and the output I / F 110D2 is an example of a "second output unit" according to the technology of the present invention.
[0211] The output I / F 110D3 is connected to the control circuit 110C. In addition, the signal processing circuit 34 includes a receiving I / F 34C. The output I / F 110D2 outputs the attribute information to the signal processing circuit 34. The receiving I / F 34A of the signal processing circuit 34 receives the attribute information output by the output I / F 110D2. In addition, the output I / F 110D3 outputs the digital image data 70B to the signal processing circuit 34 under the control of the control circuit 110C. The receiving I / F 34C of the signal processing circuit 34 receives the digital image data 70B output by the output I / F 110D3.
[0212] according to Fig.23 In the illustrated configuration, the output I / F 110D2 outputs the attribute information, and the output I / F 110D3 outputs the digital image data 70B, so that the attribute information and the digital image data 70B can be output in parallel.
[0213] In the above embodiment, the lens-interchangeable camera device 10 is exemplified, but the technology of the present invention is not limited to this. Fig.24 As shown, the smart device 200 may be equipped with a camera body 214 having a structure and function equivalent to the camera body 12 described in the above embodiment.
[0214] As an example, Fig.24As shown, the smart device 200 includes a housing 212, and a camera body 214 is accommodated in the housing 212. A smart phone can be cited as an example of the smart device 200. Here, as the smart device 200, for example, a smart phone or a tablet terminal as an electronic device with a camera function can be cited.
[0215] The imaging lens 216 is mounted on the frame 212. Fig.24 In the example shown, the imaging lens 216 is exposed from the back surface 212A at the upper left portion of the back surface 212A of the frame body 212 when the smart device 200 is placed vertically. The center of the imaging lens 216 is located on the optical axis L1. The imaging element 38 is built into the camera body 214. The camera body 214 takes in the subject light from the imaging lens 216. The subject light taken into the camera body 214 is formed on the imaging element 38, and the image is captured as described in the above embodiment.
[0216] As an example, Fig.25 As shown, an indication key 222 and a touch panel display 224 are provided on the front surface 212B of the frame 212. The indication key 222 is arranged at the lower part of the front surface 212B when the smart device 200 is placed vertically, and the touch panel display 224 is arranged above the indication key 222. The touch panel display 224 has a display 226 and a touch panel 228, and the touch panel 228 is overlapped on the display 226. The touch panel display 224 and the indication key 222 have the same functions as the UI system device 17 described in the above embodiment. That is, the touch panel display 224 and the indication key 222 prompt information to the user or receive instructions from the user. The instructions from the user are realized by, for example, touching a soft key (for example, a soft key equivalent to the release button 20, etc.) displayed on the touch panel display 224.
[0217] In addition, the display 226 is an example of a "display unit (display)" involved in the technology of the present invention. In addition, the smart device 200 is an example of a "camera" involved in the technology of the present invention. Fig.24 In the example shown, only the camera body 214 is built into the smart device 200, but the technology of the present invention is not limited to this. Multiple digital cameras can be built into the smart device 200. In this case, it is sufficient that the camera body 214 is installed in at least one of the digital cameras.
[0218] Furthermore, in the above embodiment, an imaging element in which the photoelectric conversion element 42, the processing circuit 110, and the memory 112 are integrated into a single chip is exemplified as the imaging element 38, but the technology of the present invention is not limited thereto. For example, at least the photoelectric conversion element 42 and the memory 112 among the photoelectric conversion element 42, the processing circuit 110, and the memory 112 may be integrated into a single chip.
[0219] Furthermore, in the above-mentioned embodiment, the attribute information is exemplified as information including address, image size, shooting time and shooting conditions, but the technology of the present invention is not limited to this. For example, the attribute information may be information including address among address, image size, shooting time and shooting conditions, or information including address, image size, shooting time and / or shooting conditions.
[0220] Furthermore, in the above embodiment, the communication I / Fs are connected according to the PCI-e connection standard, but the technology of the present invention is not limited thereto. Other connection standards such as LVDS, SATA, SLVS-EC or MIPI may also be used to replace the PCI-e connection standard.
[0221] Furthermore, in the above-mentioned embodiment, the communication between the imaging element 38 and the signal processing circuit 34, the communication between the controller 15 and the imaging element 38, and the communication between the signal processing circuit 34 and the controller 15 are all wired communications. However, the technology of the present invention is not limited thereto. The communication between the imaging element 38 and the signal processing circuit 34, the communication between the controller 15 and the imaging element 38, and / or the communication between the signal processing circuit 34 and the controller 15 may also be wireless communications.
[0222] Furthermore, in the above-mentioned embodiment, the UI system device 17 is assembled in the camera body 12, but at least a part of the multiple components included in the UI system device 17 may be external to the camera body 12. Furthermore, at least a part of the multiple components included in the UI system device 17 may be connected to the external I / F 104 as a separate body for use.
[0223] Furthermore, in the above-mentioned embodiment, 120 fps is exemplified as the frame rate, but the technology of the present invention is not limited to this, and the image capture frame rate (for example, Fig. 9 The frame rate used in the video recording process shown in FIG. 1 ) and the frame rate used for output (for example, Fig. 9The frame rate used in the output process shown and / or the frame rate for output of the digital image data 70B) is set to a different frame rate. The frame rate for imaging and / or the frame rate for output may be a fixed frame rate or a variable frame rate. In the case of a variable frame rate, the frame rate may be changed, for example, when a prescribed condition (for example, a condition that an instruction to change the frame rate is received by the receiving device 84 and / or a condition that a timing predetermined as the timing for changing the frame rate arrives) is satisfied. In the case of a variable frame rate, the specific numerical value of the frame rate may be changed, for example, according to an instruction received by the receiving device 84, or may be changed according to the operating rate of the subsequent circuit 13 and / or the imaging element 38.
[0224] Furthermore, in the above embodiment, the signal processing circuit 34 is illustrated, but the technology of the present invention is not limited thereto, and one or more signal processing circuits may be used in addition to the signal processing circuit 34. In this case, the imaging element 38 may be directly connected to each of the plurality of signal processing circuits.
[0225] Furthermore, in the above embodiment, the processing circuit 110 is implemented by devices including ASIC and FPGA, but the technology of the present invention is not limited thereto. For example, at least the control circuit 110C among the multiple devices included in the processing circuit 110 may be implemented by a computer-based software structure.
[0226] In this case, for example, Fig.26 As shown, a computer 852 is built in the imaging element 38, and an attribute information output program 902 for causing the computer 852 to execute the attribute information output process involved in the above-mentioned embodiment and a data erasure program 904 for causing the computer 852 to execute the data erasure process involved in the above-mentioned embodiment are stored in a storage medium 900. As an example of the storage medium 900, any portable storage medium such as an SSD or a USB memory as a non-temporary storage medium can be cited. In the following, for the sake of convenience of explanation, when it is not necessary to distinguish between the attribute information output process and the data erasure process, they are referred to as "imaging element side process", and when it is not necessary to distinguish between the attribute information output program 902 and the data erasure program 904, they are referred to as "imaging element side program".
[0227] The computer 852 includes a CPU 852A, a storage 852B, and a memory 852C. The storage 852B is a nonvolatile storage device such as an EEPROM, and the memory 852C is a volatile storage device such as a RAM. The imaging element side program stored in the storage medium 900 is installed in the computer 852. The CPU 852A executes the imaging element side processing according to the imaging element side program.
[0228] The imaging element side program may be stored in the storage 852B instead of the storage medium 900. In this case, the CPU 852A reads the imaging element side program from the storage 852B and expands the read imaging element side program to the memory 852C. Then, the CPU 852A executes the imaging element side processing according to the imaging element side program expanded to the memory 852C.
[0229] Furthermore, here, an example is given in which the attribute information output program 902 and the data erasure program 904 are stored in the storage 852B, but the present invention is not limited to this. One of the attribute information output program 902 and the data erasure program 904 may be stored in the storage 852B and the other may be stored in the storage medium 900.
[0230] Furthermore, the imaging element side program may be stored in a storage unit such as another computer or server device connected to the computer 852 via a communication network (not shown), and downloaded and installed on the computer 852 in response to a request from the imaging device 10 or smart device 200.
[0231] In addition, it is not necessary to store all imaging element side programs in a storage unit such as another computer or server device connected to the computer 852, and a part of the imaging element side program (for example, the attribute information output program 902 or the data erasing program 904) may be stored.
[0232] exist Fig.26 In the illustrated example, the computer 852 is built into the imaging element 38 , but the technology of the present invention is not limited to this. For example, the computer 852 may be provided outside the imaging element 38 .
[0233] exist Fig.26 In the example shown, the CPU 852A is a single CPU, but it may be a plurality of CPUs. Furthermore, a GPU may be used instead of the CPU 852A.
[0234] exist Fig.26 In the example shown, the computer 852 is illustrated, but the technology of the present invention is not limited thereto, and a device including an ASIC, FPGA, and / or PLD may be used instead of the computer 852. Furthermore, the computer 852 may be replaced by a combination of a hardware structure and a software structure.
[0235] As hardware resources for executing the imaging element side processing described in the above embodiments, various processors shown below can be used. As a processor, for example, a general-purpose processor, i.e., a CPU, can be cited, which functions as a hardware resource for executing the imaging element side processing by executing software, i.e., a program, as described above. In addition, as a processor, for example, a dedicated circuit as a processor can be cited, and the processor has a circuit structure specially designed for executing specific processing, such as an FPGA, a PLD, or an ASIC. Any processor has a memory built in or connected, and any processor uses the memory to execute the imaging element side processing.
[0236] The hardware resource for performing imaging element side processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware resource for performing imaging element side processing may be a single processor.
[0237] As an example of a system composed of one processor, there is a first method in which a processor is composed of a combination of one or more CPUs and software, such as a client and server computer, and the processor functions as a hardware resource for performing imaging element side processing. Second, there is a method in which a processor is used to realize the functions of the entire system including multiple hardware resources for performing imaging element side processing, such as a SoC, etc., using one IC chip. In this way, the imaging element side processing is realized by using one or more of the above-mentioned various processors as hardware resources.
[0238] Furthermore, as the hardware configuration of these various processors, more specifically, a circuit combining circuit elements such as semiconductor elements can be used.
[0239] And, in Figure 1 In the example shown, the imaging device 10 is illustrated. Fig.24 In the example shown, the smart device 200 is illustrated, but the technology of the present invention is not limited thereto. That is, the technology of the present invention can also be applied to various electronic devices (for example, fixed-lens cameras, personal computers, or wearable terminal devices, etc.) that have built-in imaging devices having structures and functions equivalent to the imaging device body 12 described in the above embodiment, and even in these electronic devices, the same functions and effects as those of the imaging device 10 and the smart device 200 can be obtained.
[0240] Furthermore, in the above-mentioned embodiments, the displays 26 and 226 are exemplified, but the technology of the present invention is not limited thereto. For example, a separate display attached to the imaging device may be used as the "display unit" involved in the technology of the present invention.
[0241] Furthermore, the above-mentioned various processes are merely examples, and therefore, it is of course possible to delete unnecessary steps, add new steps, or switch the processing order without departing from the scope of the subject matter.
[0242] The recorded contents and illustrated contents shown above are detailed descriptions of the parts involved in the technology of the present invention, and are only an example of the technology of the present invention. For example, the descriptions related to the above-mentioned structure, function, action and effect are descriptions related to an example of the structure, function, action and effect of the parts involved in the technology of the present invention. Therefore, without departing from the scope of the main purpose of the technology of the present invention, it is of course possible to delete unnecessary parts of the recorded contents and illustrated contents shown above, or to add new elements, or to replace them. In addition, in order to avoid complication and to make it easy to understand the parts involved in the technology of the present invention, in the recorded contents and illustrated contents shown above, descriptions related to technical common sense, etc. that do not need to be specifically explained in terms of the technology that can implement the present invention are omitted.
[0243] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" may mean only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" is added to express three or more items, the same concept as "A and / or B" may also be applied.
[0244] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each document, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.
[0245] Regarding the above embodiment, the following supplementary notes are further disclosed.
[0246] (Note 1)
[0247] An imaging element having:
[0248] processor; and
[0249] A memory for storing image data obtained by shooting and built into the imaging element,
[0250] The above processor proceeds as follows:
[0251] controlling storage of the image data into the memory,
[0252] storing the attribute information of the image data in the memory,
[0253] outputting the image data stored in the memory,
[0254] receiving instructions related to the attribute information,
[0255] The attribute information corresponding to the received instruction is output.
Claims
1. An imaging element comprising: a memory for storing image data obtained by shooting and built into the imaging element; a control circuit that controls storage of the image data in the memory, stores attribute information of the image data in the memory, and is built into the imaging element; An output interface, built into the imaging element; and A receiving interface for receiving an indication related to the attribute information, The output interface outputs the attribute information corresponding to the instruction received by the receiving interface, The receiving interface receives the address acquired from the attribute information output by the output interface, The output interface outputs the image data selectively acquired from the memory by the control circuit according to the address received by the receiving interface. The output interface includes a first output interface and a second output interface. The first output interface outputs the image data, The second output interface outputs the attribute information. The image data and the attribute information are output in parallel through the output interface.
2. The imaging element according to claim 1, wherein The output interface outputs the attribute information at a timing when the receiving interface receives the instruction.
3. The imaging element according to claim 1 or 2, wherein The indication is a frame synchronization signal from the outside.
4. The imaging element according to claim 1 or 2, wherein The output interface outputs the attribute information related to the latest image data among the image data stored in the memory.
5. The imaging element according to claim 4, wherein The output interface outputs the attribute information related to the latest image data at the timing when the receiving interface receives the instruction.
6. The imaging element according to any one of claims 1, 2, and 5, wherein: The output interface is capable of outputting respective attribute information corresponding to the plurality of image data. The attribute information is output by the output interface in the order of shooting.
7. The imaging element according to any one of claims 1, 2, and 5, wherein: The attribute information is information including at least one of an address, an image size, a shooting time, and a shooting condition.
8. The imaging element according to any one of claims 1, 2, and 5, wherein: When the image data is erased from the memory in conjunction with the next imaging, the attribute information output from the output interface includes erase information indicating that the image data is erased from the memory in conjunction with the next imaging.
9. The imaging element according to claim 8, wherein When erasing the image data from the memory, the control circuit erases the attribute information related to the erasure target image data, which is the image data to be erased in the memory, from the memory, and erases the erasure target image data from the memory.
10. The imaging element according to any one of claims 1, 2, 5, and 9, comprising a compression circuit for compressing the image data. The control circuit stores the image data of a specified number of frames in the memory, stores compressed image data obtained by compressing the image data by the compression circuit in the memory, and establishes a corresponding association between the attribute information related to the compressed image data and the compression determination information that can determine that the image data is compressed and the compressed image data.
11. The imaging element according to claim 10, wherein When the compressed image data is stored in the memory, the attribute information output from the output interface includes compression schedule information indicating that the compressed image data is scheduled to be stored in the memory.
12. The imaging element according to claim 10, wherein The attribute information output from the output interface includes information capable of identifying a compression format of the compressed image data.
13. The imaging element according to any one of claims 1, 2, 5, 9, 11, and 12, wherein: The instruction includes an output amount based on the attribute information of the output interface, The output interface outputs the attribute information in the output amount.
14. The imaging element according to claim 13, wherein The output amount is specified by the number of frames of the image data.
15. The imaging element according to any one of claims 1, 2, 5, 9, 11, 12, and 14, further comprising: a receiver that receives the attribute information transmitted from a subsequent circuit located after the imaging element, The control circuit acquires the image data from the memory according to the attribute information received by the receiver, and causes the output interface to output the acquired image data to the subsequent circuit.
16. The imaging element according to any one of claims 1, 2, 5, 9, 11, 12, and 14, wherein at least a photoelectric conversion element and the memory are integrated into a single chip.
17. The imaging element according to claim 16, wherein The imaging element is a stacked type imaging element in which the memory is stacked on the photoelectric conversion element.
18. A camera device comprising: The imaging element according to any one of claims 1 to 17; and The control device performs at least one of control of causing a display to display an image based on the image data outputted from the output interface and control of storing the image data outputted from the output interface in a storage device.
19. A method for operating an imaging element, wherein the imaging element includes a memory for storing image data obtained by shooting, and an output interface having a first output interface and a second output interface, the method comprising: controlling storage of the image data into the memory; storing the attribute information of the image data in the memory, receiving an indication related to the attribute information; and outputting the attribute information corresponding to the received instruction, The working method of the imaging element further includes: receiving an address obtained from the outputted attribute information; and outputting the image data selectively acquired from the memory according to the received address, The first output interface outputs the image data, The second output interface outputs the attribute information. The image data is output in parallel with the attribute information.
20. A computer-readable non-transitory storage medium storing a program for causing a computer to execute a process, the computer being adapted for use with an imaging element having a memory for storing image data obtained by capturing images and an output interface having a first output interface and a second output interface, the process comprising: controlling storage of the image data into the memory; storing the attribute information of the image data in the memory, receiving an indication related to the attribute information; and outputting the attribute information corresponding to the received instruction, The processing also includes: receiving an address obtained from the outputted attribute information; and outputting the image data selectively acquired from the memory according to the received address, The first output interface outputs the image data, The second output interface outputs the attribute information. The image data is output in parallel with the attribute information.
