Imaging element, imaging device, method of operating imaging element, and program

Abstract

To provide an image pick-up device, an imaging apparatus, a method for operating an image pick-up device, and a program that can prevent a delay in the output of image data, compared to a case where image data is output to a processing unit (an external processor arranged outside the image pick-up device) only from a single communication I / F.SOLUTION: An image pick-up device comprises: a first communication interface that outputs, to an external processor, first image data based on image data obtained by picking up an image of a subject, and is built in the image pick-up device; a memory that stores the image data and is built in the image pick-up device; and a second communication interface that outputs, to the external processor, second image data based on the image data stored in the memory, and is built in the image pick-up device. An output system of the first communication interface and an output system of the second communication interface are different from each other.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The technology of the present disclosure relates to an imaging device, an imaging apparatus, a method of operating the imaging device, and a program.

Background Art

[0002] Japanese Unexamined Patent Application Publication No. 2018-6806 discloses an imaging apparatus including a stacked image sensor having a sensor unit, a first logic unit, and a first memory unit, and a second logic unit.

[0003] The sensor unit is a so-called CMOS (Complementary Metal Oxide Semiconductor) image sensor unit. The sensor unit converts received light into an electrical signal. Then, the sensor unit digitizes the electrical signal and transmits the obtained RAW data after digitization to the first logic unit.

[0004] The first logic unit includes a first memory control unit, a first inter-chip communication I / F (Interface), a simple development unit, and a first display control unit. The first memory control unit is a so-called memory controller, which writes the RAW data from the sensor unit to the first memory unit. The first inter-chip communication I / F accesses the first memory unit via the first memory control unit and transfers the RAW data read from the first memory unit to the second logic unit. The simple development unit accesses the first memory unit via the first memory control unit, performs development processing on the RAW data read from the first memory unit, and generates display data that can be displayed on the display unit. The simple development unit writes back the display data to the first memory unit via the first memory control unit. The first display control unit reads the display data from the first memory unit via the first memory control unit and outputs it to the second logic unit while reading.

[0005] In the imaging apparatus described in Japanese Unexamined Patent Application Publication No. 2018-6806, image data is output from the stacked image sensor to the second logic unit via each of two output paths.

Summary of the Invention

[0006] One embodiment of the technology according to the present disclosure provides an image pickup device, an imaging apparatus, a method of operating the image pickup device, and a program that can suppress a delay in output of image data compared to a case where image data is output from only a single communication I / F to a processing unit (an external processor disposed outside the image pickup device).

Means for Solving the Problems

[0007] A first aspect of the technology according to the present disclosure is an image pickup device that outputs first image data based on image data obtained by imaging a subject to a processing unit outside the image pickup device, and includes a first output unit built in the image pickup device, a storage unit that stores the image data and is built in the image pickup device, and outputs second image data based on the image data stored in the storage unit to the external processing unit, and a second output unit built in the image pickup device, and the output method of the first output unit and the output method of the second output unit are different. Thereby, even when image data is stored in the storage unit, the image data can be output without delay.

[0008] A second aspect of the technology according to the present disclosure is an image pickup device according to the first aspect, in which the output of the first image data by the first output unit and the output of the second image data by the second output unit are performed independently of each other. Thereby, the output timing of the first image data to the processing unit and the output timing of the second image data to the processing unit can be freely changed.

[0009] A third aspect of the technology according to the present disclosure is an image pickup device according to the first aspect, in which the first output unit outputs the first image data during a period different from the output period of the second image data by the second output unit. Thereby, the image data can be output to the processing unit without delay.

[0010] A fourth aspect of the technology according to the present disclosure is an imaging device according to a third aspect in which the second output unit outputs second image data in response to a request from an external processing unit. Thereby, it is possible to avoid the second image data from being output to the processing unit even when the processing unit is not in a state of receiving the second image data.

[0011] A fifth aspect of the technology according to the present disclosure is an imaging device according to a third or fourth aspect in which the output period is a vertical blanking period after one frame of first image data has been output from the first output unit. Thereby, it is possible to avoid the output from the imaging device to the processing unit from being stalled due to the writing operation of the image data to the storage unit.

[0012] A sixth aspect of the technology according to the present disclosure is an imaging device according to a third or fourth aspect in which the output period is a vertical blanking period before one frame of first image data is output from the first output unit. Thereby, it is possible to avoid the output from the imaging device to the processing unit from being stalled due to the writing operation of the image data to the storage unit.

[0013] A seventh aspect of the technology according to the present disclosure is an imaging device according to a third or fourth aspect in which the output period is a horizontal blanking period after one line of first image data has been output from the first output unit. Thereby, it is possible to avoid the output from the imaging device to the processing unit from being stalled due to the writing operation of the image data to the storage unit.

[0014] An eighth aspect of the technology according to the present disclosure is an imaging device according to a third or fourth aspect in which the output period is a horizontal blanking period before one line of first image data is output from the first output unit. Thereby, it is possible to avoid the output from the imaging device to the processing unit from being stalled due to the writing operation of the image data to the storage unit.

[0015] A ninth aspect of the technology according to the present disclosure is an image sensor according to the third or fourth aspect, including a first A / D converter that A / D-converts analog image data, and an output period being an A / D conversion period by the first A / D converter before first image data for one line is output from the first output unit. Thereby, it is possible to avoid the output from the image sensor to the processing unit from stagnating due to the operation of writing the image data to the storage unit.

[0016] A tenth aspect of the technology according to the present disclosure is an image sensor according to any one of the first to ninth aspects, including a second A / D converter that A / D-converts analog image data, and a memory controller that stores the digital image data obtained by digitizing the analog image data by the second A / D converter in a storage unit. The output method of the first output unit is an output method that outputs the digital image data obtained from the second A / D converter as first image data without storing it in the storage unit, and the output method of the second output unit is an output method that outputs the digital image data read from the storage unit by the memory controller as second image data. Thereby, even during the period when the image data is being written to the storage unit, the output from the image sensor to the processing unit can be continued.

[0017] An eleventh aspect of the technology according to the present disclosure is an image sensor according to any one of the first to tenth aspects, wherein the storage unit is a memory with different write timings and read timings. Thereby, even if the storage unit is a memory with different write timings and read timings, the output from the image sensor to the processing unit can be continued.

[0018] A twelfth aspect of the technology according to the present disclosure is an image sensor according to the tenth aspect, wherein the storage unit is a DRAM. Thereby, even if the storage unit is a DRAM, the output from the image sensor to the processing unit can be continued.

[0019] The 13th aspect according to the technology of the present disclosure is an imaging device according to any one of the 1st to 12th aspects in which at least a photoelectric conversion element and a memory unit are integrated on one chip. Thereby, the portability of the imaging device is improved as compared with an imaging device in which the photoelectric conversion element and the memory unit are not integrated on one chip.

[0020] The 14th aspect according to the technology of the present disclosure is an imaging device according to the 13th aspect, which is a stacked imaging device in which a memory unit is stacked on a photoelectric conversion element. Thereby, the transfer speed of image data from the photoelectric conversion element to the memory unit can be increased as compared with the case where the photoelectric conversion element and the memory unit are not stacked.

[0021] The 15th aspect according to the technology of the present disclosure is an imaging device including an imaging device according to any one of the 1st to 14th aspects, and a display control unit that performs control to display at least one of a first image based on first image data output by a first output unit and a second image based on second image data output by a second output unit on a display unit. Thereby, even when image data is stored in the memory unit, the image data can be output without delay.

[0022] The 16th aspect according to the technology of the present disclosure is an imaging device including an imaging device according to any one of the 1st to 14th aspects, and a storage control unit that performs control to store at least one of the first image data output by the first output unit and the second image data output by the second output unit in a storage device. Thereby, even when image data is stored in the memory unit, the image data can be output without delay.

[0023] A 17th aspect of the technology according to the present disclosure is an operation method of an image sensor incorporating a first output unit, a storage unit, and a second output unit. The first output unit outputs first image data based on image data obtained by imaging a subject to a processing unit outside the image sensor. The storage unit stores the image data. The second output unit outputs second image data based on the image data stored in the storage unit to the external processing unit. The operation method of the image sensor is characterized in that the output method of the first output unit and the output method of the second output unit are different. Thereby, even when the image data is stored in the storage unit, the image data can be output without delay.

[0024] An 18th aspect of the technology according to the present disclosure is a program for causing a computer to function as a first output unit and a second output unit included in an image sensor incorporating a first output unit, a storage unit, and a second output unit. The first output unit outputs first image data based on image data obtained by imaging a subject to a processing unit outside the image sensor. The storage unit stores the image data. The second output unit outputs second image data based on the image data stored in the storage unit to the external processing unit. The program is characterized in that the output method of the first output unit and the output method of the second output unit are different. Thereby, even when the image data is stored in the storage unit, the image data can be output without delay.

[0025] A 19th aspect of the technology according to the present disclosure is an image sensor incorporating a first processor, a memory, and a second processor. The first processor outputs first image data based on image data obtained by imaging a subject to a processing unit outside the image sensor. The memory stores the image data. The second processor outputs second image data based on the image data stored in the memory to the external processing unit. The image sensor is characterized in that the output method of the first processor and the output method of the second processor are different. Thereby, even when the image data is stored in the storage unit even so, the image data can be output without delay.

Brief Description of Drawings

[0026]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23A

Figure 23B

Figure 24

Figure 25

Embodiments for Carrying Out the Invention

[0027] Hereinafter, an example of an embodiment of an imaging device according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0028] First, the terms used in the following description will be explained.

[0029] CPU refers to the abbreviation of "Central Processing Unit". RAM refers to the abbreviation of "Random Access Memory". ROM refers to the abbreviation of "Read Only Memory". DRAM refers to the abbreviation of "Dynamic Random Access Memory". SRAM refers to the abbreviation of "Static Random Access Memory".

[0030] LSI refers to the abbreviation of "Large-Scale Integration". ASIC refers to the abbreviation of "Application Specific Integrated Circuit". PLD refers to the abbreviation of "Programmable Logic Device". FPGA refers to the abbreviation of "Field-Programmable Gate Array".

[0031] SSD refers to the abbreviation of "Solid State Drive". DVD-ROM refers to the abbreviation of "Digital Versatile Disc Read Only Memory". USB refers to the abbreviation of "Universal Serial Bus". HDD refers to the abbreviation of "Hard Disk Drive". EEPROM refers to the abbreviation of "Electrically Erasable and Programmable Read Only Memory".

[0032] CCD refers to the abbreviation of "Charge Coupled Device". CMOS refers to the abbreviation of "Complementary Metal Oxide Semiconductor". EL refers to the abbreviation of "Electro-Luminescence". A / D refers to the abbreviation of "Analog / Digital". I / F refers to the abbreviation of "Interface". UI refers to the abbreviation of "User Interface".

[0033] LVDS refers to the abbreviation of "Low Voltage Differential Signaling". PCI-e refers to the abbreviation of "Peripheral Component Interconnect Express". SATA refers to the abbreviation of "Serial Advanced Technology Attachment". SLVS-EC refers to the abbreviation of "Scalable Low Signaling with Embedded Clock". MIPI refers to the abbreviation of "Mobile Industry Processor Interface".

[0034] [First Embodiment] As an example, as shown in FIG. 1, the imaging device 10 is an interchangeable-lens camera. The imaging device 10 includes an imaging device body 12 and an interchangeable lens 14 that is detachably attached to the imaging device body 12, and is a digital camera in which a reflex mirror is omitted.

[0035] An image sensor 44 is provided in the imaging device body 12. When the interchangeable lens 14 is attached to the imaging device body 12, subject light indicating a subject passes through the interchangeable lens 14 and forms an image on the image sensor 44, and the image sensor 44 generates image data 69 (see FIGS. 3 and 4) indicating an image of the subject.

[0036] A hybrid finder (registered trademark) 16 is provided in the imaging device body 12 The hybrid viewfinder 16 mentioned here refers to a viewfinder in which, for example, an optical viewfinder (hereinafter referred to as "OVF") and an electronic viewfinder (hereinafter referred to as "EVF") are selectively used. Note that OVF refers to the abbreviation of "optical viewfinder". Also, EVF refers to the abbreviation of "electronic viewfinder".

[0037] On the front surface of the imaging device main body 12, a finder switching lever 18 is provided. The optical image visible through the OVF and the live view image, which is an electronic image visible through the EVF, are switched by rotating the finder switching lever 18 in the direction of arrow SW. The "live view image" mentioned here refers to a display moving image based on image data 69 (see FIGS. 3 and 4) obtained by imaging with the imaging element 44. The live view image is generally also referred to as a through image. On the upper surface of the imaging device main body 12, a release button 20 and a dial 22 are provided. The dial 22 is operated when setting the operation mode of the imaging system and the operation mode of the playback system, etc.

[0038] The release button 20 functions as an imaging preparation instruction unit and an imaging instruction unit, and can detect a two-stage pressing operation of an imaging preparation instruction state and an imaging instruction state. The imaging preparation instruction state refers to a state where it is pressed from the standby position to the intermediate position (half-press position), for example, and the imaging instruction state refers to a state where it is pressed to the final pressing position (full-press position) exceeding the intermediate position.

[0039] In the imaging device 10, an imaging mode and a playback mode are selectively set according to the user's instruction as operation modes. The imaging mode is roughly classified into a display video imaging mode and a recording imaging mode.

[0040] As shown in FIG. 2 as an example, on the back surface of the imaging device main body 12, a touch panel display 26, an instruction key 28, and a finder eyepiece part 30 are provided.

[0041] The touch panel display 26 includes a first display 32 and a touch panel 34 (see also FIG. 5). As an example of the first display 32, a liquid crystal display can be mentioned. Note that the first display 32 may be other displays such as an organic EL display instead of a liquid crystal display.

[0042] The first display 32 displays images, character information, etc. The first display 32 is used for displaying a live view image obtained by continuous imaging when the imaging device 10 is in the imaging mode. Further, the first display 32 is also used for displaying a still image obtained by imaging when an instruction for still image imaging is given. Furthermore, the first display 32 is also used for displaying a playback image and a menu screen, etc. when the imaging device 10 is in the playback mode.

[0043] The touch panel 34 is a transmissive touch panel and is overlaid on the surface of the display area of the first display 32. The touch panel 34 detects contact by an indicator such as a finger or a stylus pen.

[0044] The instruction keys 28 receive various instructions such as selection of one or more menus, confirmation of selection contents, deletion of selection contents, zooming, and frame advance.

[0045] As shown in FIG. 3 as an example, the interchangeable lens 14 has an imaging lens 40. The imaging lens 40 includes an objective lens 40A, a focus lens 40B, and a diaphragm 40C. The objective lens 40A, the focus lens 40B, and the diaphragm 40C are arranged in the order of the objective lens 40A, the focus lens 40B, and the diaphragm 40C along the optical axis L1 from the subject side to the imaging device body 12 side. The focus lens 40B and the diaphragm 40C operate by receiving power from a drive source (not shown) such as a motor. That is, the focus lens 40B and the diaphragm 40C move along the optical axis L1 according to the applied power. Further, the diaphragm 40C adjusts the exposure by operating according to the applied power.

[0046] The imaging device main body 12 includes a mechanical shutter 42, an imaging element 44, and a processing unit 45. The mechanical shutter 42 operates by receiving power from a drive source (not shown) such as a motor. The imaging element 44 includes a photoelectric conversion element 61 having a light receiving surface 61A. When the interchangeable lens 14 is attached to the imaging device main body 12, subject light indicating a subject passes through the imaging lens 40 and is imaged on the light receiving surface 61A of the imaging element 44 via the mechanical shutter 42. The photoelectric conversion element 61 generates image data 69 indicating an image of the subject by photoelectrically converting the subject light imaged on the light receiving surface 61A. The imaging element 44 digitizes the image data 69 generated by the photoelectric conversion element 61 and outputs it to the processing unit 45 via each of the communication lines 53 and 55.

[0047] The imaging device main body 12 includes a processing unit 45 and a UI system device 48. The processing unit 45 is an external processor disposed outside the imaging element 44. The processing unit 45 is an example of the "processing unit outside the imaging element" according to the technology of the present disclosure. The processing unit 45 is an electric circuit located downstream of the imaging element 44 and includes a controller 46 and a signal processing circuit 50.

[0048] The controller 46 controls the entire imaging device 10. The UI system device 48 is a device that presents information to the user and receives instructions from the user. The UI system device 48 is connected to the controller 46, and the controller 46 acquires various information from the UI system device 48 and controls the UI system device 48.

[0049] The imaging element 44 is connected to the controller 46 via a communication line 57 and generates image data 69 by imaging a subject under the control of the controller 46.

[0050] The imaging element 44 is connected to the signal processing circuit 50 via the communication line 53 and the communication line 55. Specifically, between the imaging element 44 and the signal processing circuit 50, they are connected in parallel by the communication line 53 and the communication line 55. Between the imaging element 44 and the signal processing circuit 50, they are connected according to the PCI-e connection standard via the communication line 53, and are connected according to the LVDS connection standard via the communication line 55.

[0051] Here, PCI-e and LVDS are exemplified as the connection standards, but the technology of the present disclosure is not limited thereto, and other connection standards may be used. Examples of other connection standards include SATA, SLVS-EC, and MIPI, etc. However, these connection standards are merely examples, and any connection standard that enables independent communication between the communication via the communication line 53 and the communication via the communication line 55 between the imaging element 44 and the signal processing circuit 50 is acceptable.

[0052] Also, here, an example form in which communication is performed in a wired form using the communication line 53 and the communication line 55 between the imaging element 44 and the signal processing circuit 50 is given, but the technology of the present disclosure is not limited thereto. For example, instead of communication in a wired form via each of the communication line 53 and the communication line 55 between the imaging element 44 and the signal processing circuit 50, wireless communication may be performed between the imaging element 44 and the signal processing circuit 50. In this case, a first wireless communication path corresponding to the wired communication path via the communication line 53 and a second wireless communication path corresponding to the wired communication path via the communication line 55 may be ensured. The first communication path and the second communication path are communication paths that enable wireless communication in wireless forms with different communication standards from each other and in non-interfering frequency bands. Also, between the imaging element 44 and the signal processing circuit 50, independent communication may be performed using two communication paths, a wired communication path and a wireless communication path.

[0053] The signal processing circuit 50 is an LSI, specifically a device including an ASIC. A controller 46 is connected to the signal processing circuit 50 via a communication line 60, and the controller 46 acquires various information from the signal processing circuit 50 and controls the signal processing circuit 50.

[0054] Image data 69 is input to the signal processing circuit 50 from the image sensor 44 via communication lines 53 and 55. Although details will be described later, the signal processing circuit 50 performs various signal processes on the image data 69 input via the communication lines 53 and 55.

[0055] In this embodiment, a device including an ASIC is adopted as the signal processing circuit 50. However, this is merely an example, and the signal processing circuit 50 may be a device including an ASIC, an FPGA, and / or a PLD. Also, the signal processing circuit 50 may be a computer including a CPU, a ROM, and a RAM. The CPU may be singular or plural. Further, the signal processing circuit 50 may be realized by a combination of a hardware configuration and a software configuration.

[0056] The image sensor 44 is an example of the "stacked image sensor" according to the technology of the present disclosure. In this embodiment, the image sensor 44 is a CMOS image sensor. Also, here, a CMOS image sensor is exemplified as the image sensor 44, but the technology of the present disclosure is not limited thereto. For example, even if the image sensor 44 is a CCD image sensor, the technology of the present disclosure holds.

[0057] As an example, as shown in FIG. 4, a readout synchronization signal is input to the image sensor 44 from the controller 46 via the communication line 57. The readout synchronization signal includes a vertical synchronization signal and a horizontal synchronization signal. The vertical synchronization signal is a synchronization signal that defines the start timing of reading out image data 69 for each frame from the photoelectric conversion element 61. The horizontal synchronization signal is a synchronization signal that defines the start timing of reading out image data 69 for each horizontal line from the photoelectric conversion element 61. In the image sensor 44, the image data 69 is read out from the photoelectric conversion element 61 according to the frame rate determined according to the vertical synchronization signal input from the controller 46 via the communication line 57.

[0058] In the example shown in FIG. 4, as the frame rate of the image sensor 44, a frame rate at which reading out for 8 frames from the photoelectric conversion element 61 is performed within the period T is shown. As an example of a specific frame rate, 120 fps (frame per second) can be mentioned.

[0059] As an example, as shown in FIG. 5, the controller 46 includes a CPU 46A, a ROM 46B, a RAM 46C, a connection I / F 46D, and an input I / F 46E. The CPU 46A, the ROM 46B, the RAM 46C, the connection I / F 46D, and the input I / F 46E are interconnected via a bus line 88.

[0060] Various programs are stored in the ROM 46B. The CPU 46A reads out various programs from the ROM 46B and expands the read-out various programs in the RAM 46C. The CPU 46A controls the entire imaging device 10 according to the various programs expanded in the RAM 46C.

[0061] The connection I / F 46D is a communication device having an FPGA and is connected to the image sensor 44 via the communication line 57. The CPU 46A controls the image sensor 4 4 via the connection I / F 46D.

[0062] The input I / F 46E is a communication device having an FPGA and is connected to the signal processing circuit 50 via the communication line 60. Image data 69 (see FIGS. 3 and 4) subjected to various signal processes by the signal processing circuit 50 is input to the input I / F 46E via the communication line 60. The input I / F 46E transfers the image data 69 input from the signal processing circuit 50 to the CPU 46A.

[0063] The bus line 88 is connected to the secondary storage device 80 and the external I / F 82. The secondary storage device 80 is a non-volatile memory such as an SSD, an HDD, or an EEPROM. The CPU 46A reads and writes various information to and from the secondary storage device 80. Note that the secondary storage device 80 is an example of the "storage device" according to the technology of the present disclosure.

[0064] The external I / F 82 is a communication device having an FPGA. External devices (not shown) such as a USB memory and a memory card are connected to the external I / F 82. The external I / F 82 controls the exchange of various information between the CPU 46A and the external devices. Note that external devices such as a USB memory and a memory card are examples of the "storage device" according to the technology of the present disclosure.

[0065] The UI system device 48 includes the hybrid viewfinder 16, the touch panel display 26, and the reception device 84. The first display 32 and the touch panel 34 are connected to the bus line 88. Therefore, the CPU 46A displays various information on the first display 32 and operates according to various instructions received by the touch panel 34.

[0066] The reception device 84 includes the touch panel 34 and the hard key unit 25. The hard key unit 25 is a plurality of hard keys and has a release button 20, a dial 22, and an instruction key 28. The hard key unit 25 is connected to the bus line 88, and the CPU 46A operates according to various instructions received by the hard key unit 25.

[0067] The hybrid viewfinder 16 is provided with a second display 86, and the CPU 46A causes various information to be displayed on the second display 86. As an example of the second display 86, a liquid crystal display can be mentioned. Note that the second display 86 may be other displays such as an organic EL display instead of a liquid crystal display.

[0068] As shown in FIG. 6 as an example, the hybrid viewfinder 16 includes an OVF 90 and an EVF 92. The OVF 90 is an inverse Galilean viewfinder and has an eyepiece lens 94, a prism 96, and an objective lens 98. The EVF 92 has a second display 86, a prism 96, and an eyepiece lens 94.

[0069] Along the optical axis L2 of the objective lens 98 and on the subject side of the objective lens 98, a liquid crystal shutter 100 is disposed. The liquid crystal shutter 100 blocks light so that an optical image does not enter the objective lens 98 when the EVF 92 is used.

[0070] The prism 96 reflects the electronic image or various information displayed on the second display 86 and guides it to the eyepiece lens 94, and combines the optical image with the electronic image and / or various information displayed on the second display 86. As the electronic image displayed on the second display 86, a live view image 102 based on the image data 69 can be mentioned.

[0071] In the case of the OVF mode, the CPU 46A controls the liquid crystal shutter 100 to be in a non-light-blocking state so that the optical image can be visually recognized from the eyepiece lens 94. Also, the CPU 46A In the case of the EVF mode, the CPU 46A controls the liquid crystal shutter 100 to be in a light-blocking state so that only the electronic image displayed on the second display 86 can be visually recognized from the eyepiece lens 94.

[0072] In the following, for convenience of explanation, when there is no need to distinguish between the first display 32 (see FIGS. 2 and 5) and the second display 86, they are referred to as "display" without reference numerals. The display is an example of the "display unit (display)" according to the technology of the present disclosure. Further, the CPU 46A is an example of the "display control unit (display processor)" and the "memory control unit (memory processor)" according to the technology of the present disclosure.

[0073] As shown in FIG. 7 as an example, the imaging device 44 incorporates a photoelectric conversion element 61, a processing circuit 62, and a memory 64. The imaging device 44 is an imaging device in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are integrated into one chip. That is, the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are packaged together. In the imaging device 44, the processing circuit 62 and the memory 64 are stacked on the photoelectric conversion element 61. Specifically, the photoelectric conversion element 61 and the processing circuit 62 are electrically connected to each other by bumps (not shown) having conductivity such as copper, and the processing circuit 62 and the memory 64 are also electrically connected to each other by bumps (not shown) having conductivity such as copper. Here, a three-layer structure of the photoelectric conversion element 61, the processing circuit 62, and the memory 64 is illustrated, but the technology of the present disclosure is not limited thereto, and a two-layer structure including a memory layer in which the processing circuit 62 and the memory 64 are integrated into one layer and the photoelectric conversion element 61 may be used. Note that the memory 64 is an example of the "memory unit (memory)" according to the technology of the present disclosure.

[0074] The processing circuit 62 is, for example, an LSI. The memory 64 is a memory having different writing timings and reading timings. Here, as an example of the memory 64, DRAM is adopted. However, the technology of the present disclosure is not limited thereto, and SRAM may be adopted instead of DRAM as the memory 64.

[0075] The processing circuit 62 is a device including an ASIC and an FPGA, and controls the entire imaging device 44 according to the instructions of the controller 46. Here, an example in which the processing circuit 62 is realized by a device including an ASIC and an FPGA is given, but the technology of the present disclosure is not limited thereto. For example, it may be a device including an ASIC, an FPGA, and / or a PLD. Further, as the processing circuit 62, a computer including a CPU, a ROM, and a RAM may be employed. The CPU may be singular or plural. Further, the processing circuit 62 may be realized by a combination of a hardware configuration and a software configuration.

[0076] The photoelectric conversion element 61 has a plurality of photodiodes arranged in a matrix. As an example of the plurality of photodiodes, photodiodes for "4896×3265" pixels can be mentioned.

[0077] A color filter is arranged on each photodiode included in the photoelectric conversion element 61. The color filter includes a G filter corresponding to G (green) that contributes most to obtaining a luminance signal, an R filter corresponding to R (red), and a B filter corresponding to B (blue). The photoelectric conversion element 61 has R pixels, G pixels, and B pixels.

[0078] An R pixel is a pixel corresponding to a photodiode on which an R filter is arranged, a G pixel is a pixel corresponding to a photodiode on which a G filter is arranged, and a B pixel is a pixel corresponding to a photodiode on which a B filter is arranged. The R pixels, G pixels, and B pixels are arranged with a predetermined periodicity in each of the row direction (horizontal direction) and the column direction (vertical direction). In the present embodiment, the R pixels, G pixels, and B pixels are arranged with a periodicity corresponding to the X-Trans (registered trademark) array However, here, the X-Trans array is exemplified, but the technology of the present disclosure is not limited thereto, and the arrangement of the R pixels, G pixels, and B pixels may be a Bayer array, a honeycomb array, or the like.

[0079] The imaging element 44 has a so-called electronic shutter function, and by operating the electronic shutter function under the control of the controller 46, the charge accumulation time of each photodiode in the photoelectric conversion element 61 is controlled. The charge accumulation time refers to the so-called shutter speed.

[0080] In the imaging device 10, imaging for still images and imaging for moving images are performed in the rolling shutter method. Imaging for still images is realized by operating the electronic shutter function and also operating the mechanical shutter 42 (see FIG. 3), and imaging for moving images is realized by operating the electronic shutter function without operating the mechanical shutter 42. Here, the rolling shutter method is exemplified, but the technology of the present disclosure is not limited to this, and a global shutter method may be applied instead of the rolling shutter method.

[0081] The processing circuit 62 reads out the image data 69 (see FIGS. 3 and 4) obtained by imaging the subject by the photoelectric conversion element 61. The image data 69 is the signal charge accumulated in the photoelectric conversion element 61. The processing circuit 62 performs A / D conversion on the analog image data 69 read from the photoelectric conversion element 61. The processing circuit 62 stores the digital image data 69 obtained by performing A / D conversion on the analog image data 69 in the memory 64.

[0082] The processing circuit 62 is connected to the signal processing circuit 50 via the communication line 53 and the communication line 55. The processing circuit 62 is also connected to the controller 46 via the communication line 57.

[0083] Between the processing circuit 62 and the signal processing circuit 50, communication is performed according to the PCI-e connection standard via the communication line 53, and communication is performed according to the LVDS connection standard via the communication line 55.

[0084] As an example, as shown in FIG. 8, the processing circuit 62 is an example of a "processor" according to the technology of the present disclosure. It includes a read circuit 62A, a digital processing circuit 62B, a selector 62C, a control circuit 62D, and communication I / Fs 62E1, 62E2, 62E3. The communication I / F 62E2 is an example of a "second output unit (second communication interface)" according to the technology of the present disclosure, and the communication I / F 62E3 is an example of a "first output unit (first communication interface)" according to the technology of the present disclosure. Also, the control circuit 62D is an example of a "memory controller" according to the technology of the present disclosure.

[0085] The read circuit 62A is connected to each of the photoelectric conversion element 61, the digital processing circuit 62B, and the control circuit 62D. The memory 64 is connected to the control circuit 62D. The selector 62C is connected to each of the digital processing circuit 62B, the control circuit 62D, and the communication I / F 62E3. Each of the communication I / Fs 62E1, 62E2, 62E3 is connected to the control circuit 62D.

[0086] The above-described image data 69 is roughly classified into analog image data 69A and digital image data 69B, as shown in FIG. 8 as an example. Hereinafter, for the convenience of explanation, when there is no need to distinguish between the analog image data 69A and the digital image data 69B, it is referred to as "image data 69".

[0087] The communication I / F 62E1 is a communication device having an FPGA and is connected to the controller 46 via the communication line 57. The controller 46 outputs a read synchronization signal to the communication I / F 62E1 via the communication line 57. The communication I / F 62E1 receives the read synchronization signal from the controller 46 via the communication line 57 and outputs the received read synchronization signal to the control circuit 62D.

[0088] The communication I / F 62E2 is a communication device having an FPGA, and is connected to the signal processing circuit 50 via the communication line 53 in accordance with the connection standard of PCI-e. The communication I / F 62E2 controls the communication between the signal processing circuit 50 and the control circuit 62D. Here, a communication device having an FPGA is adopted as the communication I / F 62E2, but this is merely an example. The communication I / F 62E2 may be a device including an ASIC, an FPGA, and / or a PLD. Also, the communication I / F 62E2 may be a computer including a CPU, a ROM, and a RAM. The CPU may be single or plural. Further, the communication I / F 62E2 may be realized by a combination of a hardware configuration and a software configuration.

[0089] The communication I / F 62E3 is a communication device having an FPGA, and is connected to the signal processing circuit 50 via the communication line 55 in accordance with the connection standard of LVDS. The communication I / F 62E2 controls the communication between the signal processing circuit 50 and the selector 62C, and the communication between the signal processing circuit 50 and the control circuit 62D. Here, a communication device having an FPGA is adopted as the communication I / F 62E3, but this is merely an example. The communication I / F 62E3 may be a device including an ASIC, an FPGA, and / or a PLD. Also, the communication I / F 62E3 may be a computer including a CPU, a ROM, and a RAM. The CPU may be single or plural. Further, the communication I / F 62E3 may be realized by a combination of a hardware configuration and a software configuration.

[0090] The readout circuit 62A controls the photoelectric conversion element 61 under the control of the control circuit 62D, and reads out the analog image data 69A from the photoelectric conversion element 61. The readout of the analog image data 69A from the photoelectric conversion element 61 is performed in accordance with the readout synchronization signal input from the controller 46 to the processing circuit 62.

[0091] Specifically, first, the communication I / F 62E1 receives a read synchronization signal from the controller 46 and outputs the received read synchronization signal to the control circuit 62D. Next, the control circuit 62D transfers the read synchronization signal input from the communication I / F 62E1 to the read circuit 62A. That is, the vertical synchronization signal and the horizontal synchronization signal are transferred to the read circuit 62A. Then, the read circuit 62A starts reading the analog image data 69A in frame units from the photoelectric conversion element 61 according to the vertical synchronization signal transferred from the control circuit 62D. Also, the read circuit 62A starts reading the analog image data 69A in horizontal line units according to the horizontal synchronization signal transferred from the control circuit 62D.

[0092] The read circuit 62A performs analog signal processing on the analog image data 69A read from the photoelectric conversion element 61. The analog signal processing includes known processes such as noise cancellation processing and analog gain processing. The noise cancellation processing is a process of canceling noise caused by variations in characteristics between pixels included in the photoelectric conversion element 61. The analog gain processing is a process of applying a gain to the analog image data 69A. The analog image data 69A on which the analog signal processing has been performed in this way is output to the digital processing circuit 62B by the read circuit 62A.

[0093] The digital processing circuit 62B includes an A / D converter 62B1. The A / D converter 62B1 performs A / D conversion on the analog image data 69A. The A / D converter 62B1 is an example of the "first A / D converter" and the "second A / D converter" according to the technology of the present disclosure.

[0094] The digital processing circuit 62B performs digital signal processing on the analog image data 69A input from the read circuit 62A. The digital signal processing includes, for example, correlated double sampling, A / D conversion by the A / D converter 62B1, and digital gain processing.

[0095] For the analog image data 69A, correlation double sampling is performed by the digital processing circuit 62B. For the analog image data 69A on which the signal processing of correlation double sampling has been performed, A / D conversion is performed by the A / D converter 62B1, whereby the analog image data 69A is digitized and digital image data 69B is obtained as RAW data. Then, digital gain processing is performed on the digital image data 69B by the digital processing circuit 62B. Digital gain processing refers to the process of applying a gain to the digital image data 69B. The digital image data 69B obtained by performing such digital signal processing is output to the selector 62C by the digital processing circuit 62B.

[0096] The selector 62C selectively transfers the digital image data 69B input from the digital processing circuit 62B to two transfer destinations. That is, the selector 62C selectively transfers the digital image data 69B input from the digital processing circuit 62B to the control circuit 62D and the communication I / F 62E3 according to the instruction of the control circuit 62D.

[0097] The control circuit 62D stores the digital image data 69B input from the selector 62C in the memory 64. The memory 64 is a memory capable of storing digital image data 69B of a plurality of frames. The memory 64 has a storage area (not shown) in units of pixels, and the digital image data 69B is stored in the corresponding storage area of the memory 64 in units of pixels by the control circuit 62D.

[0098] The control circuit 62D is capable of random access to the memory 64, and acquires the digital image data 69B from the memory 64 in response to a request from the signal processing circuit 50 via the communication I / F 62E2. The control circuit 62D outputs the digital image data 69B acquired from the memory 64 to the communication I / F 62E2.

[0099] In the processing circuit 62, as outputs of the digital image data 69B to the signal processing circuit 50, a first output via the communication line 53 and a second output via the communication line 55 are performed independently of each other under the control of the control circuit 62D. The first output and the second output are outputs of different output methods. That is, in the first output and the second output, the transmission path of the digital image data 69B transmitted until the digital image data 69B is output to the signal processing circuit 50 is different, and the connection standard between the imaging element 44 and the signal processing circuit 50 is also different.

[0100] The first output refers to the output of the digital image data 69B to the signal processing circuit 50 via the first transmission path. The first transmission path refers to the path through which the digital image data 69B is transmitted in the order of the selector 62C, the communication I / F 62E3, and the signal processing circuit 50 without passing through the control circuit 62D. That is, the output method of the first output is an output method that outputs the digital image data 69B obtained from the A / D converter 62B1 without storing it in the memory 64. Note that the digital image data 69B transmitted through the first transmission path is an example of the "first image data obtained by imaging a subject" according to the technology of the present disclosure.

[0101] The second output refers to the output of the digital image data 69B to the signal processing circuit 50 via the second transmission path. The second transmission path refers to the path through which the digital image data 69B is transmitted in the order of the memory 64, the control circuit 62D, the communication I / F 62E2, and the signal processing circuit 50. That is, the output method of the second output is an output method that outputs the digital image data 69B read from the memory 64 by the control circuit 62D. Note that the digital image data 69B transmitted through the second transmission path is an example of the "second image data based on the image data stored in the storage unit" according to the technology of the present disclosure.

[0102] The first output is realized by using the communication I / F 62E3 and the communication line 55. That is, when the digital image data 69B is input from the selector 62C to the communication I / F 62E3, the input digital image data 69B is output to the signal processing circuit 50 via the communication line 55.

[0103] The second output is realized by using the communication I / F 62E2 and the communication line 53. That is, when the digital image data 69B is input from the control circuit 62D to the communication I / F 62E2, the input digital image data 69B is output to the signal processing circuit 50 via the communication line 53.

[0104] The signal processing circuit 50 performs various signal processes on the digital image data 69B input from the processing circuit 62 via the communication lines 53 and 55. The various signal processes include known signal processes such as, for example, demosaicing, digital downsampling, and digital addition processes.

[0105] The demosaicing process is a process of calculating all color information for each pixel from a mosaic image corresponding to the array of color filters. For example, in the case of an image sensor composed of an RGB three-color color filter, all color information of RGB is calculated for each pixel from the mosaic image composed of RGB. The digital downsampling process is a process of downsampling the pixels included in the digital image data 69B in line units. The line unit refers to, for example, a horizontal line unit and / or a vertical line unit. The digital addition process is a process of adding and averaging pixel values for a plurality of pixels included in the digital image data 69B, for example.

[0106] Note that the various signal processes also include other known signal processes. Examples of other known signal processes include white balance adjustment, sharpness adjustment, gamma correction, color space conversion process, and color difference correction.

[0107] As shown in FIG. 9 as an example, in the imaging device 44, processes including an imaging process and an output process are performed. In the imaging process, after the Nth (N is a natural number) exposure, the Nth readout, the Nth reset, the Nth digital signal processing, and the Nth storage are performed, the (N + 1)th exposure, the (N + 1)th readout, the (N + 1)th reset, and the (N + 1)th digital signal processing are performed. Then, after the output process is performed, N is incremented by 1, and the imaging process and the output process are repeated.

[0108] When the imaging process is started, the photoelectric conversion element 61 is reset by the readout circuit 62A, and the residual charges of each pixel in the photoelectric conversion element 61 are erased. The Nth exposure is performed by the photoelectric conversion element 61 between the previous reset of the photoelectric conversion element 61 by the readout circuit 62A and the Nth readout.

[0109] The Nth readout is performed by the readout circuit 62A when the Nth vertical synchronization signal is input to the readout circuit 62A. The Nth readout refers to the readout of the analog image data 69A performed by the readout circuit 62A in response to the input of the Nth vertical synchronization signal to the readout circuit 62A.

[0110] The Nth reset refers to the reset of the photoelectric conversion element 61 performed by the readout circuit 62A corresponding to the Nth readout. The Nth digital signal processing refers to the digital signal processing performed by the digital processing circuit 62B on the analog image data 69A obtained by the Nth readout. is performed.

[0111] The Nth storage refers to the storage of the digital image data 69B obtained by the Nth digital signal processing in the memory 64. The Nth storage is realized by using the selector 62C, the control circuit 62D, and the memory 64. That is, the digital image data 69B obtained by the Nth digital signal processing is input to the control circuit 62D via the selector 62C and stored in the memory 64 by the control circuit 62D.

[0112] The (N + 1)-th exposure is performed by the photoelectric conversion element 61 between the N-th reset and the (N + 1)-th readout.

[0113] The (N + 1)-th readout is performed by the readout circuit 62A when the (N + 1)-th vertical synchronization signal is input to the readout circuit 62A. The (N + 1)-th readout refers to the readout of the analog image data 69A performed by the readout circuit 62A in response to the input of the (N + 1)-th vertical synchronization signal to the readout circuit 62A.

[0114] The (N + 1)-th reset refers to the reset of the photoelectric conversion element 61 performed by the readout circuit 62A corresponding to the (N + 1)-th readout. The (N + 1)-th digital signal processing refers to the digital signal processing performed by the digital processing circuit 62B on the analog image data 69A obtained by the (N + 1)-th readout.

[0115] In the output process, the first output and the second output are performed in parallel. That is, the latest digital image data 69B is output to the signal processing circuit 50 via the first transmission path, and the digital image data 69B one frame before is output to the signal processing circuit 50 via the second transmission path.

[0116] Here, the latest digital image data 69B refers to the digital image data 69B obtained by the (N + 1)-th digital signal processing. Also, the digital image data 69B one frame before refers to the digital image data 69B stored in the memory 64 at the current time. The digital image data 69B stored in the memory 64 at the current time refers to the digital image data 69B obtained by the N-th digital signal processing, input to the control circuit 62D via the selector 62C, and stored in the memory 64 by the control circuit 62D.

[0117] In the imaging device 44, since the memory 64 is a DRAM, writing and reading cannot be performed simultaneously on the memory 64. Therefore, as an example, as shown in FIG. 10, the first output and the second output are performed during a period when writing to the memory 64 is not possible (the "writing impossible period" shown in FIG. 10). In other words, in the imaging device 44, using the writing impossible period, digital image data for two consecutive frames is output to the signal processing circuit 50 in parallel.

[0118] In the example shown in FIG. 10, the first output is performed in accordance with the horizontal synchronization signal input from the controller 46 via the communication I / F 62E. That is, the communication I / F 62E3 outputs the digital image data 69B input from the selector 62C (see FIG. 8) to the signal processing circuit 50 for each horizontal line in accordance with the horizontal synchronization signal input from the controller 46 via the communication I / F 62E1 and the control circuit 62D.

[0119] On the other hand, the second output is performed in parallel with the first output. That is, while the first output is being performed, the control circuit 62D acquires one frame of digital image data 69B obtained one frame before the digital image data 69B output from the communication I / F 62E3 from the memory 64 and outputs it to the communication I / F 62E2. The communication I / F 62E2 outputs the one frame of digital image data 69B input from the control circuit 62D to the signal processing circuit 50. from the input one frame of digital image data 69B to the signal processing circuit 50.

[0120] Next, the operation of the imaging apparatus 10 will be described.

[0121] First, the flow of the control process executed by the processing circuit 62 of the imaging device 44 will be described with reference to FIG. 11.

[0122] In the control process shown in FIG. 11, first, in step ST10, the control circuit 62D determines whether digital image data 69B is stored in the memory 64. In step ST10, if the digital image data 69B is stored in the memory 64, the determination is negative, and the control process proceeds to step ST22. In step ST10, if the digital image data 69B is not stored in the memory 64, the determination is positive, and the control process proceeds to step ST12.

[0123] In step ST12, the control circuit 62D determines whether the vertical synchronization signal from the controller 46 has been received by the communication I / F 62E1. In step ST12, if the vertical synchronization signal from the controller 46 has not been received by the communication I / F 62E1, the determination is negative, and the control process proceeds to step ST20. In step ST12, if the vertical synchronization signal from the controller 46 has been received by the communication I / F 62E1, the determination is positive, and the control process proceeds to step ST14.

[0124] In step ST14, the readout circuit 62A reads the analog image data 69A and resets the photoelectric conversion element 61, and then the control process proceeds to step ST16.

[0125] In step ST16, the digital processing circuit 62B performs digital signal processing on the analog image data 69A, and then the control process proceeds to step ST18.

[0126] The digital image data 69B obtained by performing digital signal processing on the analog image data 69A in step ST16 is output to the selector 62C, and the selector 62C transfers the digital image data 69B to the control circuit 62D.

[0127] In step ST18, the control circuit 62D stores the digital image data 69B in the memory 64, and then the control process proceeds to step ST20.

[0128] In step ST20, the control circuit 62D determines whether a condition for ending the control process (hereinafter referred to as the "control process end condition") is satisfied. As an example of the control process end condition, there is a condition that an instruction to end the control process has been received by the reception device 84 (see FIG. 5). In step ST20, if the control process end condition is not satisfied, the determination is negative and the control process proceeds to step ST10. In step ST20, if the control process end condition is satisfied, the determination is affirmative and the control process ends.

[0129] In step ST22, the control circuit 62D determines whether a vertical synchronization signal from the controller 46 has been received by the communication I / F 62E1. In step ST22, if the vertical synchronization signal from the controller 46 has not been received by the communication I / F 62E1, the determination is negative and the control process proceeds to step ST30. In step ST22, if the vertical synchronization signal from the controller 46 has been received by the communication I / F 62E1, the determination is affirmative and the control process proceeds to step ST24.

[0130] In step ST24, the readout circuit 62A reads the analog image data 69A and resets the photoelectric conversion element 61, and then the control process proceeds to step ST26.

[0131] In step ST26, the digital processing circuit 62B performs digital signal processing on the analog image data 69A, and then the control process proceeds to step ST28.

[0132] The digital image data 69B obtained by performing digital signal processing on the analog image data 69A in step ST26 is output to the selector 62C, and the selector 62C transfers the digital image data 69B to the communication I / F 62E3.

[0133] In step ST28, the processing circuit 62 performs the first output and the second output, and then proceeds to step ST30. The first output and the second output are outputs in different output formats. That is, the first output is an output performed using the first transmission line (see FIG. 9) and in accordance with the LVDS connection standard, and the second output is an output performed using the second transmission line (see FIG. 9) and in accordance with the PCI-e connection standard.

[0134] In step ST28, the communication I / F 62E3 outputs the digital image data 69B transferred from the selector 62C to the signal processing circuit 50 via the communication line 55 (the first output). On the other hand, in response to a request from the controller 46, the control circuit 62D acquires the digital image data 69B one frame before from the memory 64 and outputs it to the signal processing circuit 50 via the communication line 53 from the communication I / F 62E2 (the second output).

[0135] In step ST30, the control circuit 62D determines whether or not the control process end condition is satisfied. In step ST30, if the control process end condition is not satisfied, the determination is negative and the control process proceeds to step ST10. In step ST20, if the control process end condition is satisfied, the determination is positive and the control process ends.

[0136] When the digital image data 69B output from the communication I / F 62E3 to the signal processing circuit 50 via the communication line 55 is input to the signal processing circuit 50 by executing this control process, it is transferred to the controller 46. On the other hand, when the digital image data 69B output from the communication I / F 62E2 to the signal processing circuit 50 via the communication line 53 is also input to the signal processing circuit 50, it is transferred to the controller 46. In the controller 46, the digital image data 69B is input to the input I / F 46E, and an image based on the digital image data 69B is displayed on the display by the CPU 46A. Note that the image based on the digital image data 69B input to the input I / F 46E is an example of the "first image based on the first image data" and the "second image based on the second image data" according to the technology of the present disclosure.

[0137] In addition, the digital image data 69B input to the input I / F 46E is stored in the secondary storage device 80 by the CPU 46A, or stored in an external device such as a USB memory (not shown) and / or a memory card (not shown) via the external I / F 82.

[0138] As described above, in the imaging device 10, the digital image data 69B obtained by imaging a subject is output to the signal processing circuit 50 by the communication I / F 62E3. Also, the digital image data 69B stored in the memory 64 is output to the signal processing circuit 50 by the communication I / F 62E2. And the output method of the communication I / F 62E3 and the output method of the communication I / F 62E2 are different. That is, the latest digital image data 69B is output to the signal processing circuit 50 via the communication line 55 by the first transmission path (see FIG. 9), and the digital image data 69B one frame before is output to the signal processing circuit 50 via the communication line 53 by the second transmission path (see FIG. 9). Also, the communication I / F 62E3 and the signal processing circuit 50 are connected according to the LVDS connection standard, and the communication I / F 62E2 and the signal processing circuit 50 are connected according to the PCI-e connection standard. Therefore, according to the imaging device 10, compared with the case where the digital image data 69B is output from only a single communication I / F to the processing unit 45, it is possible to suppress the stagnation of the output of the digital image data 69B.

[0139] In the imaging device 10, the first output and the second output (see FIGS. 8 to 10) are performed independently of each other under the control of the control circuit 62D. Therefore, according to the imaging device 10, it is possible to freely change the timing at which the first output is performed and the timing at which the second output is performed.

[0140] In the imaging device 10, the second output is performed in response to a request from the controller 46. Therefore, according to the imaging device 10, it is possible to avoid the second output being performed even though the processing unit 45 is not in a state of accepting the second output.

[0141] In addition, in the imaging device 10, as the output method of the first output, an output method is adopted in which the digital image data 69B obtained from the A / D converter 62B1 is output without being stored in the memory 64. Further, as the output method of the second output, an output method is adopted in which the digital image data 69B read from the memory 64 by the control circuit 62D is output. That is, even when the second output cannot be performed, the first output is continuously performed. Therefore, according to the imaging device 10, even during the period when the digital image data 69B is written into the memory 64, the output from the imaging element 44 to the signal processing circuit 50 can be continued.

[0142] In addition, in the imaging device 10, as the memory 64, a memory with different write timing and read timing is adopted. In the imaging device 10, the first output and the second output are performed at a timing that avoids the write timing for the memory 64. Therefore, according to the imaging device 10, even if the memory 64 is a memory with different write timing and read timing, the output of the digital image data 69B from the imaging element 44 to the processing unit 45 can be continued.

[0143] In addition, in the imaging device 10, as the memory 64, a DRAM is adopted. In the imaging device 10, the first output and the second output are performed at a timing that avoids the write timing for the DRAM. Therefore, according to the imaging device 10, even if the memory 64 is a DRAM, the output of the digital image data 69B from the imaging element 44 to the processing unit 45 can be continued.

[0144] In addition, in the imaging device 10, as the imaging element 44, an imaging element in which a photoelectric conversion element 61, a processing circuit 62, and a memory 64 are integrated into one chip is adopted. Thereby, compared with an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are not integrated into one chip, the portability of the imaging element 44 is enhanced. Also, compared with an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are not integrated into one chip, the degree of freedom in design can be increased. Furthermore, compared with an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are not integrated into one chip, it can contribute to the miniaturization of the imaging device main body 12.

[0145] Also, as shown in FIG. 7, as the imaging element 44, a stacked imaging element in which a memory 64 is stacked on a photoelectric conversion element 61 is adopted. Thereby, since the wiring connecting the photoelectric conversion element 61 and the memory 64 can be shortened, wiring delay can be reduced. As a result, compared with the case where the photoelectric conversion element 61 and the memory 64 are not stacked, the transfer speed of the image data 69 from the photoelectric conversion element 61 to the memory 64 can be increased. The improvement in the transfer speed also contributes to the speeding up of the processing in the entire processing circuit 62. Also, compared with the case where the photoelectric conversion element 61 and the memory 64 are not stacked, the degree of freedom in design can be increased. Furthermore, compared with the case where the photoelectric conversion element 61 and the memory 64 are not stacked, it can contribute to the miniaturization of the imaging device main body 12.

[0146] In addition, in the imaging device 10, a live view image or the like based on the digital image data 69B is displayed on the second display 86. Thereby, the user can visually recognize the image indicated by the digital image data 69B.

[0147] Furthermore, in the imaging device 10, the latest digital image data 69B output from the communication I / F 62E2 to the signal processing circuit 50 is stored in the secondary storage device 80, USB memory, and / or memory card, etc. by the CPU 46A. Also, the digital image data 69B from one frame before output from the communication I / F 62E3 to the signal processing circuit 50 is also stored in the secondary storage device 80, USB memory, and / or memory card, etc. by the CPU 46A. As a result, it is possible to manage without omission all the digital image data 69B for all frames obtained by imaging the subject.

[0148] Note that in the above first embodiment, the case where the second output is performed in response to a request from the controller 46 has been described, but the technology of the present disclosure is not limited thereto. For example, the second output may be started on the condition that the transfer of the digital image data 69B obtained by the above-described "N+1th digital signal processing" to the communication I / F 62E3 by the selector 62C is started. Also, the second output may be started on the condition that the output of the digital image data 69B obtained by the above-described "N+1th digital signal processing" to the selector 62C by the digital processing circuit 62B is started. In any case, the second output may be started during a period when writing to the memory 64 is not performed.

[0149] In the first embodiment described above, an example has been described in which the digital image data 69B transferred from the selector 62C to the communication I / F 62E3 is output to the signal processing circuit 50 via the communication line 55. However, the technology of the present disclosure is not limited to this. For example, the image data obtained by performing some image processing on the digital image data 69B by an image processing circuit (not shown) between the selector 62C and the communication I / F 62E3 may be output to the signal processing circuit 50 by the communication I / F 62E3. Here, examples of the above image processing include known image processing such as thinning processing and addition processing. Note that the image data obtained by performing some image processing on the digital image data 69B by an image processing circuit between the selector 62C and the communication I / F 62E3 is an example of the "first image data" according to the technology of the present disclosure.

[0150] In the first embodiment described above, an example has been described in which the digital image data 69B stored in the memory 64 is output to the signal processing circuit 50 by the communication I / F 62E2. However, the technology of the present disclosure is not limited to this. For example, the image data obtained by performing the above image processing on the digital image data 69B stored in the memory 64 by the control circuit 62D may be output to the signal processing circuit 50 via the communication I / F 62E2. Note that the image data obtained by performing the above image processing on the digital image data 69B stored in the memory 64 by the control circuit 62D is an example of the "second image data" according to the technology of the present disclosure.

[0151] In the first embodiment described above, an example has been given in which each image based on the digital image data 69B output from each of the communication I / F 62E2 and the communication I / F 62E3 is displayed on the display by the CPU 46A. However, the technology of the present disclosure is not limited to this. For example, an image based on the digital image data 69B output from the communication I / F 62E2 or the communication I / F 62E3 to the signal processing circuit 50 may be displayed on the display by the CPU 46A.

[0152] In the first embodiment described above, the digital image data 69B output from each of the communication I / Fs 62E2 and 62E3 is stored in the secondary storage device 80 or the like by the CPU 46A. Although an example of the form in which this is stored has been given, the technology of the present disclosure is not limited to this. For example, the digital image data 69B output from the communication I / F 62E2 or the communication I / F 62E3 to the signal processing circuit 50 may be stored in the secondary storage device 80 or the like by the CPU 46A.

[0153] In the first embodiment described above, as the imaging device 44, an imaging device in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are integrated on one chip has been exemplified. However, the technology of the present disclosure is not limited to this. For example, at least the photoelectric conversion element 61 and the memory 64 among the photoelectric conversion element 61, the processing circuit 62, and the memory 64 may be integrated on one chip.

[0154] [Second Embodiment] In the first embodiment described above, an example in which the first output and the second output are performed in parallel has been described. In the second embodiment, an example in which the first output and the second output are performed alternately will be described. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted. Hereinafter, the differences from the first embodiment will be described.

[0155] The imaging device 10 according to the second embodiment is different from the first embodiment in that, as shown in FIG. 12 as an example, a communication line 59 branched from the communication line 57 is connected to the signal processing circuit 50. In the description of the second embodiment, for convenience of explanation, the imaging device 10 according to the second embodiment is simply referred to as the "imaging device 10".

[0156] Since the communication line 59 branched from the communication line 57 is connected to the signal processing circuit 50, the read synchronization signal output from the controller 46 is input to the signal processing circuit 50 via the communication line 59. Therefore, the signal processing circuit 50 can perform an operation according to the read synchronization signal input from the controller 46 via the communication line 59.

[0157] A vertical synchronization signal is input to the signal processing circuit 50 from the controller 46 via the communication line 59. The signal processing circuit 50 specifies the vertical blanking period according to the input timing of the vertical synchronization signal. Then, when entering the vertical blanking period, the signal processing circuit 50 generates an output request signal for requesting the start of the second output to the processing circuit 62, and outputs the generated output request signal to the communication I / F 62E2 via the communication line 53. The output request signal is transferred from the communication I / F 62E2 to the control circuit 62D. When the output request signal is transferred from the communication I / F 62E2 to the control circuit 62D, the second output is started. In the second output, as described in the first embodiment, the digital image data 69B is transmitted using the second transmission path.

[0158] That is, when the output request signal is transferred from the communication I / F 62E2, the control circuit 62D acquires the digital image data 69B from the memory 64, and outputs the acquired digital image data 69B to the communication I / F 62E2. The communication I / F 62E2 outputs the digital image data 69B input from the control circuit 62D to the signal processing circuit 50 via the communication line 53.

[0159] As an example, as shown in FIGS. 13 and 14, in the imaging device 44, imaging processing and output processing are performed. In the imaging processing shown in FIG. 13, similar to the first embodiment, exposure by the photoelectric conversion element 61, reading of the analog image data 69A, reset of the photoelectric conversion element 61, digital signal processing, and storage of the digital image data 69B in the memory 64 are performed.

[0160] As an example, as shown in FIG. 14, in the output processing, the first output and the second output are alternately performed. When the vertical synchronization signal is input from the controller 46 to the processing circuit 62, the first output described in the first embodiment is performed. Then, when an output request signal is input from the signal processing circuit 50 to the processing circuit 62 during the vertical blanking period, the second output described in the first embodiment is performed.

[0161] Incidentally, when a vertical synchronization signal is input from the controller 46 to the imaging device 44, the reading of the analog image data 69A for one frame from the photoelectric conversion element 61 is started. Then, the digital image data 69B obtained by performing digital signal processing on the analog image data 69A is transferred to the control circuit 62D by the selector 62C and stored in the memory 64 by the control circuit 62D. Since the memory 64 is a DRAM, the writing period to the memory 64 by the control circuit 62D does not allow reading from the memory 64 by the control circuit 62D.

[0162] Therefore, as an example, as shown in FIG. 15, during the writing period to the memory 64, a first output that does not rely on the control circuit 62D is performed. That is, the latest digital image data 69B for one frame is output from the digital processing circuit 62B to the selector 62C and transferred to the communication I / F 62E3 by the selector 62C. Then, the latest digital image data 69B for one frame is output to the signal processing circuit 50 via the communication line 55 by the communication I / F 62E3.

[0163] When the first output is completed, the vertical blanking period is entered. During the vertical blanking period, since writing to the memory 64 is not performed, it becomes possible to read from the memory 64 by the control circuit 62D.

[0164] Therefore, as an example, as shown in FIG. 15, during the vertical blanking period, that is, during the reading period from the memory 64, a second output involving reading of the digital image data 69B from the memory 64 by the control circuit 62D is performed. That is, the digital image data 69B for one frame obtained one frame before is read from the memory 64 by the control circuit 62D and transferred to the communication I / F 62E2. Then, the digital image data 69B for one frame read from the memory 64 is output to the signal processing circuit 50 via the communication line 53 by the communication I / F 62E2.

[0165] When the vertical synchronization signal is input to the imaging device 44, the first output and the second output are sequentially performed until the next vertical synchronization signal is input to the imaging device 44. As a result, as shown in FIG. 15 as an example, the first output and the second output are alternately performed. This means that the first output is performed during a period different from the period during which the second output is performed. That is, the second output is performed during the vertical blanking period before the first output is performed and during the vertical blanking period after the first output is performed. Note that the period different from the period during which the second output is performed is an example of the "period different from the output period of the second image data by the second output unit" according to the technology of the present disclosure.

[0166] Next, the operation of the imaging device 10 will be described.

[0167] First, the flow of the imaging process executed by the processing circuit 62 of the imaging device 44 will be described with reference to FIG. 16.

[0168] In the imaging process shown in FIG. 16, first, in step ST50, the control circuit 62D determines whether the vertical synchronization signal from the controller 46 has been received by the communication I / F 62E1. In step ST50, if the vertical synchronization signal from the controller 46 has not been received by the communication I / F 62E1, the determination is negative, and the imaging process proceeds to step ST58. In step ST50, if the vertical synchronization signal from the controller 46 has been received by the communication I / F 62E1, the determination is affirmative, and the imaging process proceeds to step ST52.

[0169] In step ST52, the readout circuit 62A reads out the analog image data 69A and resets the photoelectric conversion element 61, and then the imaging process proceeds to step ST54. After that, the imaging process proceeds to step ST54.

[0170] In step ST54, the digital processing circuit 62B performs digital signal processing on the analog image data 69A, and then the imaging process proceeds to step ST56.

[0171] In step ST54, the digital image data 69B obtained by performing digital signal processing on the analog image data 69A is output to the selector 62C, and the selector 62C transfers the digital image data 69B to the control circuit 62D.

[0172] In step ST56, the control circuit 62D stores the digital image data 69B in the memory 64, and then the imaging process proceeds to step ST58.

[0173] In step ST58, the control circuit 62D determines whether or not a condition for ending the imaging process (hereinafter referred to as the "imaging process end condition") is satisfied. As an example of the imaging process end condition, there is a condition that an instruction to end the imaging process has been received by the reception device 84 (see FIG. 5). In step ST58, if the imaging process end condition is not satisfied, the determination is negative and the imaging process proceeds to step ST50. In step ST58, if the imaging process end condition is satisfied, the determination is positive and the imaging process ends.

[0174] Next, the flow of the output process executed by the processing circuit 62 of the imaging device 44 will be described with reference to FIG. 17.

[0175] In the output process shown in FIG. 17, in step ST100, the control circuit 62D determines whether or not the vertical synchronization signal from the controller 46 has been received by the communication I / F 62E1. In step ST100, if the vertical synchronization signal from the controller 46 has not been received by the communication I / F 62E1, the determination is negative and the output process proceeds to step ST106. In step ST100, if the vertical synchronization signal from the controller 46 has been received by the communication I / F 62E1, the determination is positive and the output process proceeds to step ST102.

[0176] In step ST102, the control circuit 62D starts the first output by controlling the selector 62C and the communication I / F 62E3, and then the output process proceeds to step ST104. During the first output, the imaging process shown in FIG. 16 is executed, and the digital image data 69B is written to the memory 64.

[0177] In step ST104, the control circuit 62D determines whether the first output has ended. The end of the first output refers to the end of the output of the latest digital image data 69B for one frame. In step ST104, if the first output has not ended, the determination is negative and the determination in step ST104 is made again. In step ST104, if the first output has ended, the determination is affirmative and the output process proceeds to step ST106.

[0178] In step ST106, the control circuit 62D determines whether it has entered the vertical blanking period. In step ST106, if it has not entered the vertical blanking period, the determination is negative and the output process proceeds to step ST114. In step ST106, if it has entered the vertical blanking period, the determination is affirmative and the output process proceeds to step ST108.

[0179] When entering the vertical blanking period, an output request signal is output from the signal processing circuit 50 to the communication I / F 62E2 via the communication line 53.

[0180] Therefore, in step ST108, the control circuit 62D determines whether the output request signal has been received by the communication I / F 62E2. In step ST108, if the output request signal has not been received by the communication I / F 62E2, the determination is negative and the output process proceeds to step ST114. In step ST108, if the output request signal has been received by the communication I / F 62E2, the determination is affirmative and the output process proceeds to step ST110.

[0181] In step ST110, control circuit 62D starts the second output, and then the output process proceeds to step ST112. When the second output is started, digital image data 69B for one frame stored in memory 64 is read out and output to signal processing circuit 50 via communication line 53 by communication I / F 62E2.

[0182] In step ST112, control circuit 62D determines whether the second output has ended. The end of the second output refers to the end of the output of digital image data 69B for one frame stored in memory 64, that is, digital image data 69B for one frame obtained one frame before. In step ST112, if the second output has not ended, the determination is negative and the determination in step ST112 is made again. In step ST112, if the second output has ended, the determination is affirmative and the output process proceeds to step ST114.

[0183] In step ST114, control circuit 62D determines whether a condition for ending the output process (hereinafter referred to as the "output process end condition") is satisfied. An example of the output process end condition is a condition that an instruction to end the output process is received by reception device 84 (see FIG. 5). In step ST114, if the output process end condition is not satisfied, the determination is negative and the output process proceeds to step ST100. In step ST114, if the output process end condition is satisfied, the determination is affirmative and the output process ends.

[0184] As described above, in imaging device 10, the first output is performed during a period different from the period during which the second output is performed. Thereby, digital image data 69B can be output to signal processing circuit 50 without delay.

[0185] Also, in the imaging device 10, the second output is performed during the vertical blanking period before the first output is performed and during the vertical blanking period after the first output is performed. Thereby, it is possible to avoid the output from the imaging element 44 to the signal processing circuit 50 from being stagnated due to the writing operation of the digital image data 69B to the memory 64.

[0186] In addition, in the above second embodiment, a form example in which the second output is performed during both the vertical blanking period before the first output is performed and the vertical blanking period after the first output is performed has been described. However, the technology of the present disclosure is not limited to this. The second output may be performed during the vertical blanking period before the first output is performed or during the vertical blanking period after the first output is performed.

[0187] [Third Embodiment] In the above second embodiment, a form example in which the first output and the second output are alternately performed in response to the input of the vertical synchronization signal has been shown. In the third embodiment, a case where the first output and the second output are alternately performed in response to the input of the horizontal synchronization signal will be described. In the third embodiment, the same components as those in the second embodiment are denoted by the same reference numerals, and the description thereof is omitted. Hereinafter, the parts different from the second embodiment will be described.

[0188] In the imaging device 10 according to the third embodiment, as an example, imaging processing and output processing are performed as shown in FIGS. 18 and 19. In the imaging process shown in FIG. 18, exposure by the photoelectric conversion element 61, reading of the analog image data 69A, reset of the photoelectric conversion element 61, digital signal processing, and storage of the digital image data 69B in the memory 64 are performed.

[0189] As an example, as shown in FIG. 12, a horizontal synchronization signal is input to the signal processing circuit 50 from the controller 46 via the communication line 59. The signal processing circuit 50 specifies the horizontal blanking period according to the input timing of the horizontal synchronization signal. Then, when entering the horizontal blanking period, the signal processing circuit 50 generates an output request signal for requesting the start of the second output to the processing circuit 62, and outputs the generated output request signal to the communication I / F 62E2 via the communication line 53. The output request signal is transferred from the communication I / F 62E2 to the control circuit 62D. When the output request signal is transferred from the communication I / F 62E2 to the control circuit 62D, the second output is started. In the second output, as described in the first and second embodiments, the digital image data 69B is transmitted using the second transmission path.

[0190] As an example, as shown in FIGS. 18 and 19, in the imaging device 44, imaging processing and output processing are performed. In the imaging processing shown in FIG. 18, similar to the first embodiment, exposure by the photoelectric conversion element 61, reading of the analog image data 69A, reset of the photoelectric conversion element 61, digital signal processing, and storage of the digital image data 69B in the memory 64 are performed.

[0191] When a horizontal synchronization signal is input from the controller 46 to the imaging device 44, as an example, as shown in FIG. 18, reading of the analog image data 69A for one horizontal line and reset of the photoelectric conversion element 61 are performed, and digital signal processing is performed on the read analog image data 69A for one horizontal line. The digital image data 69B for one horizontal line obtained by performing digital signal processing on the analog image data 69A for one horizontal line is output from the digital processing circuit 62B to the selector 62C. Then, the digital image data 69B for one horizontal line is transferred from the selector 62C to the control circuit 62D and stored in the memory 64 by the control circuit 62D.

[0192] As an example, as shown in FIG. 19, in the output process, the first output and the second output are alternately performed. When the horizontal synchronization signal is input from the controller 46 to the processing circuit 62, the first output is performed. Then, when the output request signal is input from the signal processing circuit 50 to the processing circuit 62 during the horizontal blanking period, the second output is performed.

[0193] By the way, when the horizontal synchronization signal is input from the controller 46 to the imaging element 44, the reading of the analog image data 69A for one horizontal line from the photoelectric conversion element 61 is started. Then, the digital image data 69B for one horizontal line obtained by performing digital signal processing on the analog image data 69A for one horizontal line is transferred to the control circuit 62D by the selector 62C and stored in the memory 64 by the control circuit 62D. Since the memory 64 is a DRAM, the writing period of the memory 64 by the control circuit 62D does not allow reading from the memory 64 by the control circuit 62D.

[0194] Therefore, as an example, as shown in FIG. 20, during the writing period to the memory 64 (the "writing period" shown in FIG. 20), the first output that does not rely on the control circuit 62D is performed. That is, the latest digital image data 69B for one horizontal line is output from the digital processing circuit 62B to the selector 62C and transferred to the communication I / F 62E3 by the selector 62C. Then, the latest digital image data 69B for one horizontal line is output to the signal processing circuit 50 via the communication line 55 by the communication I / F 62E3.

[0195] When the first output is completed, the horizontal blanking period is entered. The horizontal blanking period is a non-writing period. The non-writing period refers to the period during which writing to the memory 64 is not performed. Thus, during the horizontal blanking period, since writing to the memory 64 is not performed, it becomes possible to read from the memory 64 by the control circuit 62D.

[0196] Therefore, as an example, as shown in FIG. 20, during the horizontal blanking period, that is, the read period from the memory 64, a second output accompanied by the reading of the digital image data 69B from the memory 64 by the control circuit 62D is performed. That is, the digital image data 69B for one horizontal line obtained one line before is read from the memory 64 by the control circuit 62D and transferred to the communication I / F 62E2. Then, the digital image data 69B for one horizontal line read from the memory 64 is output to the signal processing circuit 50 via the communication line 53 by the communication I / F 62E2.

[0197] When the horizontal synchronization signal is input to the imaging device 44, the first output and the second output are sequentially performed until the next horizontal synchronization signal is input to the imaging device 44. As a result, as shown in FIG. 20 as an example, the first output and the second output are alternately performed. This means that the first output is performed in a period different from the period in which the second output is performed. That is, the second output is performed during the horizontal blanking period before the first output is performed and during the horizontal blanking period after the first output is performed. Note that "DT" shown in FIG. 20 means the digital image data 69B for one line.

[0198] Next, the operation of the imaging device 10 will be described.

[0199] First, the flow of the imaging process executed by the processing circuit 62 of the imaging device 44 will be described with reference to FIG. 21.

[0200] In the imaging process shown in FIG. 21, first, in step ST200, the control circuit 62D determines whether the horizontal synchronization signal from the controller 46 has been received by the communication I / F 62E1. In step ST200, if the horizontal synchronization signal from the controller 46 has not been received by the communication I / F 62E1, the determination is negative and the imaging process proceeds to step ST208. In step ST200, if the horizontal synchronization signal from the controller 46 has been received by the communication I / F 62E1, the determination is positive and the imaging process proceeds to step ST202.

[0201] In step ST202, the reading circuit 62A reads the analog image data 69A and resets the photoelectric conversion element 61, and then the imaging process proceeds to step ST204.

[0202] In step ST204, the digital processing circuit 62B performs digital signal processing on the analog image data 69A, and then the control process proceeds to step ST206.

[0203] The digital image data 69B obtained by performing digital signal processing on the analog image data 69A in step ST204 is output to the selector 62C, and the selector 62C transfers the digital image data 69B to the control circuit 62D.

[0204] In step ST206, the control circuit 62D stores the digital image data 69B in the memory 64, and then the imaging process proceeds to step ST208.

[0205] In step ST208, the control circuit 62D determines whether the above-described imaging process end condition is satisfied. In step ST208, if the imaging process end condition is not satisfied, the determination is negative and the imaging process proceeds to step ST200. In step ST208 if the imaging process end condition is satisfied, the determination is affirmative and the imaging process ends.

[0206] Next, the flow of the output process executed by the processing circuit 62 of the imaging element 44 will be described with reference to FIG. 22.

[0207] In the output process shown in FIG. 22, at step ST250, the control circuit 62D determines whether the horizontal synchronization signal from the controller 46 has been received by the communication I / F 62E1. In step ST250, if the horizontal synchronization signal from the controller 46 has not been received by the communication I / F 62E1, the determination is negative and the output process proceeds to step ST256. In step ST250, if the horizontal synchronization signal from the controller 46 has been received by the communication I / F 62E1, the determination is positive and the output process proceeds to step ST252.

[0208] At step ST252, the control circuit 62D starts the first output by controlling the selector 62C and the communication I / F 62E3, and then the output process proceeds to step ST254. While the first output is being performed, the imaging process shown in FIG. 18 is being executed, and the digital image data 69B is being written to the memory 64.

[0209] At step ST254, the control circuit 62D determines whether the first output has ended. The end of the first output refers to the end of the output of the latest digital image data 69B for one horizontal line. In step ST254, if the first output has not ended, the determination is negative and the determination at step ST254 is made again. In step ST254, if the first output has ended, the determination is positive and the output process proceeds to step ST256.

[0210] At step ST256, the control circuit 62D determines whether the horizontal blanking period has started. In step ST256, if the horizontal blanking period has not started, the determination is negative and the output process proceeds to step ST264. In step ST256, if the horizontal blanking period has started, the determination is positive and the output process proceeds to step ST258.

[0211] When entering the horizontal blanking period, an output request signal is output from the signal processing circuit 50 to the communication I / F 62E2 via the communication line 53.

[0212] Therefore, in step ST258, the control circuit 62D determines whether an output request signal has been received by the communication I / F 62E2. In step ST258, if the output request signal has not been received by the communication I / F 62E2, the determination is negative, and the output process proceeds to step ST264. In step ST258, if the output request signal has been received by the communication I / F 62E2, the determination is positive, and the output process proceeds to step ST260.

[0213] In step ST260, the control circuit 62D starts the second output, and then the output process proceeds to step ST262. When the second output is started, the digital image data 69B for one frame stored in the memory 64 is read out and output to the signal processing circuit 50 via the communication line 53 by the communication I / F 62E2.

[0214] In step ST262, the control circuit 62D determines whether the second output has ended. The end of the second output refers to the end of the output of the digital image data 69B for one frame stored in the memory 64, that is, the digital image data 69B for one frame obtained one frame before. In step ST262, if the second output has not ended, the determination is negative, and the determination in step ST262 is made again. In step ST262, if the second output has ended, the determination is positive, and the output process proceeds to step ST264.

[0215] In step ST264, the control circuit 62D determines whether the above-described output process end condition is satisfied. In step ST264, if the output process end condition is not satisfied, the determination is negative, and the output process proceeds to step ST250. In step ST264, if the output process end condition is satisfied, the determination is positive, and the output process ends.

[0216] As described above, in the imaging device 10, the second output is performed during the horizontal blanking period before the first output is performed and during the horizontal blanking period after the first output is performed. Thereby, it is possible to avoid the output from the imaging element 44 to the signal processing circuit 50 from stagnating due to the writing operation of the digital image data 69B to the memory 64.

[0217] Note that, in the above-described third embodiment, an example has been described in which the second output is performed both during the horizontal blanking period before the first output is performed and during the horizontal blanking period after the first output is performed. However, the technology of the present disclosure is not limited to this. The second output may be performed during the horizontal blanking period before the first output is performed or during the horizontal blanking period after the first output is performed.

[0218] Also, in the above-described third embodiment, the case where the second output is performed during the horizontal blanking period has been described. However, the technology of the present disclosure is not limited to this. For example, as shown in FIG. 23A, the second output may be performed during the digital signal processing period before the first output is performed. Here, the digital signal processing period refers to the period during which digital signal processing is performed by the digital processing circuit 62B. The digital signal processing period is included in the non-writing period described above. Since writing to the memory 64 is not performed during the non-writing period, it is possible to read the digital image data 69B from the memory 64. That is, it is possible to transmit the digital image data 69B from the memory 64 to the signal processing circuit 50 using the second transmission path described above. As an example, as shown in FIG. 23A, by performing the second output during the digital signal processing period, it is possible to avoid the output from the imaging element 44 to the signal processing circuit 50 from stagnating due to the writing operation of the digital image data 69B to the memory 64.

[0219] Also, the digital signal processing period includes an A / D conversion period. The A / D conversion period refers to the period during which A / D conversion is performed by the A / D converter 62B1 (see FIG. 12). Thus, since the A / D conversion period is included in the digital signal processing period, as an example, as shown in FIG. 23B, the second output may be performed during the A / D conversion period. Thereby, it is possible to avoid the output from the imaging device 44 to the signal processing circuit 50 from stagnating due to the writing operation of the digital image data 69B to the memory 64.

[0220] Also, in each of the above embodiments, a form example in which the processing circuit 62 is realized by a device including an ASIC and an FPGA has been described, but the technology of the present disclosure is not limited thereto. For example, the above-described imaging process may be realized by a software configuration using a computer.

[0221] In this case, for example, as shown in FIG. 24, various programs for causing the computer 852 built into the imaging device 44 to execute the above-described control process, imaging process, and output process are stored in the storage medium 900.

[0222] The various programs refer to a control program 902, an imaging program 904, and an output program 906. The control program 902 is a program for causing the computer 852 to execute the above-described control process. The imaging program 904 is a program for causing the computer 852 to execute the above-described imaging process. The output program 906 is a program for causing the computer 852 to execute the above-described output process.

[0223] As an example, as shown in FIG. 24, the computer 852 includes a CPU 852A, a ROM 852B, and a RAM 852C. Various programs stored in the storage medium 900 are installed in the computer 852. The CPU 852A executes the above-described control process according to the control program 902. Further, the CPU 852A executes the above-described imaging process according to the imaging program 904. Furthermore, the CPU 852A executes the above-described output process according to the output program 906.

[0224] Here, a single CPU is exemplified as the CPU 852A, but the technology of the present disclosure is not limited thereto, and a plurality of CPUs may be employed instead of the CPU 852A. Note that the storage medium 900 is a non-transitory storage medium. Examples of the storage medium 900 include any portable storage medium such as an SSD or a USB memory.

[0225] In the example shown in FIG. 24, various programs are stored in the storage medium 900, but the technology of the present disclosure is not limited thereto. For example, various programs may be stored in the ROM 852B in advance, and the CPU 852A may read the various programs from the ROM 852B, expand them in the RAM 852C, and execute the expanded various programs.

[0226] Also, various programs may be stored in a storage unit of another computer or a server device connected to the computer 852 via a communication network (not shown), and the various programs may be downloaded to the computer 852 in response to a request from the imaging device 10. In this case, the downloaded various programs are executed by the CPU 852A of the computer 852.

[0227] Also, the computer 852 may be provided outside the imaging element 44. In this case, the computer 852 may control the processing circuit 62 according to various programs.

[0228] As hardware resources for executing the control processing, imaging processing, and output processing (hereinafter referred to as "various processes") described in each of the above embodiments, the following various processors can be used. As the processor, for example, as described above, a general-purpose processor such as a CPU that functions as a hardware resource for executing various processes by executing software, that is, a program, can be mentioned. Further, as the processor, for example, a dedicated electric circuit which is a processor having a circuit configuration specifically designed for executing specific processes such as an FPGA, a PLD, or an ASIC can be mentioned.

[0229] The hardware resources for executing various processes may be configured by one of these various processors, or may be configured by a combination of two or more processors of the same type or different types (for example, a combination of a plurality of FPGAs, or a combination of a CPU and an FPGA). Further, the hardware resources for executing various processes may be one processor.

[0230] As an example of configuration by one processor, first, as represented by computers such as clients and servers, one processor is configured by a combination of one or more CPUs and software, and this processor functions as a hardware resource for executing in-chip processing of an imaging element. Second, as represented by a SoC (System-on-a-chip), there is a form in which a processor that realizes the functions of the entire system including a plurality of hardware resources for executing various processes is used in one IC chip. Thus, the in-chip processing of the imaging element is realized as a hardware resource using one or more of the above various processors is.

[0231] Furthermore, as a more specific hardware structure of these various processors, an electric circuit combining circuit elements such as semiconductor elements can be used.

[0232] In addition, in each of the above embodiments, the imaging device 10 is exemplified by an interchangeable-lens camera, but the technology of the present disclosure is not limited thereto. For example, the technology of the present disclosure may be applied to the smart device 950 shown in FIG. 25. As an example, the smart device 950 shown in FIG. 25 is an example of an imaging device according to the technology of the present disclosure. The imaging element 44 described in the above embodiment is mounted on the smart device 950. Even with the smart device 950 configured as described above, the same operations and effects as those of the imaging device 10 described in each of the above embodiments can be obtained. Note that the technology of the present disclosure is applicable not only to the smart device 950 but also to a personal computer or a wearable terminal device.

[0233] In addition, in each of the above embodiments, the first display 32 and the second display 86 are exemplified, but the technology of the present disclosure is not limited thereto. For example, a separate display attached to the imaging device main body 12 may be used as the "display unit (display)" according to the technology of the present disclosure.

[0234] Also, the above various processes are merely examples. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed within the scope not departing from the gist.

[0235] The description content and the illustrated content shown above are detailed descriptions of the part related to the technology of the present disclosure and are merely examples of the technology of the present disclosure. For example, the descriptions regarding the above configuration, function, operation, and effect are descriptions regarding an example of the configuration, function, operation, and effect of the part related to the technology of the present disclosure. Therefore, it goes without saying that within the scope not departing from the gist of the technology of the present disclosure, unnecessary parts may be deleted, new elements may be added, or replacements may be made to the description content and the illustrated content shown above. In addition, in order to avoid complication and facilitate the understanding of the part related to the technology of the present disclosure, descriptions regarding common technical knowledge that does not particularly require explanation for implementing the technology of the present disclosure are omitted from the description content and the illustrated content shown above.

[0236] In this specification, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0237] All documents, patent applications, and technical standards described in this specification are incorporated by reference herein to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A processor; A memory built into or connected to the processor, the processor has a first communication interface and a second communication interface; the first communication interface outputs image data obtained by capturing an image of a subject as first image data to an external processor disposed outside the image capture element without storing the image data in the memory; The memory stores the image data; the second communication interface outputs the image data stored in the memory to the external processor as second image data; an output method of the first communication interface and an output method of the second communication interface are different from each other; The first image data and the second image data are output to the external processor during a period when data cannot be written to the memory. Image sensor.

2. The image sensor according to claim 1 , wherein the first image data is output by the first communication interface and the second image data is output by the second communication interface independently of each other.

3. 3. The image sensor according to claim 1, wherein the second communication interface outputs the second image data in response to a request from the external processor.

4. An A / D converter that converts analog image data into digital data, the processor has a memory controller that stores in the memory digital image data obtained by digitizing the analog image data by the A / D converter; The output method of the first communication interface is to an output method in which digital image data is output as the first image data without being stored in the memory, 4 . The image sensor according to claim 1 , wherein an output method of the second communication interface is an output method in which the digital image data read from the memory by the memory controller is output as the second image data.

5. 4. The image sensor according to claim 1, wherein the memory has a write timing and a read timing that are different from each other.

6. The image sensor according to claim 5 , wherein the memory is a DRAM.

7. 7. The image sensor according to claim 1, wherein at least the photoelectric conversion element and the memory are integrated into a single chip.

8. The imaging element according to claim 7 , wherein the imaging element is a stacked type imaging element in which the memory is stacked on the photoelectric conversion element.

9. The imaging element according to any one of claims 1 to 8, a display processor that controls a display to display at least one of a first image based on the first image data output by the first communication interface and a second image based on the second image data output by the second communication interface; An imaging device comprising:

10. The imaging element according to any one of claims 1 to 8, a storage processor that controls a storage device to store at least one of the first image data output by the first communication interface and the second image data output by the second communication interface; An imaging device comprising:

11. 1. A method for operating an imaging device comprising: a processor; and a memory included in or connected to the processor, the processor having a first communication interface and a second communication interface, the method comprising: the first communication interface outputs image data obtained by capturing an image of a subject as first image data to an external processor disposed outside the image capture element without storing the image data in the memory; The memory stores the image data; the second communication interface outputs the image data stored in the memory to the external processor as second image data; The first image data and the second image data are output to the external processor during a period when data cannot be written to the memory. How the image sensor works.

12. A program for causing a computer to function as a first communication interface and a second communication interface included in an imaging element having a processor and a memory built in or connected to the processor, the processor comprising: the first communication interface outputs image data obtained by capturing an image of a subject as first image data to an external processor disposed outside the image capture element without storing the image data in the memory; The memory stores the image data; the second communication interface outputs the image data stored in the memory to the external processor as second image data; The first image data and the second image data are output to the external processor during a period when data cannot be written to the memory. program.

Images