Processing device, imaging device, processing method, and processing program

Abstract

To provide a processing apparatus, an imaging apparatus, a processing method, and a processing program.SOLUTION: A digital camera 100 comprises: a system control unit 11 which processes an image pickup signal output from an image pickup device 5A including a pixel unit 30 in which pixels 31 for converting light to an electric signal and outputting the electric signal are arranged, and a peripheral circuit 60; and a memory 16. The system control unit 11 corrects the image pickup signal based on an operation state of the peripheral circuit 60.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a processing device, an imaging device, a processing method, and a processing program. [Background technology]

[0002] Patent Document 1 describes an image data correction device that includes an image sensor having an aperture pixel region and a light-shielding pixel region, a readout means for reading pixel data from the image sensor, a calculation means for calculating one-dimensional dark shading correction data corresponding to each column of the pixel data from the pixel data read out from the light-shielding pixel region of the image sensor by the readout means, and a correction means for correcting one-dimensional dark shading of the image sensor using the one-dimensional dark shading correction data calculated by the calculation means while the pixel data is being read out from the aperture pixel region of the image sensor by the readout means.

[0003] Patent Document 2 describes an imaging device that includes a pixel unit including multiple photoelectric conversion elements within one microlens, each having a transfer means for each of the photoelectric conversion elements, a row scanning circuit for selecting one row in order to transfer the pixel data of that row, a column scanning circuit for reading out the data of one row read out by the row scanning circuit column by column, a thinning control circuit for controlling the readout thinning rate of the column scanning circuit, correction value generation means for generating a dark shading correction value based on the readout data, storage means for storing the generated correction value, and correction means for correcting pixel signals based on the generated correction value, wherein the correction value generation means generates a correction value according to the readout thinning rate from data read out with a small amount of thinning or without thinning when reading out data from some of the photoelectric conversion elements corresponding to one microlens. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-336343 [Patent Document 2] Japanese Patent Application Laid-Open No. 2015-080114 Summary of the Invention [Means for solving the problem]

[0005] A processing device according to an embodiment of the technology of the present disclosure includes: a processor that processes an image pickup signal output from an image pickup element including a pixel unit in which pixels that convert light into an electrical signal and output the electrical signal are arranged, and peripheral circuits; At least a program executed by the processor is stored. a memory, wherein the processor The activation rate of the peripheral circuit is the percentage of active elements in the peripheral circuit that are in operation. Based on the operating state of the peripheral circuits , the above image signal It is something that corrects.

[0006] An imaging device according to one embodiment of the technique of the present disclosure includes the processing device and the imaging element.

[0007] A processing method according to one embodiment of the technology of the present disclosure is a processing method for processing an image pickup signal output from an image pickup element including a pixel unit in which pixels that convert light into an electrical signal and output the electrical signal are arranged, and a peripheral circuit, The activation rate of the peripheral circuit is the ratio of active elements in the peripheral circuit that are in operation. Based on the operating state of the peripheral circuits , the above image signal It is something that corrects.

[0008] A processing program according to one embodiment of the technology of the present disclosure is a processing program for processing an image pickup signal output from an image pickup element including a pixel unit in which pixels that convert light into an electrical signal and output the electrical signal are arranged, and a peripheral circuit, an activation rate of the peripheral circuit, which is the rate of active elements in the peripheral circuit that are operating; Based on the operating state of the peripheral circuits , the above image signal The processor is caused to execute the step of correcting. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing a schematic configuration of a digital camera 100 which is an embodiment of an imaging apparatus of the present invention. [Figure 2] FIG. 2 is a schematic diagram showing the general configuration of an imaging element 5A. [Figure 3] FIG. 2 is a schematic diagram showing a general configuration of an imaging element 5B. [Figure 4] FIG. 2 is a schematic diagram showing the general configuration of an imaging element 5C. [Figure 5] FIG. 2 is a schematic diagram showing the general configuration of an image sensor 5D. [Figure 6] 10 is a schematic diagram for explaining horizontal shading of pixel signals output from pixel rows of an image sensor 5A. FIG. [Figure 7] 10 is a schematic diagram for explaining shading of pixel signals output from an imaging element 5B. FIG. [Figure 8] 10 is a schematic diagram for explaining the dark output level of a pixel signal output from an image sensor 5D. FIG. [Figure 9] 10 is a schematic diagram for explaining a change in the reference potential when the imaging element 5A is in operation. FIG. [Figure 10] 1 shows the appearance of a smartphone 200. [Figure 11] 11 is a block diagram showing the configuration of the smartphone 200 shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] FIG. 1 is a diagram showing a schematic configuration of a digital camera 100, which is an embodiment of an imaging device of the present invention.

[0011] 1 includes a lens device 40 having an imaging lens 1, an aperture 2, a lens control unit 4, a lens driver 8, and an aperture driver 9, and a main body 100A. The main body 100A includes an imaging element 5, a system control unit 11, an operation unit 14, a display device 22, a memory 16 including RAM (Random Access Memory) and ROM (Read Only Memory), a memory control unit 15 that controls the recording of data to the memory 16 and the reading of data from the memory 16, a digital signal processing unit 17, and an external memory control unit 20 that controls the recording of data to a recording medium 21 and the reading of data from the recording medium 21. The system control unit 11 and the memory 16 form a processing device.

[0012] The lens device 40 may be detachable from the main body 100A, or may be integrated with the main body 100A. The imaging lens 1 includes a focus lens that is movable in the optical axis direction.

[0013] The lens control unit 4 of the lens device 40 is configured to be able to communicate with the system control unit 11 of the main body 100A via a wired or wireless connection. In accordance with commands from the system control unit 11, the lens control unit 4 controls the focus lens included in the imaging lens 1 via the lens driving unit 8 to change the position of the principal point of the focus lens, and controls the aperture value of the aperture 2 via the aperture driving unit 9.

[0014] The image sensor 5 has an imaging surface on which a plurality of pixels are arranged two-dimensionally, and converts a subject image formed on this imaging surface by an imaging optical system into pixel signals using the plurality of pixels and outputs the pixel signals. A CMOS (complementary metal-oxide semiconductor) image sensor is preferably used as the image sensor 5. In the following description, the image sensor 5 is described as a CMOS image sensor. The collection of pixel signals output from the image sensor 5 is referred to as an imaging signal.

[0015] A system control unit 11 that controls the entire electrical control system of the digital camera 100 drives the image sensor 5 to output an image of a subject captured through the imaging optical system of the lens device 40 as an imaging signal.

[0016] An instruction signal from the user is input to the system control unit 11 through the operation unit 14. The operation unit 14 includes a touch panel integrated with the display surface 22b, various buttons, and the like.

[0017] The system control unit 11 controls the entire digital camera 100, and its hardware configuration consists of various processors that execute programs, including a processing program, to perform processing. The programs executed by the system control unit 11 are stored in the ROM of the memory 16.

[0018] The various types of processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs to perform various processes, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), or a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for performing specific processes. More specifically, the structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0019] The system control unit 11 may be configured with one of various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0020] The display device 22 includes a display surface 22b configured by an organic EL (electroluminescence) panel, a liquid crystal panel, or the like, and a display controller 22a that controls the display on the display surface 22b.

[0021] The memory control unit 15 , digital signal processing unit 17 , external memory control unit 20 , and display controller 22 a are interconnected by a control bus 24 and a data bus 25 , and are controlled by commands from the system control unit 11 .

[0022] Next, a description will be given of an example configuration of the imaging element 5. Examples of the configuration of the imaging element 5 include imaging element 5A, imaging element 5B, imaging element 5C, and imaging element 5D, which will be described below. Fig. 2 is a schematic diagram showing the general configuration of imaging element 5A.

[0023] The image sensor 5A shown in FIG. 2 includes a pixel section 30, a peripheral circuit 60, and a drive circuit (not shown) that drives the pixel section 30.

[0024] The pixel section 30 is a region in which pixels 31, which convert light into electrical signals and output them, are arranged two-dimensionally in a V direction (vertical direction) and an H direction (horizontal direction) perpendicular to the V direction. In the pixel section 30, a plurality of pixel columns 31C, each consisting of a plurality of pixels 31 lined up in the V direction, are arranged in the H direction. The pixel section 30 can also be said to be an arrangement in the V direction of a plurality of pixel rows, each of which is a collection of a plurality of pixels 31 lined up in the H direction.

[0025] The peripheral circuit 60 includes a processing circuit group 70, a timing generator (TG) 80, and a digital gain circuit 90.

[0026] The processing circuit group 70 has processing circuits 71 provided corresponding to each pixel column 31C of the pixel unit 30. The processing circuits 71 included in the processing circuit group 70 are arranged side by side in the H direction. The processing circuits 71 include a CDS circuit 72 that performs CDS (Correlated Double Sampling) processing on analog pixel signals output from the pixels 31 in the pixel column 31C, and an ADC (Analog-to-Digital Converter) circuit 73 that converts the pixel signals processed by the CDS circuit 72 into digital signals and outputs the digital signals.

[0027] The digital gain circuit 90 multiplies the pixel signal output from the ADC 73 by a gain according to the imaging sensitivity, etc., and outputs the result. The pixel signal output from the digital gain circuit 90 is processed by the digital signal processing unit 17, thereby generating image data suitable for display or storage.

[0028] The TG 80 generates timing signals necessary for the operation of the drive circuit, processing circuit 71, and digital gain circuit 90 included in the image sensor 5A, and supplies these to them.

[0029] The processing circuit group 70 is disposed adjacent to one side (bottom side in the figure) of the pixel unit 30 in the V direction. The TG 80 and the digital gain circuit 90 are disposed near one end of the processing circuit group 70 in the H direction.

[0030] Fig. 3 is a schematic diagram showing the general configuration of an image sensor 5B. The image sensor 5B has the same configuration as the image sensor 5A in Fig. 2, except that the positions of the TG80 and the digital gain circuit 90 in the peripheral circuit 60 have been changed. In the image sensor 5B, the TG80 and the digital gain circuit 90 are stacked with the pixel unit 30 in directions perpendicular to the V direction and H direction, and are disposed on the back surface of the pixel unit 30.

[0031] FIG. 4 is a schematic diagram showing the overall configuration of an image sensor 5C. The image sensor 5C has the same configuration as the image sensor 5A of FIG. 2, except that a processing circuit group 70A and a digital gain circuit 90A are further added as peripheral circuits 60. The processing circuit group 70A has the same configuration as the processing circuit group 70. The digital gain circuit 90A has the same configuration as the digital gain circuit 90. The processing circuit group 70A and the digital gain circuit 90A are arranged adjacent to the other side of the pixel unit 30 in the V direction. In the image sensor 5C, pixel signals output from, for example, half of the pixels 31 included in a pixel column 31C are processed by the processing circuit group 70, and pixel signals output from the remaining half of the pixels 31 included in the pixel column 31C are processed by the processing circuit group 70A.

[0032] FIG. 5 is a schematic diagram showing the overall configuration of an image sensor 5D. The image sensor 5B has the same configuration as the image sensor 5A of FIG. 2, except that a memory circuit 32 is added as one of the components of the peripheral circuit 60. The memory circuit 32 has a memory element that stores pixel signals output from each pixel 31 of the pixel section 30. The memory circuit 32 is configured, for example, with a DRAM (Dynamic RAM) or the like. The memory circuit 32 is stacked with the pixel section 30 in a direction perpendicular to the V direction and H direction and is disposed on the back surface of the pixel section 30. In the image sensor 5D, pixel signals output from the pixels 31 and stored in the memory circuit 32 are processed by a processing circuit group 70. Note that when the memory circuit 32 is not operating, the pixel signals output from the pixels 31 are processed by the processing circuit 71 without passing through the memory circuit 32.

[0033] In the image sensor 5A, the processing circuit 71 located near the TG80 and the digital gain circuit 90 is affected differently by the magnetic flux generated when the TG80 and the digital gain circuit 90 are operating, compared to the processing circuit 71 located away from the TG80 and the digital gain circuit 90. Specifically, the processing circuit 71 located near the TG80 and the digital gain circuit 90 is affected relatively strongly by the magnetic flux, and this magnetic flux causes relatively large noise to be mixed into the signals output from one or both of the CDS circuit 72 and the ADC 73. On the other hand, the processing circuit 71 located far from the TG80 and the digital gain circuit 90 is less affected by the magnetic flux, and therefore the noise mixed into the signals output from one or both of the CDS circuit 72 and the ADC 73 is smaller.

[0034] Therefore, in the image sensor 5A, the pixel signals output from each pixel row in the dark have a higher dark output level for pixels 31 located closer to the TG80 and the digital gain circuit 90, causing horizontal shading due to the magnetic flux.

[0035] In this specification, the activation rate is defined as the ratio of the number of active elements (transistors, etc.) that are operating to the total number of active elements included in each circuit (TG80, digital gain circuit 90, processing circuit 71, memory circuit 32, etc.) included in the peripheral circuit 60. The noise that can be generated by the magnetic flux increases as the activation rates of the TG80 and digital gain circuit 90 increase. In other words, when the operating state (specifically, the activation rate) of the TG80 and digital gain circuit 90 changes, the shape of the horizontal shading described above also changes.

[0036] FIG. 6 is a schematic diagram illustrating horizontal shading of pixel signals output from pixel rows of the image sensor 5A. The horizontal axis of FIG. 6 represents the position of each pixel 31 in the pixel row in the H direction. The vertical axis of FIG. 6 represents the level of the pixel signal output from the unexposed pixel 31 and processed by the processing circuit 71 (so-called dark output level). The pixel signal group OP1 shown in FIG. 6 represents an example where the activation rate of one or both of the TG80 and the digital gain circuit 90 is equal to or less than the first threshold. The pixel signal group OP2 represents an example where the activation rate of one or both of the TG80 and the digital gain circuit 90 exceeds the first threshold. As shown in FIG. 6, when the activation rates of the TG80 and the digital gain circuit 90 are high, the dark output level of the pixel 31 located near the TG80 and the digital gain circuit 90 also increases.

[0037] The activation rates of the TG80 and the digital gain circuit 90 correlate with the imaging mode (whether the pixels 31 of the pixel unit 30 are thinned out before reading or not), the operating frequency of the ADC 73 in the processing circuit 71, the digital conversion resolution set in the ADC 73, or the time required for digital conversion set in the ADC 73 (hereinafter referred to as the conversion speed). If the operating frequency, resolution, or conversion speed of the ADC 73 changes, the activation rate also changes. Therefore, if the operating state of the processing circuit 71 (operating frequency, resolution, conversion speed, or activation rate) changes, the shape of the horizontal shading described above will change.

[0038] The digital camera 100 can set a horizontal crop mode as an imaging mode. In the horizontal crop mode, pixel signals are read from the pixels 31 in the center of each pixel row in the pixel unit 30 in the H direction, and pixel signals are not read from the pixels 31 at both ends in the H direction. In this horizontal crop mode, the system control unit 11 controls the processing circuits 71 corresponding to pixel columns 31C from which pixel signals are not read to an inactive state. As a result, the activation rate of the digital gain circuit 90 downstream of the processing circuit 71 is reduced compared to the normal mode in which pixel signals are read from all pixels 31 in a pixel row. Therefore, in the normal mode, the shape of the horizontal shading becomes, for example, like the pixel signal group OP2 in FIG. 6, and in the horizontal crop mode, the shape of the horizontal shading becomes, for example, like the pixel signal group OP1 in FIG. 6. In this way, when the operating state of the processing circuit group 70 (the position of the operating processing circuit 71) changes, the shape of the horizontal shading described above also changes.

[0039] In the image sensor 5C shown in FIG. 4, horizontal shading occurs, similar to that of the image sensor 5A, whether only one of the processing circuit groups 70 and 70A is operating or both are operating. When both the processing circuit groups 70 and 70A are operating, the activation rate of TG80, for example, increases, resulting in a horizontal shading shape similar to that of the pixel signal group OP2 in FIG. 6. On the other hand, when only one of the processing circuit groups 70 and 70A is operating, the activation rate of TG80, for example, decreases, resulting in a horizontal shading shape similar to that of the pixel signal group OP1 in FIG. 6. In other words, in the image sensor 5C, if the number of operating processing circuits in the processing circuit groups 70 and 70A changes, the shape of the horizontal shading described above also changes.

[0040] In the image sensor 5B shown in Fig. 3, the pixels 31 arranged near the TG80 and the digital gain circuit 90 and the pixels 31 arranged away from the TG80 and the digital gain circuit 90 are affected differently by the magnetic flux generated when the TG80 and the digital gain circuit 90 are in operation. Specifically, the pixels 31 arranged near the TG80 and the digital gain circuit 90 are affected relatively strongly by the magnetic flux, and the effect of this magnetic flux causes relatively large noise to be mixed into the pixel signals output from the pixels 31. On the other hand, the pixels 31 arranged away from the TG80 and the digital gain circuit 90 are affected relatively strongly by the magnetic flux. Pixel 31 In this case, the influence of this magnetic flux is small, so the amount of noise mixed into the pixel signal output from pixel 31 is small.

[0041] Therefore, the pixel signals output from each pixel row of the image sensor 5B have a higher dark output level for pixels 31 closer to the center in the H direction, and the pixel signals output from each pixel column of the image sensor 5B have a higher dark output level for pixels 31 closer to the center in the V direction. In other words, horizontal shading and vertical shading occur in the image sensor 5B due to the influence of magnetic flux from the TG80 and the digital gain circuit 90. The shapes of this horizontal shading and vertical shading change depending on the activation rates of the TG80 and the digital gain circuit 90, respectively.

[0042] Fig. 7 is a schematic diagram illustrating shading of pixel signals output from image sensor 5B. The horizontal axis of the graph shown in the upper part of Fig. 7 represents the position of each pixel 31 in the pixel row in the H direction. The horizontal axis of the graph shown in the lower part of Fig. 7 represents the position of each pixel 31 in the pixel column 31C in the V direction. The vertical axis of Fig. 7 represents the dark output level.

[0043] 7 shows an example where the activation rates of one or both of the TG80 and the digital gain circuit 90 are equal to or less than the first threshold. Pixel signal group OP4 shows an example where the activation rates of one or both of the TG80 and the digital gain circuit 90 exceed the first threshold. As shown in FIG. 7, in the image sensor 5B, when the activation rates of the TG80 and the digital gain circuit 90 are high, the dark output level also increases.

[0044] 5, the effect of the magnetic flux generated when the memory circuit 32 is operating differs between when the memory circuit 32 is operating and when the memory circuit 32 is not operating. Specifically, when the memory circuit 32 is operating, all pixels 31 are affected by the magnetic flux, and this magnetic flux influence causes noise to be mixed into the pixel signals output from the pixels 31. On the other hand, when the memory circuit 32 is not operating, there is no influence of the magnetic flux from the memory circuit 32, and therefore less noise is mixed into the pixel signals output from the pixels 31.

[0045] Fig. 8 is a schematic diagram illustrating the dark output level of pixel signals output from the image sensor 5D. The horizontal axis of the graph shown in the upper part of Fig. 8 represents the position of each pixel 31 in the pixel row in the H direction. The horizontal axis of the graph shown in the lower part of Fig. 8 represents the position of each pixel 31 in the pixel column 31C in the V direction. The vertical axis of Fig. 8 represents the dark output level.

[0046] The pixel signal group OP6 shown in Figure 8 shows an example when the memory circuit 32 is in an inactive state (in other words, a state in which the activation rate of the memory circuit 32 is below the threshold). The pixel signal group OP5 shown in Figure 8 shows an example when the memory circuit 32 is in an active state (in other words, a state in which the activation rate of the memory circuit 32 exceeds the threshold). In this way, in the image sensor 5D, the dark output level also varies depending on the active state of the memory circuit 32.

[0047] 9 is a schematic diagram illustrating changes in the reference potential when the image sensor 5A is in operation. The horizontal axis of each graph shown in FIG. 9 represents the position in the V direction of the image sensor 5A. The vertical axis of each graph shown in FIG. 9 represents the potential, with the minimum value being 0 V, for example. The thick solid line in each graph shown in FIG. 9 represents the reference potential of the pixel unit 30 and the processing circuit 71.

[0048] FIG. 9 shows timings T1 to T6 during driving of the image sensor 5A. Timing T1 represents the state before a drive current is supplied to the pixel unit 30 and processing circuit 71 of the image sensor 5A. Timing T2 represents the state immediately after a drive current is supplied to the pixel unit 30 and processing circuit 71 of the image sensor 5A. Timing T3 represents the state during readout of pixel signals, a predetermined time after timing T2. Timing T4 represents the state during readout of pixel signals, a predetermined time after timing T3. Timing T5 represents the state at the start of a vertical blanking interval after timing T4. Timing T6 represents the state during a vertical blanking interval, a predetermined time after timing T5. Note that the image sensor 5A is provided with a reference terminal connected to ground, and the distance between the pixel unit 30 and the reference terminal is greater than the distance between the processing circuit 71 and the reference terminal.

[0049] At timing T2, when the supply of drive current to the pixel unit 30 and the processing circuit 71 begins, the reference potential shifts to potential Va in the processing circuit 71 located close to the reference terminal. On the other hand, in the pixel unit 30 located far from the reference terminal, the reference potential shifts to potential Va in the areas close to the processing circuit 71, but the amount of shift in the reference potential decreases with increasing distance from the processing circuit 71. As time passes from timing T2 to timing T3, the amount of shift in the reference potential in the pixel unit 30 gradually increases, and at timing T4, the reference potential becomes potential Va throughout the entire pixel unit 30.

[0050] At timing T5, when the vertical blanking period starts, the drive current supplied to the pixel unit 30 and the processing circuit 71 becomes zero, and the reference potential of the processing circuit 71 drops to 0 V. Furthermore, the reference potential of the pixel unit 30 drops to a value close to 0 V in locations close to the processing circuit 71, but is close to potential Va in locations farther from the processing circuit 71. Then, as time passes, the reference potential of the pixel unit 30 drops as shown at timing T6, and then returns to the state shown at timing T1.

[0051] In this way, the reference potential of the pixel unit 30 fluctuates while the pixel unit 30 is operating. When the reference potential fluctuates from location to location in the pixel unit 30, this causes vertical shading. The shape of this vertical shading changes depending on the length of the vertical blanking period. In other words, in each of the image sensors 5A to 5D, the dark output level of the pixel 31 also fluctuates depending on the length of the blanking period of the peripheral circuit 60.

[0052] As described above, the shapes of the horizontal shading and vertical shading of each of the image sensors 5A to 5D change depending on the operating state of the peripheral circuit 60, and therefore it is preferable to change the correction data for correcting the horizontal shading and vertical shading in the dark in accordance with the operating state.

[0053] Specifically, the system control unit 11 changes the method of correction (correction to subtract the dark output level) of the pixel signal output from the image sensor 5 based on the operating state of the peripheral circuit 60, thereby correcting dark unevenness (horizontal shading, vertical shading, or both) of the image sensor signal regardless of the operating state of the peripheral circuit 60. This makes it possible to improve the quality of the image captured by the image sensor 5.

[0054] When the image sensor 5 is the image sensor 5A, if the activation rate of the TG80 or the activation rate of the digital gain circuit 90 is equal to or lower than a first threshold, the system control unit 11 uses first correction data to correct the horizontal shading of the pixel signal group OP1 as shown in FIG. 6, and if the activation rate of the TG80 or the activation rate of the digital gain circuit 90 exceeds the first threshold, the system control unit 11 uses second correction data different from the first correction data to correct the horizontal shading of the pixel signal group OP2 as shown in FIG. 6.

[0055] When the image sensor 5 is the image sensor 5C, the system control unit 11 corrects the horizontal shading of the pixel signal group OP1 as shown in FIG. 6 using the third correction data when one of the processing circuit group 70 and the processing circuit group 70A is operating, and corrects the horizontal shading of the pixel signal group OP2 as shown in FIG. 6 using the fourth correction data different from the third correction data when both the processing circuit group 70 and the processing circuit group 70A are operating.

[0056] When the imaging element 5 is the imaging element 5B, if the activation rate of the TG80 or the activation rate of the digital gain circuit 90 is equal to or lower than the first threshold, the system control unit 11 corrects the horizontal shading and vertical shading of the pixel signal group OP3 as shown in FIG. 7 using the fifth correction data, and when both the processing circuit group 70 and the processing circuit group 70A are operating, the system control unit 11 corrects the horizontal shading and vertical shading of the pixel signal group OP4 as shown in FIG. 7 using sixth correction data different from the fifth correction data.

[0057] When the imaging element 5 is the imaging element 5D, the system control unit 11 uses the seventh correction data to correct the pixel signal group OP5 shown in FIG. 8 to zero when the memory circuit 32 is operating, and uses eighth correction data different from the seventh correction data to correct the pixel signal group OP6 shown in FIG. 8 to zero when the memory circuit 32 is not operating.

[0058] For any of the image sensors 5A to 5D, a plurality of first to eighth correction data are prepared according to the length of the vertical blanking interval, and the system control unit 11 performs shading using one of the first to eighth correction data according to the length of the vertical blanking interval.

[0059] In this way, according to the digital camera 100, dark shading is corrected based on the operating state of the peripheral circuit 60, thereby improving the quality of captured images.

[0060] Next, the configuration of a smartphone, which is another embodiment of the imaging device of the present invention, will be described.

[0061] Fig. 10 shows the external appearance of smartphone 200. Smartphone 200 shown in Fig. 10 has a flat housing 201, and is provided on one surface of housing 201 with display panel 202 as a display unit and display input unit 204 which is an integrated unit of operation panel 203 as an input unit.

[0062] Such housing 201 also includes a speaker 205, a microphone 206, an operation unit 207, and a camera unit 208. The configuration of housing 201 is not limited to this, and for example, it is also possible to adopt a configuration in which the display unit and the input unit are independent, or a configuration having a foldable structure or a sliding mechanism.

[0063] FIG. 11 is a block diagram showing the configuration of the smartphone 200 shown in FIG.

[0064] As shown in FIG. 11, the main components of the smartphone include a wireless communication unit 210, a display input unit 204, a call unit 211, an operation unit 207, a camera unit 208, a memory unit 212, an external input / output unit 213, a GNSS (Global Navigation Satellite System) receiving unit 214, a motion sensor unit 215, a power supply unit 216, and a main control unit 220.

[0065] The smartphone 200 also has, as its main function, a wireless communication function for performing mobile wireless communication via a base station device BS (not shown) and a mobile communication network NW (not shown).

[0066] The wireless communication unit 210 performs wireless communication with a base station device BS accommodated in the mobile communication network NW in accordance with instructions from the main control unit 220. Using this wireless communication, various file data such as audio data and image data, e-mail data, etc. are sent and received, and web data, streaming data, etc. are received.

[0067] The display input unit 204 is a so-called touch panel that displays images (still images and moving images) or text information, etc. under the control of the main control unit 220 to visually convey information to the user and detects user operations on the displayed information, and is equipped with a display panel 202 and an operation panel 203.

[0068] The display panel 202 uses an LCD (Liquid Crystal Display), an OLED (Organic Electro-Luminescence Display), or the like as a display device.

[0069] The operation panel 203 is placed so that an image displayed on the display surface of the display panel 202 can be seen, and is a device that detects one or more coordinates operated by a user's finger or a stylus. When this device is operated by the user's finger or a stylus, a detection signal generated by the operation is output to the main control unit 220. Next, the main control unit 220 detects the operation position (coordinates) on the display panel 202 based on the received detection signal.

[0070] As shown in FIG. 11, a display panel 202 and an operation panel 203 of a smartphone 200 are integrated to form a display input unit 204, and the operation panel 203 is disposed so as to completely cover the display panel 202.

[0071] When such an arrangement is adopted, operation panel 203 may also have a function to detect user operations in areas outside display panel 202. In other words, operation panel 203 may have a detection area for the overlapping portion that overlaps display panel 202 (hereinafter referred to as a display area), and a detection area for the other outer edge portion that does not overlap display panel 202 (hereinafter referred to as a non-display area).

[0072] The size of the display area and the size of the display panel 202 may be completely the same, but they do not necessarily have to be the same. Also, the operation panel 203 may have two sensitive areas: an outer edge portion and an inner portion other than the outer edge portion. Furthermore, the width of the outer edge portion is designed appropriately depending on the size of the housing 201, etc.

[0073] Furthermore, the position detection method used in the operation panel 203 may be a matrix switch method, a resistive film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, a capacitance method, or the like, and any method may be used.

[0074] The call unit 211 is equipped with a speaker 205 or a microphone 206, and converts the user's voice input through the microphone 206 into voice data that can be processed by the main control unit 220 and outputs it to the main control unit 220, or decodes voice data received by the wireless communication unit 210 or the external input / output unit 213 and outputs it from the speaker 205.

[0075] Also, as shown in FIG. 10, for example, the speaker 205 can be mounted on the same surface as the display input unit 204, and the microphone 206 can be mounted on the side of the housing 201.

[0076] The operation unit 207 is a hardware key using a key switch or the like, and receives instructions from a user. For example, as shown in Fig. 10, the operation unit 207 is a push-button switch mounted on the side of the housing 201 of the smartphone 200, which turns on when pressed with a finger or the like, and turns off when the finger is released by the restoring force of a spring or the like.

[0077] The storage unit 212 stores the control program and control data of the main control unit 220, application software, address data associated with names or telephone numbers of communication partners, data of emails sent and received, web data downloaded by web browsing, downloaded content data, and also temporarily stores streaming data, etc. The storage unit 212 is also used for storing ... sent and received email data, web data downloaded by web browsing, downloaded content data, and temporarily stores streaming data, etc. The storage unit 212 is also used for storing the control program and control data of the main control unit 220, application software, address data associated with names or telephone numbers of communication partners, etc. The storage unit 212 is also used for temporarily storing streaming data, etc. The storage unit 212 is also used for temporarily storing streaming data, etc. for It is configured with an external storage unit 218 having slots.

[0078] The internal memory unit 217 and the external memory unit 218 constituting the memory unit 212 are realized using storage media such as a flash memory type, a hard disk type, a multimedia card micro type, a card-type memory (e.g., MicroSD (registered trademark) memory, etc.), a RAM (Random Access Memory), a ROM (Read Only Memory), etc.

[0079] The external input / output unit 213 is an interface with all external devices connected to the smartphone 200. It acts as an interface for other external devices. communication (e.g., Universal System It is for direct or indirect connection via a USB, IEEE1394, Bluetooth (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA) (registered trademark), UWB (Ultra Wideband) (registered trademark), ZigBee (registered trademark), etc.) or a network (e.g., Ethernet (registered trademark), wireless LAN (Local Area Network), etc.).

[0080] The external devices connected to the smartphone 200 include, for example, a wired / wireless headset, a wired / wireless external charger, a wired / wireless data port, a memory card connected via a card socket, a SIM (Subscriber Identity Module), and the like. Module ) / UIM(User Identity Module ) card, external audio / video devices connected via the audio / video I / O (Input / Output) terminal, external audio / video devices connected wirelessly, smartphones connected by wire or wirelessly, Wired / wireless connected personal computers, There are earphones etc.

[0081] The external input / output unit 213 can transmit data received from such external devices to each component inside the smartphone 200, or transmit data inside the smartphone 200 to external devices.

[0082] The GNSS receiver 214 receives GNSS signals transmitted from GNSS satellites ST1 to STn in accordance with instructions from the main controller 220, executes positioning calculation processing based on the received multiple GNSS signals, and detects a position consisting of the latitude, longitude, and altitude of the smartphone 200. When the GNSS receiver 214 can acquire position information from the wireless communication unit 210 or the external input / output unit 213 (for example, a wireless LAN), it can also detect the position using the position information.

[0083] The motion sensor unit 215 includes, for example, a three-axis acceleration sensor, and detects the physical movement of the smartphone 200 in accordance with instructions from the main control unit 220. By detecting the physical movement of the smartphone 200, the direction of movement or acceleration of the smartphone 200 is detected. The detection result is output to the main control unit 220.

[0084] The power supply unit 216 supplies power stored in a battery (not shown) to each unit of the smartphone 200 in accordance with instructions from the main control unit 220.

[0085] The main control unit 220 includes a microprocessor, operates according to the control program and control data stored in the storage unit 212, and controls all the units of the smartphone 200. The microprocessor of the main control unit 220 has the same functions as the system control unit 11. The main control unit 220 also has a mobile communication control function that controls all the units of the communication system to perform voice communication or data communication via the wireless communication unit 210, and an application processing function.

[0086] The application processing function is realized by the main control unit 220 operating in accordance with the application software stored in the storage unit 212. Examples of the application processing function include an infrared communication function that controls the external input / output unit 213 to perform data communication with a partner device, an email function that sends and receives emails, and a web browsing function that views web pages.

[0087] Furthermore, the main control unit 220 performs the following operations based on image data (still image or moving image data) such as received data or downloaded streaming data: image The image processing function is to display the image on the display input unit 204.

[0088] The image processing function refers to a function in which the main control unit 220 decodes the image data, performs image processing on the decoded result, and displays the image on the display input unit 204.

[0089] Furthermore, the main control unit 220 executes display control for the display panel 202 and operation detection control for detecting user operations via the operation unit 207 and the operation panel 203 .

[0090] By executing display control, the main control unit 220 displays software keys such as icons or scroll bars for starting application software, or displays a window for creating an e-mail.

[0091] The scroll bar refers to a software key for receiving an instruction to move the displayed portion of an image, such as a large image that cannot fit in the display area of ​​the display panel 202.

[0092] In addition, by executing operation detection control, the main control unit 220 detects user operations through the operation unit 207, accepts operations on the above icons and input of character strings into the input field of the above window through the operation panel 203, or accepts requests to scroll the displayed image through the scroll bar.

[0093] Furthermore, by executing operation detection control, the main control unit 220 determines whether the operation position on the operation panel 203 is an overlapping portion (display area) that overlaps the display panel 202 or an outer edge portion (non-display area) that does not overlap the display panel 202, and is equipped with a touch panel control function that controls the sensitive area of ​​the operation panel 203 or the display position of the software key.

[0094] Furthermore, the main control unit 220 can also detect a gesture operation on the operation panel 203 and execute a preset function in response to the detected gesture operation.

[0095] Gesture operation is not a simple touch operation as in the past, but rather an operation in which a trajectory is drawn with a finger or the like, multiple positions are specified simultaneously, or a combination of these is used to draw a trajectory for at least one of multiple positions.

[0096] The camera unit 208 includes the image sensor 5 and the digital signal processing unit 17 in the digital camera shown in FIG.

[0097] The image data generated by the camera unit 208 can be stored in the storage unit 212 or output via the external input / output unit 213 or the wireless communication unit 210 .

[0098] In the smartphone 200 shown in FIG. 10, the camera unit 208 is mounted on the same surface as the display input unit 204, but the mounting position of the camera unit 208 is not limited to this, and it may be mounted on the back surface of the display input unit 204.

[0099] Furthermore, the camera unit 208 can be used for various functions of the smartphone 200. For example, an image acquired by the camera unit 208 can be displayed on the display panel 202, or an image from the camera unit 208 can be used as one of the operation inputs for the operation panel 203.

[0100] Furthermore, when the GNSS receiver 214 detects a position, it can also detect the position by referring to an image from the camera unit 208. Furthermore, it can also refer to an image from the camera unit 208 to determine the optical axis direction of the camera unit 208 of the smartphone 200 or determine the current usage environment, without using a triaxial acceleration sensor or by using the image in combination with a triaxial acceleration sensor. Of course, the image from the camera unit 208 can also be used in application software.

[0101] In addition, image data of still or video images can be added with location information acquired by the GNSS receiving unit 214, audio information acquired by the microphone 206 (which may be converted to text information by the main control unit, etc.), posture information acquired by the motion sensor unit 215, etc., and stored in the memory unit 212, or output via the external input / output unit 213 or wireless communication unit 210.

[0102] Even in the smartphone 200 configured as described above, shading correction can be performed with high precision, thereby improving the quality of captured images.

[0103] As explained above, this specification describes at least the following items. Note that the elements in parentheses correspond to those in the above-described embodiments, but are not limited to these.

[0104] (1) a processor (system control unit 11) that processes an imaging signal output from an imaging element (imaging element 5 (5A, 5B, 5C, 5D)) that includes a pixel unit (pixel unit 30) in which pixels (pixels 31) that convert light into an electrical signal and output the signal, and a peripheral circuit (peripheral circuit 60); a memory (memory 16); The processor corrects the imaging signal based on the operating state of the peripheral circuit.

[0105] (2) The processing device according to (1), The processor changes the method of correcting the imaging signal based on the operating state of the peripheral circuit.

[0106] (3) The processing device according to (1) or (2), The processor changes correction data used to correct the imaging signal based on the operating state of the peripheral circuit.

[0107] (4) The processing device according to any one of (1) to (3), The operating state of the peripheral circuit is an activation rate of the peripheral circuit.

[0108] (5) The processing device according to any one of (1) to (4), The peripheral circuit is a processing device including a processing circuit (processing circuit 71) that processes the output signals of the pixels and outputs the digital image pickup signals.

[0109] (6) (5) The processing device according to the present invention, The peripheral circuit includes a processing circuit group (processing circuit group 70) in which a plurality of the processing circuits are arranged in one direction (H direction), The processor changes a correction method for the imaging signal based on a position of the processing circuit operating in the processing circuit group.

[0110] (7) The processing device according to (5) or (6), the peripheral circuit includes a plurality of processing circuit groups in which a plurality of the processing circuits are arranged in one direction; The processor changes the method of correcting the imaging signal based on the number of processing circuit groups that are in operation among the plurality of processing circuit groups (processing circuit groups 70, 70A).

[0111] (8) The processing device according to any one of (5) to (7), The processor changes a correction method for the imaging signal based on an operating frequency of the processing circuit.

[0112] (9) The processing device according to any one of (1) to (8), The peripheral circuit is a processing device including a conversion circuit (ADC 73) that converts analog output signals output from the pixels into digital signals.

[0113] (10) (9) The processing device according to (9), The operating state of the peripheral circuits is a processing device including an activation rate of a circuit (digital gain circuit 90) that processes the output of the conversion circuit.

[0114] (11) The processing device according to (9) or (10), The processor changes a correction method for the imaging signal based on a resolution set in the conversion circuit.

[0115] (12) The processing device according to any one of (9) to (11), The processor changes a correction method for the imaging signal based on a time required for digital conversion in the conversion circuit.

[0116] (13) The processing device according to any one of (1) to (12), The processor changes a method of correcting the imaging signal based on the length of a blanking period of the peripheral circuit.

[0117] (14) The processing device according to any one of (1) to (13), The peripheral circuit is a processing device including a memory circuit (memory circuit 32) that stores an output signal from the pixel unit.

[0118] (15) The processing device according to (14), The processor changes the method of correcting the imaging signal depending on whether the memory circuit is in operation or not.

[0119] (16) The processing device according to any one of (1) to (15), The correction is a dark unevenness correction.

[0120] (17) A processing device according to any one of (1) to (16), An imaging device (digital camera 100) including the imaging element.

[0121] (18) A processing method for processing an image pickup signal output from an image pickup element including a pixel section in which pixels that convert light into an electrical signal and output the electrical signal are arranged, and a peripheral circuit, A processing method for correcting the imaging signal based on the operating state of the peripheral circuit.

[0122] (19) A processing program for processing an image pickup signal output from an image pickup element including a pixel unit in which pixels that convert light into an electrical signal and output the electrical signal are arranged, and peripheral circuits, a processing program that causes a processor to execute a step of correcting the imaging signal based on the operating state of the peripheral circuit; [Explanation of symbols]

[0123] 1 Imaging lens OP1~OP6 pixel signal group T1~T6 timing 4 Lens control unit 5A, 5B, 5C, 5D, 5 Image sensor 8 Lens drive unit 9 aperture Drive unit 11 System control section 14,207 Operation section 15 Memory control unit 16 memory 17 Digital Signal Processing Unit 20 External memory control unit 21 Recording Media 22a Display Controller 22b Display surface 22 Display device 24 control bus 25 Data Bus 30 pixel section 31C pixel row 31 pixels 32 Memory circuit 40 Lens device 60 Peripheral Circuits 70A, 70 Processing circuit group 71 Processing circuit 72 CDS circuit 73 ADC 80TG 90A,90 Digital gain circuit 100A main body 100 digital cameras 200 smartphones 201 Case 202 Display Panel 203 Operation Panel 204 Display and input section 205 Speaker 206 Microphone 208 Camera Club 210 Radio Communication Department 211 Telephone section 212 Storage section 213 External input / output section 214 GNSS receiver 215 Motion sensor unit 216 Power supply section 217 Internal storage 218 External Memory Unit 220 Main control unit

Claims

1. a processor that processes an image signal output from an image sensor including a pixel unit in which pixels that convert light into an electrical signal and output the electrical signal are arranged, and peripheral circuits; a memory that stores at least a program executed by the processor; The processor corrects the imaging signal based on an operating state of the peripheral circuit, including an activation rate of the peripheral circuit, which is the rate of active elements in the peripheral circuit that are operating.

2. 2. The processing device according to claim 1, The processor changes a correction method for the imaging signal based on an operating state of the peripheral circuit.

3. 3. The processing device according to claim 1 or 2, The processor changes correction data used to correct the imaging signal based on the operating state of the peripheral circuit.

4. A processing device according to any one of claims 1 to 3, The peripheral circuit includes a processing circuit that processes the output signals of the pixels and outputs the digital image signal.

5. A processing device according to claim 4, the peripheral circuit includes a processing circuit group in which a plurality of the processing circuits are arranged in one direction; The processor changes a correction method for the imaging signal based on a position of the processing circuit operating in the processing circuit group.

6. A processing device according to claim 4 or 5, the peripheral circuit includes a plurality of processing circuit groups in which a plurality of the processing circuits are arranged in one direction; The processor changes a correction method for the imaging signal based on the number of processing circuit groups that are operating among the plurality of processing circuit groups.

7. A processing device according to any one of claims 4 to 6, The processor changes a correction method for the imaging signal based on an operating frequency of the processing circuit.

8. A processing device according to any one of claims 1 to 7, The peripheral circuit is a processing device including a conversion circuit that converts an analog output signal output from the pixel into a digital signal.

9. The processing device according to claim 8, The operating state of the peripheral circuits includes an activation rate of a circuit that performs processing on the output of the conversion circuit.

10. The processing device according to claim 8 or 9, The processor changes a correction method for the imaging signal based on a resolution set in the conversion circuit.

11. A processing device according to any one of claims 8 to 10, The processor changes a correction method for the imaging signal based on a time required for digital conversion in the conversion circuit.

12. A processing device according to any one of claims 1 to 11, The processor changes a correction method for the imaging signal based on the length of a blanking period of the peripheral circuit.

13. A processing device according to any one of claims 1 to 12, The peripheral circuit includes a memory circuit that stores an output signal from the pixel unit.

14. The processing device according to claim 13, The processor changes the correction method for the imaging signal depending on whether the memory circuit is in operation or not.

15. A processing device according to any one of claims 1 to 14, comprising: The processing device, wherein the correction is dark unevenness correction.

16. A processing device according to any one of claims 1 to 15, An imaging device comprising the imaging element.

17. A processing method for processing an image pickup signal output from an image pickup element including a pixel section in which pixels that convert light into an electrical signal and output the electrical signal are arranged, and a peripheral circuit, A processing method for correcting the imaging signal based on the operating state of the peripheral circuit, including an activation rate of the peripheral circuit, which is the ratio of active elements in the peripheral circuit that are operating.

18. A processing program for processing an image pickup signal output from an image pickup element including a pixel unit in which pixels that convert light into an electrical signal and output the electrical signal are arranged, and a peripheral circuit, A processing program that causes a processor to execute a step of correcting the imaging signal based on an operating state of the peripheral circuit, including an activation rate of the peripheral circuit, which is the ratio of active elements in the peripheral circuit that are operating.

Images