Detection device, camera device, detection method, and storage medium

Abstract

As one example, the present application provides a detection device, an image pickup device, a detection method, and a program capable of detecting a defective physical pixel from among a plurality of physical pixels possessed by an image sensor, from a plurality of images obtained by the image sensor before and after movement of the image sensor. The processor of the detection device performs the following: controls so that the image sensor moves to a position where a first physical pixel after movement is located adjacent to a second physical pixel different from a first physical pixel before movement, among the physical pixel arrangements before and after movement of the image sensor having a plurality of physical pixels arranged in a first direction and a second direction intersecting the first direction; causes the image sensor to perform imaging before and after movement; and detects a defective physical pixel from among the plurality of physical pixels based on a degree of difference in pixel value between image pixels corresponding to the second physical pixel before movement and the first physical pixel after movement among a plurality of image pixels included in each of the plurality of images obtained by the image sensor.

Description

Technical Field

[0001] The present invention relates to a detection device, a camera device, a detection method, and a storage medium. Background Technology

[0002] Patent Document 1 discloses a photodetector characterized by comprising a vertical CCD section, a charge accumulation gate section, a horizontal CCD section, and a floating diffusion amplifier. The vertical CCD section has a TDI (Transmit-to-Difference) function, which converts light into charge and transmits the generated charge in a vertical direction. The charge accumulation gate section accumulates the charge transmitted from the vertical CCD section for a duration equal to one pixel or multiple pixels, based on an external driving signal. The horizontal CCD section transmits the charge transmitted from the charge accumulation gate section in a horizontal direction. For the charge transmitted from the horizontal CCD section, the floating diffusion amplifier accumulates the charge for a duration equal to one pixel or multiple pixels, based on an external driving signal, and converts it into a voltage signal of intensity corresponding to the accumulated charge amount, which is then output.

[0003] Patent Document 2 discloses a solid-state imaging device that acquires image data without relative displacement of the imaging element relative to an image of a subject imaged by an optical system, and acquires one or more image data images by only relative displacement of a predetermined amount in the horizontal and vertical directions, and combines these images to create a single image data image. The solid-state imaging device is characterized by having multiple imaging modes with different numbers of image data images to be captured.

[0004] Patent Document 3 discloses a method for generating high-quality images for a camera. In this method, the original video signal output from a CCD (charge-coupled device) is shifted a predetermined distance along a predetermined direction via a displacement circuit component and stored in temporary memory. The video signal stored in the temporary memory is then synthesized, processed, and recorded on a recording medium or output to a video display device. In this method, the original video signal output from the CCD is shifted horizontally by distances equivalent to N / 2 pixel spacing, N pixel spacing, and 3N / 2 pixel spacing, and vertically by distances equivalent to N / 2 pixel spacing, N pixel spacing, and 3N / 2 pixel spacing. Then, this method generates a video signal matrix with a pixel count 2N times greater than that of the original video signal matrix.

[0005] Patent Document 4 discloses a digital image data recording apparatus. This apparatus comprises a recording area predetermined in a recording medium, where frame unit image data representing a frame of an image is recorded, and a recording time required to record the frame unit image data. The frame unit image data is then divided into two field unit image data segments and recorded in the recording area. The apparatus includes an imaging component, a segmentation component, a storage component, a readout component, and a recording control component. The imaging component includes a solid-state electronic imaging element that generates image data of n times (n is a positive integer greater than or equal to 2) the frame unit image data for a frame of an image, and outputs image data representing an image of a subject obtained by capturing the subject using the solid-state electronic imaging element. The segmentation component divides the image data output from the imaging component into 2n field unit image data segments, each representing a frame of an image. The storage component temporarily stores (2n-1) of the 2n field unit image data segments divided by the segmentation component. The readout component sequentially reads (2n-1) field unit image data stored in the storage component for each of the aforementioned field unit image data. The recording control component sequentially records one field unit image data (excluding the field unit image data stored in the storage component) from the 2n field unit image data divided by the segmentation component, along with the (2n-1) field unit image data read from the storage component by the readout component, in n recording areas on the recording medium over n times the recording time.

[0006] Patent Document 5 discloses a pixel defect correction device comprising multiple solid-state imaging elements, a sampling circuit, an extraction circuit, a first arithmetic processing circuit, a second arithmetic processing circuit, a third arithmetic processing circuit, a fourth arithmetic processing circuit, a logic product circuit, and a correction circuit. Among the multiple solid-state imaging elements, the second solid-state imaging element is positioned at a position offset by half a pixel relative to the first solid-state imaging element. The sampling circuit samples signals read from the multiple solid-state imaging elements. The extraction circuit extracts from the output of the sampling circuit the value of the first pixel of the first solid-state imaging element, the values ​​of the second and third pixels of the second solid-state imaging element adjacent to the first pixel by half a pixel, and the values ​​of the fourth and fifth pixels of the first solid-state imaging element adjacent to the first pixel by one pixel. The first arithmetic processing circuit calculates the difference between the value of the first pixel and the value of the fourth pixel and compares it with a first constant value. The second arithmetic processing circuit calculates the difference between the value of the first pixel and the value of the fifth pixel and compares it with a second constant value. The third arithmetic processing circuit calculates the difference between the value of the second pixel and the value of the fourth pixel, and compares it with the value of the difference between the value of the first pixel and the value of the fourth pixel multiplied by a third coefficient. The fourth arithmetic processing circuit calculates the difference between the value of the third pixel and the value of the fifth pixel, and compares it with the value of the difference between the value of the first pixel and the value of the fifth pixel multiplied by a fourth coefficient. The logic product circuit takes the logic product of the outputs of the first, second, third, and fourth arithmetic processing circuits. The correction circuit corrects the output of the sampling circuit using the output of the logic product circuit.

[0007] Previous technical documents

[0008] Patent documents

[0009] Patent Document 1: Japanese Patent Application Publication No. 2008-236156

[0010] Patent Document 2: Japanese Patent Application Publication No. 2006-279990

[0011] Patent Document 3: Japanese Patent Application Publication No. 2000-244937

[0012] Patent Document 4: Japanese Patent Application Publication No. 10-51736

[0013] Patent Document 5: Japanese Patent Application Publication No. 6-245149 Summary of the Invention

[0014] In one aspect, one embodiment of the technology of the present invention provides a detection apparatus, camera device, detection method, and procedure capable of detecting defective physical pixels from multiple physical pixels of an image sensor based on multiple images captured by the image sensor before and after movement of the image sensor.

[0015] means for solving technical problems

[0016] The first aspect of the present invention relates to a detection device comprising: a processor; and a memory connected to or built into the processor, the processor performing the following control: in a physical pixel configuration before and after a movement of an image sensor having a plurality of physical pixels arranged along a first direction and a second direction intersecting the first direction, the image sensor is moved to a position adjacent to a second physical pixel that is different from the first physical pixel before the movement of the moved first physical pixel; the image sensor is captured before and after the movement; and defective physical pixels are detected from the plurality of physical pixels based on the difference in pixel values ​​between the image pixels corresponding to the second physical pixel before the movement and the first physical pixel after the movement, respectively, among the plurality of image pixels included in each of the plurality of images captured by the image sensor.

[0017] The second aspect of the present invention is the detection device of the first aspect, wherein the processor performs the following: performing a process of storing a plurality of images obtained by an image sensor in memory; and detecting defective physical pixels based on the difference in pixel values ​​among the plurality of image pixels contained in each of the plurality of images stored in memory.

[0018] The third aspect of the technology of the present invention is the detection device involved in the first or second aspect, wherein when the spacing between a plurality of physical pixels in the first or second direction is set to 1, the processor controls the image sensor to move along the first or second direction by a movement amount greater than 1.

[0019] The fourth aspect of the technology of the present invention is a detection device involved in any of the first to third aspects, wherein when the spacing between a plurality of physical pixels in a first direction or a second direction is set to p, the natural number is set to n and the pure decimal is set to d, the processor controls the image sensor to move along the first direction or the second direction by a movement amount of (n+d)×p.

[0020] The fifth aspect of the present invention is a detection device involved in any one of the first to fourth aspects, wherein the processor corrects the pixel value of the first image pixel corresponding to the defective physical pixel based on the pixel value of the second image pixel adjacent to the first image pixel.

[0021] The sixth aspect of the technology of the present invention is a detection device involved in any one of the first to fifth aspects, wherein the processor outputs position information corresponding to the position of the defective physical pixel.

[0022] The seventh aspect of the present invention is a detection device involved in any one of the first to sixth aspects, wherein the processor detects defective physical pixels based on the difference between the pixel values ​​of image pixels whose pixel values ​​deviate from a predetermined range and the pixel values ​​of image pixels adjacent to the image pixels whose pixel values ​​deviate from a predetermined range.

[0023] The eighth aspect of the technology of the present invention is a detection device involved in any one of the first to seventh aspects, wherein the image sensor is a monochrome image sensor.

[0024] The ninth aspect of the present invention is the detection device involved in the eighth aspect, wherein when the spacing between a plurality of physical pixels in a first direction or a second direction is set to p, the processor controls the image sensor to move along the first direction or the second direction by a movement amount of 1.5×p.

[0025] The tenth aspect of the present invention is a detection device involved in any of the first to seventh aspects, wherein the image sensor is a color-type image sensor in which multiple physical pixels are arranged in a periodic substrate and different color filters are distributed. When the spacing between the multiple physical pixels in the first or second direction is set to p, the natural number is set to n, and the number of physical pixels in each period of the substrate arrangement is set to T, the processor controls the image sensor to move along the first or second direction by a movement amount of {(T-1)+0.5}×n×p.

[0026] The eleventh aspect of the present invention is the detection device involved in the tenth aspect, wherein the substrate arrangement is a Bayer arrangement and the movement amount is specified as 1.5×p.

[0027] The 12th aspect of the present invention is the detection device involved in the 10th aspect, wherein the substrate arrangement is an X-Trans (registered trademark) type arrangement and the movement amount is specified as 5.5×p.

[0028] The 13th aspect of the present invention relates to a detection device according to any one of the 1st to 7th aspects, wherein the image sensor is a color-type image sensor in which multiple physical pixels are arranged on a periodic basis and color filters of different colors are distributed, and the processor performs the following control: in the physical pixel configuration before and after the image sensor is moved, the image sensor is moved to a position where the third physical pixel after the movement overlaps with the fourth physical pixel which is distributed with a color filter of the same color as the third physical pixel before the movement; the image sensor is captured before and after the movement; and defective physical pixels are detected from multiple physical pixels based on the difference in pixel values ​​between the image pixels of the same color corresponding to the fourth physical pixel before the movement and the third physical pixel after the movement, respectively, in each of the multiple images captured by the image sensor.

[0029] The 14th aspect of the present invention relates to a detection device according to any one of the 1st to 7th and 13th aspects, wherein the image sensor is a color-type image sensor in which multiple physical pixels are arranged on a periodic basis and color filters of different colors are distributed. The processor performs the following control: in the physical pixel configuration before and after the image sensor is moved, the image sensor is moved to a position adjacent to a second physical pixel in which the moved first physical pixel is distributed with a color filter of the same color as the moved first physical pixel; the image sensor is captured before and after the movement; and defective physical pixels are detected from multiple physical pixels based on the difference in pixel values ​​between the image pixels of the same color corresponding to the moved second physical pixel and the moved first physical pixel in each of the multiple images captured by the image sensor.

[0030] The 15th aspect of the present invention is the detection device involved in the 14th aspect, wherein when the spacing between a plurality of physical pixels in a first direction or a second direction is set to p, m is set to a natural number greater than 2 and a pure decimal is set to d, the processor controls the image sensor to move along the first direction or the second direction by a movement amount of (m+d)×p.

[0031] The 16th aspect of the present invention is the detection device involved in the 15th aspect, wherein the movement amount is specified as 2.5×p.

[0032] The 17th aspect of the present invention is a detection device involved in any of the 13th to 16th aspects, wherein the processor performs super-resolution on multiple images to generate multiple monochrome super-resolution images for each color of the color filter, and synthesizes multiple monochrome super-resolution images.

[0033] The 18th aspect of the present invention is a camera device comprising: a detection device according to any one of the 1st to 17th aspects; an image sensor; and a moving mechanism for moving the image sensor along at least one of a first direction and a second direction.

[0034] The 19th aspect of the present invention is a detection method comprising the following steps: performing control such that, in the physical pixel configuration of an image sensor having a plurality of physical pixels arranged along a first direction and a second direction intersecting the first direction before and after movement, the image sensor is moved to a position adjacent to a second physical pixel that is different from the first physical pixel before movement, where the first physical pixel after movement is located; taking pictures with the image sensor before and after movement; and detecting defective physical pixels from the plurality of physical pixels based on the difference in pixel values ​​between the image pixels corresponding to the second physical pixel before movement and the first physical pixel after movement, respectively, among the plurality of image pixels included in each of the plurality of images obtained by the image sensor.

[0035] The 20th aspect of the present invention is a storage medium storing a program for causing a computer to perform a process including the following steps: performing control to move the image sensor to a position adjacent to a second physical pixel that is different from the first physical pixel before the movement, in the physical pixel configuration of an image sensor having a plurality of physical pixels arranged along a first direction and a second direction intersecting the first direction; taking pictures of the image sensor before and after the movement; and detecting defective physical pixels from the plurality of physical pixels based on the difference in pixel values ​​between the image pixels corresponding to the second physical pixel before the movement and the first physical pixel after the movement, respectively, in each of the plurality of images obtained by the image sensor. Attached Figure Description

[0036] Figure 1 This is a perspective view showing an example of the appearance of a camera device.

[0037] Figure 2 It means Figure 1 A rear view of an example of the appearance of the rear side of the camera device shown.

[0038] Figure 3 It means Figure 1 A schematic structural diagram of an example of the hardware structure of the camera device shown.

[0039] Figure 4 It means Figure 3 The diagram shows an example of the controller's structure.

[0040] Figure 5 It means Figure 4 The diagram shows an example of the main functions of a CPU.

[0041] Figure 6 This is a front view showing an example of the structure of the image sensor according to the first embodiment.

[0042] Figure 7A This is an explanation of the reason. Figure 6 The diagram shows a concept of an example of an image sensor capturing multiple images.

[0043] Figure 7B This is to explain from Figure 7A A conceptual diagram illustrating an example of how multiple images can be used to generate a super-resolution image.

[0044] Figure 7C This is an explanation based on Figure 7B A conceptual diagram illustrating an example of detecting defective physical pixels in a super-resolution image.

[0045] Figure 8 It means by Figure 5 The flowchart shown is an example of the camera processing flow executed by the CPU.

[0046] Figure 9 This is a front view showing an example of the structure of the image sensor according to the second embodiment.

[0047] Figure 10A This is an explanation of the reason. Figure 9 The diagram shows a concept of an example of an image sensor capturing multiple images.

[0048] Figure 10B This is to explain from Figure 10A A conceptual diagram illustrating an example of how multiple images can be used to generate a super-resolution image.

[0049] Figure 10C This is an explanation based on Figure 10B A conceptual diagram illustrating an example of detecting defective physical pixels in a super-resolution image.

[0050] Figure 11 This is a front view showing an example of the structure of the image sensor according to the third embodiment.

[0051] Figure 12A This is an explanation of the reason. Figure 11 The diagram shows a concept of an example of an image sensor capturing multiple images.

[0052] Figure 12B This is to explain from Figure 12A A conceptual diagram illustrating an example of how multiple images can be used to generate a super-resolution image.

[0053] Figure 12C This is an explanation based on Figure 12B A conceptual diagram illustrating an example of detecting defective physical pixels in a super-resolution image.

[0054] Figure 13A This is a conceptual diagram illustrating an example of a situation where an image sensor according to the fourth embodiment captures multiple images and generates multiple monochrome images from those multiple images.

[0055] Figure 13B This is to explain from Figure 13A A conceptual diagram illustrating an example of how multiple monochrome images generate multiple monochrome super-resolution images.

[0056] Figure 13C It indicates magnification. Figure 13B A magnified view of an example of a monochrome super-resolution image.

[0057] Figure 14 This is a block diagram illustrating an example of the first variant of a camera device.

[0058] Figure 15 This is a block diagram illustrating an example of a second variant of a camera device.

[0059] Figure 16 This is the front view representing the first comparative example of the super-resolution image.

[0060] Figure 17 This is the front view representing the second comparative example of the super-resolution image.

[0061] Figure 18 This is the front view representing the third comparative example of the super-resolution image.

[0062] Figure 19 This is the front view representing a comparative example of a monochrome super-resolution image. Detailed Implementation

[0063] Hereinafter, an example of an embodiment of the detection device, imaging device, detection method, and storage medium involved in the technology of the present invention will be described with reference to the accompanying drawings.

[0064] First, let me explain the terms used in the following explanation.

[0065] CPU stands for Central Processing Unit. RAM stands for Random Access Memory. IC stands for Integrated Circuit. ASIC stands for Application Specific Integrated Circuit. PLD stands for Programmable Logic Device. FPGA stands for Field-Programmable Gate Array. SoC stands for System-on-a-Chip. SSD stands for Solid State Drive. USB stands for Universal Serial Bus. HDD stands for Hard Disk Drive. EEPROM stands for Electrically Erasable and Programmable Read Only Memory. EL stands for Electro-Luminescence. I / F stands for "Interface". UI stands for "User Interface". CMOS stands for "Complementary Metal Oxide Semiconductor". CCD stands for "Charge Coupled Device". LAN stands for "Local Area Network". WAN stands for "Wide Area Network".

[0066] In this specification, "vertical" means, besides perfect verticality, a degree of error that is generally permissible in the technical field to which the present invention pertains and does not violate the spirit of the present invention. In this specification, "consistent" means, besides perfect consistency, a degree of consistency that is generally permissible in the technical field to which the present invention pertains and does not violate the spirit of the present invention.

[0067] [First Implementation]

[0068] First, the first embodiment will be described. As an example, such as... Figure 1 As shown, the camera device 10 is a digital camera with an interchangeable lens and without a reflex mirror. The camera device 10 includes a camera device body 12 and an interchangeable lens 14 that is interchangeably mounted on the camera device body 12.

[0069] Furthermore, as an example of the imaging device 10, a digital camera with an interchangeable lens and without a reflex mirror can be cited. However, the technology of the present invention is not limited to this; it can also be a digital camera with a fixed lens, a digital camera using a reflex mirror, or a digital camera built into various electronic devices such as smart devices, wearable terminals, cell observation devices, ophthalmic observation devices, or surgical microscopes.

[0070] An image sensor 40 is provided in the main body 12 of the imaging device. As an example, the image sensor 40 is a CMOS image sensor. The image sensor 40 captures an image area containing a group of subjects. When the interchangeable lens 14 is mounted on the main body 12 of the imaging device, the subject light representing the subject is transmitted through the interchangeable lens 14 and imaged on the image sensor 40, and the image sensor 40 generates image data representing the image of the subject.

[0071] Furthermore, while a CMOS image sensor is exemplified as the image sensor 40, the technology of the present invention is not limited thereto. For example, the technology of the present invention will still be valid even if the image sensor 40 is another type of image sensor such as a CCD image sensor.

[0072] A release button 18 and a turntable 20 are provided on the upper surface of the camera device body 12. The turntable 20 is operated when setting the working mode of the camera system and the working mode of the playback system. By operating the turntable 20, the camera mode and the playback mode can be selectively set as the working mode in the camera device 10.

[0073] The release button 18 functions as both a camera preparation indicator and a camera indicator, capable of detecting press operations in both the camera preparation indicator state and the camera indicator state. The camera preparation indicator state refers to, for example, the state from the standby position to the middle position (half-press position), while the camera indicator state refers to the state from the middle position to the final press position (full-press position).

[0074] Furthermore, the state of "pressing from the standby position to the half-press position" will be referred to as the "half-press state" and the state of "pressing from the standby position to the fully press position" will be referred to as the "fully press state". According to the structure of the camera device 10, the camera preparation indication state can be the state in which the user's finger touches the release button 18, and the camera indication state can be the state in which the user's finger transitions from the state of touching the release button 18 to the state of separation.

[0075] As an example, such as Figure 2 As shown, a touch panel display 22 and indicator keys 24 are provided on the back of the camera device body 12. The touch panel display 22 includes a display 26 and a touch panel 28. As an example of the display 26, an organic EL display can be given. The display 26 can also be other types of displays such as liquid crystal displays or inorganic EL displays, instead of organic EL displays.

[0076] The display 26 displays images and / or character information, etc. When the camera device 10 is in camera mode, the display 26 is used to display real-time preview images obtained from continuous video recording for real-time preview purposes. "Real-time preview image" refers to a dynamic image for display based on image data captured by the image sensor 40. Real-time preview images are also generally referred to as live view images.

[0077] When the camera device 10 is given an instruction to capture a still image via the release button 18, the display 26 is also used to display the still image obtained by capturing the still image. In addition, the display 26 is also used to display the regenerated image when the camera device 10 is in playback mode and to display menu screens, etc.

[0078] Touch panel 28 is a transmissive touch panel that overlaps the surface of the display area of ​​display 26. Touch panel 28 receives instructions from the user by detecting contact based on a finger or stylus. Additionally, for ease of explanation, the "full press state" described above also includes the state where the user activates the soft key for starting video recording via touch panel 28.

[0079] Furthermore, in this embodiment, as an example of the touch panel display 22, an external touch panel display is described where the touch panel 28 overlaps the surface of the display area of ​​the display 26, but this is only one example. For example, as the touch panel display 22, an external or internal touch panel display can also be used.

[0080] Indicator key 24 receives various instructions. Here, "various instructions" refers to, for example, instructions for displaying a menu screen that allows selection of various menus, instructions for selecting one or more menus, instructions for confirming selections, instructions for deleting selections, instructions for zooming in, zooming out, and frame transmission, etc. Furthermore, these instructions can also be made by touch panel 28.

[0081] As an example, such as Figure 3 As shown, the interchangeable lens 14 includes a camera lens 30. The camera lens 30 includes an objective lens 30A, a focusing lens 30B, and an aperture 30C. The objective lens 30A, focusing lens 30B, and aperture 30C extend from the subject side (object side) to the imaging device body 12 side (image side) along the optical axis OA (see also [reference]). Figure 1 The objective lens 30A, focusing lens 30B, and aperture 30C are arranged in that order.

[0082] Furthermore, the interchangeable lens 14 includes a sliding mechanism 32, a motor 34, and a motor 36. A focusing lens 30B is mounted in the sliding mechanism 32 in a manner that allows it to slide along the optical axis OA. The motor 34 is connected to the sliding mechanism 32, and the sliding mechanism 32 moves the focusing lens 30B along the optical axis OA by receiving power from the motor 34. The aperture 30C is a variable aperture. The motor 36 is connected to the aperture 30C, and the aperture 30C adjusts the exposure by receiving power from the motor 36. Additionally, the configuration and / or operating method of the interchangeable lens 14 can be changed as needed.

[0083] Motors 34 and 36 are connected to the camera unit body 12 via mounting brackets (not shown), and their operation is controlled according to commands from the camera unit body 12. Furthermore, as an example, stepper motors are used for motors 34 and 36. Therefore, motors 34 and 36 operate synchronously with pulse signals according to commands from the camera unit body 12. Figure 3 The example shown illustrates motors 34 and 36 mounted on the interchangeable lens 14. However, this is not a limitation; one of motors 34 and 36 may be mounted on the camera body 12, or both motors 34 and 36 may be mounted on the camera body 12.

[0084] The main body 12 of the camera device includes an image sensor 40, a signal processing circuit 42, a jitter detection sensor 44, and a jitter correction mechanism 46.

[0085] Image sensor 40 has a light-receiving surface 40A. As an example, image sensor 40 is a photoelectric conversion element. Image sensor 40 is sometimes referred to as a solid-state imaging element. As an example, image sensor 40 is arranged on the imaging device body 12 (reference) with the center of the light-receiving surface 40A aligned with the optical axis OA. Figure 1 The image sensor 40 can be a monochrome image sensor or a color image sensor with color filters that assign different colors to multiple physical pixels. In the first embodiment, as an example, the image sensor 40 is a monochrome image sensor. Alternatively, as an example, the monochrome image sensor 40 in the first embodiment is an image sensor without multiple color filters, but the technology of the present invention is not limited thereto. Even with a monochrome image sensor that assigns multiple monochrome color filters to multiple physical pixels, the same effect as in the first embodiment can be obtained.

[0086] The signal processing circuit 42 is connected to the image sensor 40. When the interchangeable lens 14 is mounted on the camera device body 12, the subject light incident on the imaging lens 30 is imaged onto the light-receiving surface 40A through the imaging lens 30. Under the control of the photoelectric conversion element driver 68 (described later), the image sensor 40 performs photoelectric conversion on the subject light received by the light-receiving surface 40A and outputs an electrical signal corresponding to the amount of subject light as analog image data representing the subject light to the signal processing circuit 42. The signal processing circuit 42 digitizes the analog image data input from the image sensor 40 to generate digital image data.

[0087] The jitter detection sensor 44 is, for example, a device including a gyroscope sensor, that detects the amount of jitter in the camera device 10 along a direction perpendicular to the optical axis OA. The jitter detection sensor 44 outputs the detected jitter to the CPU 60A, which will be described later.

[0088] As an example of a direction perpendicular to the optical axis OA, the jitter detection sensor 44 detects the jitter of the camera device 10 in each of the X and Y directions. For example, the X direction is parallel to the pitch axis of the camera device 10, and the Y direction is parallel to the yaw axis of the camera device 10. The X and Y directions are orthogonal to each other. A gyroscope sensor included in the jitter detection sensor 44 detects the rotational jitter of the camera device 10 around each axis of the pitch axis, yaw axis, and roll axis (axis parallel to the optical axis OA). The jitter detection sensor 44 converts the rotational jitter around the pitch axis and the rotational jitter around the yaw axis detected by the gyroscope sensor into in-plane jitter in a two-dimensional plane parallel to the pitch and yaw axes to detect the jitter of the camera device 10 in each of the X and Y directions.

[0089] Furthermore, as an example of a jitter detection sensor 44, a device incorporating a gyroscope sensor is given, but this is only one example; the jitter detection sensor 44 can also be an accelerometer sensor. The accelerometer sensor detects the amount of jitter of the camera device 10 in a two-dimensional plane parallel to the pitch and yaw axes.

[0090] Furthermore, an example of detecting jitter using a physical sensor, jitter detection sensor 44, has been given, but the technology of the present invention is not limited thereto. For example, a motion vector obtained by comparing images stored in the image memory 62 (described later) in a time series can also be used as the jitter amount. Moreover, the final jitter amount can be derived from the jitter amount detected by the physical sensor and the motion vector obtained through image processing.

[0091] The jitter correction mechanism 46 is a mechanism that corrects jitter by applying power generated by a drive source such as a motor (e.g., a voice coil motor) to the image sensor 40, causing the image sensor 40 to move in a direction perpendicular to the optical axis OA. As an example of the direction perpendicular to the optical axis OA, the jitter correction mechanism 46 moves the image sensor 40 in both the X and Y directions. The jitter correction mechanism 46 is an example of a "movement mechanism" according to the technology of this invention.

[0092] The jitter correction mechanism 46 includes a displacement driver 50, an actuator 52, and a position detection sensor 54. Various well-known methods can be used in the jitter correction method based on the jitter correction mechanism 46. One such method involves moving the image sensor 40 via the actuator 52 according to an instruction from the CPU 60A, based on the jitter amount detected by the jitter amount detection sensor 44, thereby correcting the jitter. Specifically, jitter correction is achieved by moving the image sensor 40 in the jitter-eliminating direction by an amount equivalent to the amount of jitter elimination.

[0093] The actuator 52 is, for example, a displacement mechanism equipped with a voice coil motor, which drives the image sensor 40 to move along the X and Y directions. Alternatively, as an example of the actuator 52, a displacement mechanism equipped with a voice coil motor is used, but the technology of the present invention is not limited to this; other power sources such as stepper motors or piezoelectric elements can be used instead of voice coil motors.

[0094] The position detection sensor 54 detects the position of the image sensor 40 and outputs a position signal indicating the detected position. As an example, the position detection sensor 54 employs a device including a Hall element. The position of the image sensor 40 detected by the position detection sensor 54 refers to its position within a two-dimensional plane parallel to the light-receiving surface 40A of the image sensor 40. This two-dimensional plane is perpendicular to the optical axis OA. Furthermore, in this embodiment, as an example, a device including a Hall element is used for the position detection sensor 54; however, the technology of the present invention is not limited to this, and a magnetic sensor or optical sensor, etc., may be used instead of a Hall element.

[0095] Furthermore, the camera device body 12 includes a controller 60, an image memory 62, a UI system device 64, an external I / F 66, a photoelectric conversion element driver 68, a mechanical shutter driver 74, a mechanical shutter actuator 76, a mechanical shutter 78, and an input / output interface 80.

[0096] The input / output interface 80 is connected to a controller 60, an image memory 62, a UI system device 64, an external I / F 66, a photoelectric conversion element driver 68, a mechanical shutter driver 74, a signal processing circuit 42, a jitter detection sensor 44, and a displacement driver 50.

[0097] The controller 60 includes a CPU 60A, a memory 60B, and a memory 60C. The CPU 60A is an example of a "processor" according to the technology of the present invention, the memory 60C and the image memory 62 are examples of "memory" according to the technology of the present invention, and the controller 60 is an example of a "detection device" and a "computer" according to the technology of the present invention.

[0098] CPU 60A, memory 60B and memory 60C are connected via bus 82, which is connected to input / output interface 80.

[0099] In addition, Figure 3 In the example shown, for ease of illustration, bus 82 is represented as a single bus, but multiple buses can also be represented. Bus 82 can be a serial bus or a parallel bus that includes a data bus, an address bus, and a control bus.

[0100] Memory 60B stores various parameters and programs. Memory 60B is a non-volatile storage device. As an example of memory 60B, EEPROM is used. EEPROM is just one example; as memory 60B, it can replace EEPROM and be used with HDDs and / or SSDs, or be used in conjunction with EEPROM with HDDs and / or SSDs, etc. Furthermore, memory 60C temporarily stores various information and is used as working memory. As an example of memory 60C, RAM can be used, but it is not limited to this; other types of storage devices can also be used.

[0101] Various programs are stored in memory 60B. CPU 60A reads the required program from memory 60B and executes the read program in memory 60C. CPU 60A controls the entire camera device body 12 according to the program executed in memory 60C. Figure 3 In the example shown, the image memory 62, UI system device 64, external I / F 66, photoelectric conversion element driver 68, mechanical shutter driver 74, and displacement driver 50 are controlled by CPU 60A.

[0102] An image sensor 40 is connected to the photoelectric conversion element driver 68. The photoelectric conversion element driver 68 supplies a camera timing signal, which specifies the timing of the image capture performed by the image sensor 40, to the image sensor 40 according to an instruction from the CPU 60A. The image sensor 40 performs reset, exposure, and outputs electrical signals according to the camera timing signal supplied from the photoelectric conversion element driver 68.

[0103] As an example, the mechanical shutter 78 is a focal plane shutter, positioned between aperture 30C and the light-receiving surface 40A. The mechanical shutter 78 has a front curtain (not shown) and a rear curtain (not shown). Each of the front and rear curtains has multiple blades. The front curtain is positioned closer to the subject than the rear curtain.

[0104] The mechanical shutter actuator 76 is an actuator having a front curtain solenoid (not shown) and a rear curtain solenoid (not shown). The front curtain solenoid is the drive source for the front curtain and is mechanically connected to the front curtain. The rear curtain solenoid is the drive source for the rear curtain and is mechanically connected to the rear curtain. The mechanical shutter driver 74 controls the mechanical shutter actuator 76 according to instructions from the CPU 60A.

[0105] The front curtain is powered by a solenoid under the control of the mechanical shutter driver 74, and this power is applied to the front curtain, thereby selectively raising and lowering it. The rear curtain is also powered by a solenoid under the control of the mechanical shutter driver 74, and this power is applied to the rear curtain, thereby selectively raising and lowering it. In the imaging device 10, the opening and closing of the front and rear curtains are controlled by the CPU 60A, thereby controlling the exposure to the image sensor 40.

[0106] In the imaging device 10, there are two types of imaging: one for real-time preview of images using an exposure sequence readout method (rolling shutter method), and the other for recording still images and / or moving images. The image sensor 40 has an electronic shutter function, and the real-time preview of images is not performed when the mechanical shutter 78 is set to the fully open state, but is achieved by activating the electronic shutter function.

[0107] In contrast, the imaging accompanying the main exposure, i.e., the imaging for still images (hereinafter also referred to as "main exposure imaging"), is achieved by activating the electronic shutter function and causing the mechanical shutter 78 to actuate from the front curtain closed state to the rear curtain closed state. Images obtained by imaging for real-time preview and for recording images for recording still images and / or moving images are examples of the "images" involved in the technology of this invention.

[0108] The digital image data generated by the signal processing circuit 42 is stored as an image in the image memory 62. The CPU 60A retrieves the digital image data from the image memory 62 and uses the retrieved digital image data to perform various processes.

[0109] The UI system device 64 includes a display 26, and the CPU 60A enables the display 26 to display various information. Furthermore, the UI system device 64 includes a receiver 84. The receiver 84 includes a touch panel 28 and a hard key unit 86. The hard key unit 86 includes indicator keys 24 (see reference). Figure 2The CPU 60A operates according to various instructions received from the touch panel 28. Additionally, the UI system device 64 includes a hard key section 86, but the technology of the present invention is not limited thereto; for example, the hard key section 86 may also be connected to an external I / F 66.

[0110] The external I / F66 controls the transmission and reception of various information between the camera device 10 and devices located outside the camera device 10 (hereinafter also referred to as "external devices"). An example of an external I / F66 is a USB interface. External devices such as smart devices, personal computers, servers, USB storage devices, memory cards, and / or printers (not shown) are directly or indirectly connected to the USB interface.

[0111] The interchangeable lens 14 includes a lens CPU (not shown). The lens CPU, a CPU located within the interchangeable lens 14, controls the operation of the interchangeable lens 14 by executing a predetermined program. Furthermore, a first motor driver (not shown) and a second motor driver (not shown) are provided within the interchangeable lens 14. The lens CPU is connected to motor 34 via the first motor driver. The lens CPU is also connected to motor 36 via the second motor driver. Motor 34 is connected to the input / output interface 80 of the camera unit 12 via the first motor driver and the lens CPU, and motor 36 is connected to the input / output interface 80 of the camera unit 12 via the second motor driver and the lens CPU. The lens CPU controls the operation of the interchangeable lens 14 according to instructions given from CPU 60A via the input / output interface 80.

[0112] The lens CPU controls motor 34 via the first motor driver according to instructions from CPU 60A. Thus, by controlling motor 34, the position of the focusing lens 30B on the optical axis OA is controlled via the sliding mechanism 32. Furthermore, the lens CPU controls motor 36 via the second motor driver according to instructions from CPU 60A. Thus, by controlling motor 36, the size of the aperture 30C is controlled.

[0113] The displacement driver 50 is connected to the actuator 52. The CPU 60A acquires the jitter amount detected by the jitter amount detection sensor 44 and outputs the instruction to control the jitter correction mechanism 46 according to the acquired jitter amount to the displacement driver 50.

[0114] As an example, such as Figure 4 As shown, a camera processing program 88 is stored in memory 60B. The camera processing program 88 is an example of a "program" according to the technology of this invention. CPU 60A reads the camera processing program 88 from memory 60B and executes the read camera processing program 88 on memory 60C. CPU 60A performs camera processing according to the camera processing program 88 executed on memory 60C.

[0115] As an example, such as Figure 5 As shown, the CPU 60A operates as a camera control unit 90, a storage processing unit 92, a generation unit 94, a detection unit 96, a correction unit 98, and an output unit 100 by executing the camera processing program 88.

[0116] As an example, such as Figure 6 As shown, the image sensor 40 has a plurality of physical pixels 48. For example, the plurality of physical pixels 48 are arranged along the X and Y directions. For example, the spacing between the plurality of physical pixels 48 in the X direction is the same, and the spacing between the plurality of physical pixels 48 in the Y direction is the same. Furthermore, for example, the spacing between the plurality of physical pixels 48 in the X direction is the same as the spacing between the plurality of physical pixels 48 in the Y direction. Alternatively, the spacing between the plurality of physical pixels 48 in the X direction may differ from the spacing between the plurality of physical pixels 48 in the Y direction.

[0117] The X direction is an example of the "first direction" involved in the technology of this invention, and the Y direction is an example of the "second direction intersecting the first direction" involved in the technology of this invention. The spacing in the X direction between the plurality of physical pixels 48 is equivalent to the distance between the centers of the plurality of physical pixels 48 in the X direction. The spacing in the Y direction between the plurality of physical pixels 48 is equivalent to the distance between the centers of the plurality of physical pixels 48 in the Y direction.

[0118] exist Figure 6 The example shown illustrates a portion of multiple physical pixels 48, specifically 3 × 3 = 9 physical pixels 48. Furthermore, in Figure 6 The image shows the addresses of multiple physical pixels 48 in the X and Y directions. For example, address (1, 1) indicates that the address in the X direction is 1 and the address in the Y direction is 1, and address (2, 1) indicates that the address in the X direction is 2 and the address in the Y direction is 1. The multiple physical pixels 48 face the light-receiving surface 40A (see also...). Figure 3 As an example, each of the multiple physical pixels 48 has a photodiode that performs photoelectric conversion on the received light and outputs an electrical signal corresponding to the amount of light received.

[0119] As an example, such as Figure 7A As shown, the camera control unit 90 controls the actuator 52 of the shake correction mechanism 46 to move the image sensor 40 along the X direction. At this time, the camera control unit 90 controls the movement of the image sensor 40 to obtain the super-resolution image 120 (see reference 120) described later. Figure 7B ).

[0120] For example, corresponding to a monochrome image sensor 40, the camera control unit 90 controls the movement of the image sensor 40 as follows: When the spacing between the plurality of physical pixels 48 is set to p [μm], the camera control unit 90 controls the image sensor 40 to move in the X direction by a movement amount of 1.5 × p [μm] while keeping the position of the image sensor 40 fixed in the Y direction.

[0121] Furthermore, the camera control unit 90 controls the photoelectric conversion element driver 68 to cause the image sensor 40 to capture images before and after movement. Thus, the camera control unit 90 acquires images 110A and 110B captured by the image sensor 40 before and after movement. Image 110A is the image captured by the image sensor 40 before movement, and image 110B is the image captured by the image sensor 40 after movement. Image 110B is the image captured by the image sensor 40 at a position offset by only 1.5 × p [μm] in the X direction relative to image 110A. Figure 7A In the example shown, to easily distinguish between image 110A and image 110B, dotted shading lines are marked on image 110B, which was captured by the moved image sensor 40. Images 110A and 110B are examples of "multiple images" involved in the technology of this invention.

[0122] Image 110A contains multiple image pixels A, and image 110B contains multiple image pixels B. The multiple image pixels A correspond to multiple physical pixels 48 of the image sensor 40 before the movement, and the multiple image pixels B correspond to the multiple physical pixels 48 of the image sensor 40 after the movement. Figure 7A The example shown illustrates a portion of multiple image pixels A obtained before the image sensor 40 moves, and a portion of multiple image pixels B obtained after the image sensor 40 moves, i.e., 3×3=9 image pixels A and 3×3=9 image pixels B. For example, image pixels A and B at address (1,1) correspond to physical pixel 48 at address (1,1), and image pixels A and B at address (2,1) correspond to physical pixel 48 at address (2,1). Furthermore, for ease of explanation, image pixels A and B will be referred to as "image pixels" without any symbol when there is no need to distinguish between them.

[0123] The storage processing unit 92 performs the processing of storing the images 110A and 110B captured by the image sensor 40 in the image memory 62.

[0124] As an example, such as Figure 7BAs shown, the generation unit 94 performs super-resolution transformation on multiple low-resolution images (e.g., images 110A and 110B) acquired by the image sensor 40. Here, super-resolution transformation refers, for example, to the process of generating a high-resolution image (i.e., a higher-resolution image) by synthesizing multiple low-resolution images by staggering the positions of image pixels. The high-resolution image generated in this way is generally also referred to as a super-resolution image.

[0125] exist Figure 7B In the example shown, the generation unit 94 performs super-resolution on images 110A and 110B acquired by the image sensor 40 to generate a super-resolution image 120. Super-resolution when using a monochrome image sensor 40 is implemented, for example, by determining the arrangement order of three image pixels A and B arranged along the X direction, obtained before and after the movement of the image sensor 40, based on their absolute positions before and after the movement of the image sensor 40. The absolute position is the position relative to the center of image pixel A and the center of image pixel B.

[0126] Next, the plurality of image pixels A contained in image 110A obtained before the movement of image sensor 40 and the plurality of image pixels B contained in image 110B obtained after the movement of image sensor 40 are arranged according to the above-described arrangement order. Thus, a super-resolution image 120 is obtained by super-resolution of images 110A and 110B. Figure 7B The example shown illustrates a portion of the super-resolution image 120, namely 9 × 2 = 18 image pixels A and B obtained before and after the movement of the image sensor 40.

[0127] As illustrated above, the camera control unit 90 controls the image sensor 40 to move along the X direction by a movement amount of 1.5 × p [μm]. Thus, as shown in the super-resolution image 120, when image pixel A corresponding to any physical pixel 48 before the movement is designated as the first image pixel A, other image pixels A different from the first image pixel A are designated as the second image pixel A, and image pixel B corresponding to the aforementioned arbitrary physical pixel 48 after the movement is designated as the first image pixel B, the first image pixel B and the second image pixel A are adjacent.

[0128] exist Figure 7A and Figure 7B In the example shown, if image pixel A at address (1,1) is set as the first image pixel A, image pixel A at address (2,1) is set as the second image pixel A, and image pixel B at address (1,1) is set as the first image pixel B, then image pixel B at address (1,1), i.e., the first image pixel B, is adjacent to image pixel A at address (2,1), i.e., the first image pixel A.

[0129] Each of the plurality of image pixels A and B corresponds to the position of each of the plurality of physical pixels 48 before and after the image sensor 40 moves. Therefore, the super-resolution image 120 represents the physical pixel configuration before and after the image sensor 40 moves. For example, when the physical pixel 48 corresponding to the first image pixel A and the first image pixel B is set as the first physical pixel, and the physical pixel 48 corresponding to the second image pixel A is set as the second physical pixel, the camera control unit 90 performs the following control: in the physical pixel configuration before and after the movement of the image sensor 40 represented by the super-resolution image 120, the image sensor 40 is moved to a position adjacent to the second physical pixel, which is different from the first physical pixel before the movement, where the first physical pixel after the movement is located.

[0130] In other words, the camera control unit 90 moves the image sensor 40 so that the physical pixel configuration of the image sensor 40 before and after the movement, as represented by the super-resolution image 120, is such that adjacent image pixels contained in the super-resolution image 120 become image pixels corresponding to mutually different physical pixels 48.

[0131] As an example of such control, in the above example, the camera control unit 90 controls the image sensor 40 to move in the X direction by a movement amount of 1.5 × p [μm]. As described above, the physical pixel configuration of the image sensor 40 before and after the movement is represented, for example, by the super-resolution image 120, and the above physical pixel configuration can be achieved regardless of whether there is a space equivalent to one physical pixel between the physical pixels of the image sensor 40.

[0132] In addition, Figure 7A and Figure 7B In the example shown, for instance, when the physical pixel 48 at address (1,1) is set as the first physical pixel before the image sensor 40 moves, the physical pixel 48 at address (1,1) corresponds to an example of the "first physical pixel" involved in the technology of the present invention, and the second physical pixel, which is different from the physical pixel 48 at address (1,1) before the move, i.e., the physical pixel 48 at address (2,1), corresponds to an example of the "second physical pixel" involved in the technology of the present invention.

[0133] Furthermore, for example, image pixel A at address (1,1) is an example of "image pixel corresponding to the first physical pixel before the move" according to the technology of the present invention, image pixel A at address (2,1) is an example of "image pixel corresponding to a second physical pixel different from the first physical pixel before the move" according to the technology of the present invention, and image pixel B at address (1,1) is an example of "image pixel corresponding to the first physical pixel after the move" according to the technology of the present invention.

[0134] As an example, such as Figure 7C As shown, the detection unit 96 calculates the difference in pixel values ​​between the multiple image pixels A and multiple image pixels B contained in the super-resolution image 120 and the image pixels A and B corresponding to the second physical pixel before movement and the first physical pixel after movement, respectively. When the image sensor 40 moves along the X direction, image pixels that are adjacent to each other in the X direction are selected. Similarly, when the image sensor 40 moves along the Y direction, image pixels that are adjacent to each other in the Y direction are selected. The pixel value of image pixel A is proportional to the value of the electrical signal output from the physical pixel 48 corresponding to image pixel A, and the pixel value of image pixel B is proportional to the value of the electrical signal output from the physical pixel 48 corresponding to image pixel B.

[0135] The detection unit 96 detects defective physical pixels from a plurality of physical pixels 48 based on the calculated difference degree. A defective physical pixel refers to a physical pixel 48 that has a defect. As an example, the detection unit 96 sequentially calculates the difference degree of pixel values ​​between all image pixels A and B of the detection target (hereinafter referred to as the "detection target") that are defective physical pixels. Figure 7C In the example, image pixels A and B, which are the objects of detection, are highlighted (displayed by circles) for image pixel A at address (2,1) and image pixel B at address (1,1). Furthermore, the case where the difference in pixel value between image pixel A at address (2,1) and image pixel B at address (1,1) is calculated is shown.

[0136] Various methods can be applied to the method for detecting defective physical pixels based on the detection unit 96. For example, the detection unit 96 detects defective physical pixels using the following approach.

[0137] That is, for example, the detection unit 96 calculates the difference between the pixel values ​​of image pixels A and B that are the objects of detection as the difference between the pixel values ​​of image pixels A and B that are the objects of detection. When the calculated difference exceeds a preset threshold, the detection unit 96 acquires the pixel values ​​of image pixels A and B that are the objects of the difference calculation and the pixel values ​​of image pixels A and B around them. For example, when the difference between the pixel values ​​of image pixel A at address (2,1) and image pixel B at address (1,1) exceeds the preset threshold, the detection unit 96 acquires the pixel value of image pixel A at address (2,1) and the pixel value of image pixel B at address (1,1), and acquires the pixel value of each of the image pixels A at address (1,1), image pixel A at address (3,1), image pixel B at address (2,1), and image pixel B at address (3,1).

[0138] Then, when the arrangement of the acquired multiple pixel values, i.e. the pixel value pattern, is consistent with the abnormal pixel value pattern that predefines the image pixel A at address (2,1) and its corresponding image pixel B at address (2,1) as white scratches or black scratches, the detection unit 96 detects that the physical pixel 48 at address (2,1) corresponding to the image pixel A at address (2,1) and the image pixel B at address (2,1) is a defective physical pixel.

[0139] Furthermore, in this case, when the acquired pixel value pattern matches an abnormal pixel value pattern containing white scratches, the detection unit 96 can detect that image pixel A at address (2,1) and image pixel B at address (2,1) are white scratches. And, when the acquired pixel value pattern matches an abnormal pixel value pattern containing black scratches, the detection unit 96 can detect that image pixel A at address (2,1) and image pixel B at address (2,1) are black scratches.

[0140] White scratches refer to the phenomenon where a pixel appears white because the electrical signal output from the defective physical pixel is kept at its maximum value. Black scratches refer to the phenomenon where a pixel appears black because the defective physical pixel does not output an electrical signal.

[0141] Furthermore, when the arrangement of the acquired multiple pixel values, i.e. the pixel value pattern, is consistent with the pre-defined abnormal pixel value pattern of image pixel B at address (1,1) and its corresponding image pixel A at address (1,1) as white scratches or black scratches, the detection unit 96 detects that the physical pixel 48 at address (1,1) corresponding to image pixel B at address (1,1) and image pixel A at address (1,1) is a defective physical pixel.

[0142] Furthermore, in this case, when the acquired pixel value pattern matches an abnormal pixel value pattern containing white scratches, the detection unit 96 can detect that image pixel B at address (1,1) and image pixel A at address (1,1) are white scratches. And, when the acquired pixel value pattern matches an abnormal pixel value pattern containing black scratches, the detection unit 96 can detect that image pixel B at address (1,1) and image pixel A at address (1,1) are black scratches.

[0143] The correction unit 98 corrects the pixel value of the image pixel corresponding to the defective physical pixel detected by the detection unit 96 based on the pixel value of the image pixel adjacent to that image pixel. Various methods can be applied to correct the pixel value. For example, the correction unit 98 corrects the pixel value using the following method.

[0144] That is, the correction unit 98 performs correction by setting the pixel value of the first image pixel corresponding to the defective physical pixel detected by the detection unit 96 to the average or median value of the pixel values ​​of the second image pixel adjacent to the first image pixel.

[0145] For example, when physical pixel 48 at address (2,1) is detected as a defective physical pixel, the correction unit 98 corrects it by setting image pixel A at address (2,1) to the average or median value of the pixel values ​​of image pixels A and B adjacent to image pixel A at address (2,1), and by setting image pixel B at address (2,1) to the average or median value of the pixel values ​​of image pixels A and B adjacent to image pixel B at address (2,1). Similarly, when physical pixel 48 at address (1,1) is detected as a defective physical pixel, the correction unit 98 corrects it by setting image pixel B at address (1,1) to the average or median value of the pixel values ​​of image pixels A and B adjacent to image pixel B at address (1,1), and by setting image pixel A at address (1,1) to the average or median value of the pixel values ​​of image pixels A and B adjacent to image pixel A at address (1,1). In this way, by correcting the pixel values ​​of the image pixels corresponding to the defective physical pixels, a corrected super-resolution image 120 can be obtained. For example, when the image pixel corresponding to the defective physical pixel is a white scratch or a black scratch, a super-resolution image 120 can be obtained in a way that corrects the white scratch or black scratch in a way that makes it inconspicuous.

[0146] Furthermore, in this example, when physical pixel 48 at address (2,1) is detected as a defective physical pixel, image pixel A and image pixel B at address (2,1) correspond to an example of the "first image pixel corresponding to the defective physical pixel" according to the technology of this invention, and each image pixel A and B adjacent to image pixel A and image pixel B at address (2,1) corresponds to an example of the "second image pixel adjacent to the first image pixel" according to the technology of this invention. Also, when physical pixel 48 at address (1,1) is detected as a defective physical pixel, image pixel B and image pixel A at address (1,1) correspond to an example of the "first image pixel corresponding to the defective physical pixel" according to the technology of this invention, and image pixel A adjacent to image pixel B at address (1,1) and image pixel B adjacent to image pixel A at address (1,1) correspond to an example of the "second image pixel adjacent to the first image pixel" according to the technology of this invention.

[0147] The output unit 100 performs the process of outputting the corrected super-resolution image 120 to the display 26. Thus, the super-resolution image 120 is displayed on the display 26 while the pixel values ​​of the image pixels corresponding to the defective physical pixels are corrected. For example, when the image pixel corresponding to the defective physical pixel is a white scratch or a black scratch, the super-resolution image 120, corrected in a way that makes the white or black scratch inconspicuous, is displayed on the display 26.

[0148] Next, refer to Figure 8 The operation of the camera device 10 according to the first embodiment will be explained.

[0149] exist Figure 8 The diagram shows an example of the camera processing flow executed by the CPU 60A when a camera mode is set for the camera device 10.

[0150] exist Figure 8 In the image processing shown, firstly, in step ST100, the camera control unit 90 controls the actuator 52 of the shake correction mechanism 46 to move the image sensor 40 in the X direction. Furthermore, while controlling the movement of the image sensor 40, the camera control unit 90 controls the photoelectric conversion element driver 68 to cause the image sensor 40 to capture images before and after its movement.

[0151] In the next step ST102, the storage processing unit 92 performs the process of storing the images 110A and 110B obtained by the image sensor 40 in the image memory 62.

[0152] In the next step ST104, the generation unit 94 performs super-resolution on the images 110A and 110B obtained by the image sensor 40 to generate a super-resolution image 120.

[0153] In the next step ST106, the detection unit 96 calculates the difference in pixel values ​​between image pixels A and B contained in the super-resolution image 120 and the image pixels A and B corresponding to the second physical pixel before and the first physical pixel after the movement, respectively. Furthermore, the detection unit 96 detects defective physical pixels from the plurality of physical pixels 48 based on the calculated difference.

[0154] In the next step ST108, the correction unit 98 corrects the pixel value of the image pixel corresponding to the defective physical pixel detected by the detection unit 96 based on the pixel value of the image pixel adjacent to that image pixel.

[0155] In the next step ST110, the output unit 100 performs the process of outputting the corrected super-resolution image 120 to the display 26.

[0156] As explained above, in the imaging device 10, control is performed such that, in the physical pixel configuration before and after the movement of the image sensor 40 having a plurality of physical pixels 48, the image sensor 40 is moved to a position adjacent to a second physical pixel that is different from any first physical pixel before the movement in any of the moved first physical pixels 48. Furthermore, the image sensor 40 is controlled by the imaging control unit 90 via the photoelectric conversion element driver 68 to capture images before and after the movement using the image sensor 40. Then, based on the difference in pixel values ​​between the image pixel A corresponding to the second physical pixel before the movement and the image pixel B corresponding to the first physical pixel after the movement in each of the plurality of image pixels A and B included in each of the images 110A and 110B captured by the image sensor 40, defective physical pixels are detected from the plurality of physical pixels 48. Therefore, defective physical pixels can be detected from the plurality of physical pixels 48 based on the images 110A and 110B captured by the image sensor 40 before and after the movement.

[0157] Furthermore, in the imaging device 10, the images 110A and 110B captured by the image sensor 40 are stored in the image memory 62. Then, defective physical pixels are detected from the plurality of physical pixels 48 based on the difference in pixel values ​​between the plurality of image pixels A and the plurality of image pixels B contained in each of the images 110A and 110B stored in the image memory 62. Therefore, defective physical pixels can be detected from the plurality of physical pixels 48 based on the images 110A and 110B stored in the image memory 62.

[0158] Furthermore, in the imaging device 10, the pixel value of the first image pixel corresponding to the defective physical pixel is corrected based on the pixel value of the second image pixel adjacent to the first image pixel. Therefore, the defect of the image pixel corresponding to the defective physical pixel can be corrected.

[0159] Furthermore, in the imaging device 10, for example, the image sensor 40 is a monochrome image sensor. Therefore, it is possible to detect defective physical pixels from the plurality of physical pixels 48 included in the monochrome image sensor 40.

[0160] Furthermore, in the imaging device 10, as an example, when the spacing between the plurality of physical pixels 48 is set to p, the imaging control unit 90 controls the image sensor 40 to move in the X direction by a movement amount of 1.5 × p [μm]. Therefore, in the physical pixel arrangement before and after the movement of the monochrome image sensor 40, the moved first physical pixel can be positioned adjacent to a second physical pixel that is different from the first physical pixel before the movement.

[0161] Furthermore, the camera device 10 includes a controller 60 for detecting defective physical pixels, an image sensor 40, and a jitter correction mechanism 46 for moving the image sensor 40. Therefore, the camera device 10 can continuously perform image capture based on the image sensor 40, move the image sensor 40, and detect defective physical pixels.

[0162] Furthermore, in the first embodiment described above, the camera control unit 90 moves the image sensor 40 along the X direction, but it can also move it along the Y direction. Moreover, the camera control unit 90 can move the image sensor 40 along both the X and Y directions. Furthermore, even when the image sensor 40 moves along the Y direction or along both the X and Y directions, the processing of detecting defective physical pixels from the plurality of physical pixels 48 and the processing of correcting the pixel value of the first image pixel corresponding to the defective physical pixel based on the pixel value of the second image pixel adjacent to the first image pixel can be performed using the same method as in the first embodiment described above.

[0163] Furthermore, in the first embodiment described above, the camera control unit 90 moves the image sensor 40 by a movement amount of 1.5 × p [μm]. However, when the spacing between the plurality of physical pixels 48 is set to 1, the camera control unit 90 can also move the image sensor 40 by a movement amount greater than 1 in the X or Y direction while obtaining the super-resolution image 120 described above. With this structure, in the physical pixel configuration before and after the movement of the image sensor 40, it is also possible to position the moved first physical pixel adjacent to a second physical pixel that is different from the moved first physical pixel.

[0164] Furthermore, in the first embodiment described above, when the natural number is set to n and the decimal number is set to d, the camera control unit 90 can move the image sensor 40 along the X or Y direction by a movement amount of (n+d)×p[μm] while obtaining the super-resolution image 120. With this structure, in the physical pixel configuration before and after the movement of the image sensor 40, it is also possible to position the moved first physical pixel adjacent to a second physical pixel that is different from the first physical pixel before the movement.

[0165] Furthermore, if the camera control unit 90 moves the image sensor 40 by a movement amount of 1.5×p [μm], for example, compared to the case where the image sensor 40 is moved by a small number greater than 1.5×p or a small number less than 1.5×p and greater than 1.0×p [μm], a high-resolution super-resolution image 120 can be obtained.

[0166] Furthermore, in the first embodiment described above, the output unit 100 can output position information corresponding to the position of the defective physical pixel. This position information is, for example, information corresponding to the address of the physical pixel 48. According to this structure, the position of the defective physical pixel can be determined based on the position information corresponding to the output position of the defective physical pixel.

[0167] Furthermore, in the first embodiment described above, the detection unit 96 can extract image pixels whose pixel values ​​deviate from a predetermined range from a plurality of image pixels, and detect defective physical pixels based on the difference between the pixel value of the image pixel outside the predetermined range and the pixel value of the image pixels adjacent to the image pixel outside the predetermined range. According to this structure, compared to calculating the difference between the pixel values ​​of the image pixels corresponding to all the second physical pixels before and after the movement, the number of times the difference is calculated can be reduced.

[0168] Furthermore, in the first embodiment described above, the detection unit 96 detects defective physical pixels from a plurality of physical pixels 48 during the generation of the super-resolution image 120, but the technology of the present invention is not limited to this. For example, in a process different from the generation of the super-resolution image 120, the detection unit 96 may also detect defective physical pixels from a plurality of physical pixels 48 using the same method as described above based on images 110A and 110B obtained by the image sensor 40 before and after movement.

[0169] In addition, Figure 16 The image shown is a super-resolution image 420 obtained through a comparative example relative to the first embodiment. This super-resolution image 420 is obtained by using the image sensor 40 described above (reference 40). Figure 7A The image is obtained by moving the sensor 420 by a displacement of 0.5 × p [μm] (instead of moving it by 1.5 × p [μm]). In this case, in the physical pixel configuration before and after the movement of the image sensor 40, the first physical pixel after the movement is located adjacent to the first physical pixel before the movement. Therefore, the pixel values ​​between adjacent image pixels in the super-resolution image 420 are the same, and thus it is impossible to detect defective physical pixels from multiple physical pixels.

[0170] [Second Implementation]

[0171] Next, the second embodiment will be described. In the second embodiment, the structure of the camera device 10 is changed from that of the first embodiment as follows. Furthermore, in the second embodiment, the same reference numerals are used for elements and components that are the same as in the first embodiment, and detailed descriptions are omitted. Regarding the overall structure of the camera device 10 according to the second embodiment, please refer to... Figures 1-5 .

[0172] As an example, such as Figure 9 As shown, in the second embodiment, a color image sensor 130 is used. The image sensor 130 has a plurality of physical pixels 138. As an example, the plurality of physical pixels 138 are arranged along the X and Y directions. As an example, the X-direction spacing between the plurality of physical pixels 138 is the same, and the Y-direction spacing between the plurality of physical pixels 138 is the same. Furthermore, as an example, the X-direction spacing between the plurality of physical pixels 138 is the same as the Y-direction spacing between the plurality of physical pixels 138. Alternatively, the X-direction spacing between the plurality of physical pixels 138 may be different from the Y-direction spacing between the plurality of physical pixels 138.

[0173] exist Figure 9 In the example shown, a portion of the multiple physical pixels 138, namely 4 × 3 = 12 physical pixels 138, are illustrated. Different color filters are assigned to the multiple physical pixels 138 in a periodic basal arrangement. The basal arrangement can be any arrangement, such as a strip arrangement, a delta arrangement, a Bayer arrangement, or an X-Trans (trademarked) arrangement. As an example of a basal arrangement, a Bayer arrangement is used in the image sensor 130.

[0174] A Bayer arrangement, for example, refers to an arrangement of 2×2=4 color filters as a whole. These 2×2=4 color filters consist of one filter corresponding to the red wavelength region, two filters corresponding to the green wavelength region, and one filter corresponding to the blue wavelength region. Hereinafter, when identifying multiple physical pixels 138 using color filters assigned to them, the physical pixel 138 assigned to the red wavelength region is referred to as the R physical pixel, the physical pixel 138 assigned to the green wavelength region is referred to as the G physical pixel, and the physical pixel 138 assigned to the blue wavelength region is referred to as the B physical pixel.

[0175] The 2×2=4 physical pixels 138 are arranged as follows. That is, the two physical pixels 138 in the first column are arranged in the order of R physical pixels and G physical pixels. The two physical pixels 138 in the second column are arranged in the order of G physical pixels and B physical pixels.

[0176] exist Figure 9 The diagram shows the addresses of multiple physical pixels 138 in the X and Y directions. For example, address (1, 1) indicates that the address in the X direction is 1 and the address in the Y direction is 1, and address (2, 1) indicates that the address in the X direction is 2 and the address in the Y direction is 1. As an example, each of the multiple physical pixels 138 has a photodiode that performs photoelectric conversion on the received light and outputs an electrical signal corresponding to the amount of light received.

[0177] Next, the camera control unit 90, storage processing unit 92, generation unit 94, detection unit 96, and output unit 100 involved in the second embodiment (see reference) Figure 6 (This will be explained.)

[0178] As an example, such as Figure 10A As shown, the camera control unit 90 controls the actuator 52 of the shake correction mechanism 46 to move the image sensor 130 along the X direction. At this time, the camera control unit 90 controls the movement of the image sensor 130 to obtain the super-resolution image 150 (see reference 150) described later. Figure 10B ).

[0179] For example, corresponding to the image sensor 130 being a color image sensor with color filters of different colors arranged in a Bayer array, the camera control unit 90 controls the movement of the image sensor 130 as follows: That is, when the spacing between the plurality of physical pixels 138 is set to p [μm], the natural number is set to n, and the number of physical pixels per cycle of the Bayer array is set to T, the camera control unit 90 controls the image sensor 130 to move in the X direction with a movement amount of {(T-1)+0.5}×n×p [μm] while keeping the image sensor 130 at a fixed position in the Y direction. For example, if the number of physical pixels T per cycle of the Bayer array is 2, then when n=1, the movement amount is specified as 1.5×p. As an example, the camera control unit 90 controls the image sensor 130 to move with a movement amount of 1.5×p [μm].

[0180] Furthermore, the camera control unit 90 controls the photoelectric conversion element driver 68 to cause the image sensor 130 to capture images before and after movement. Thus, the camera control unit 90 acquires images 140A and 140B captured by the image sensor 130 before and after movement. Image 140A is the image captured by the image sensor 130 before movement, and image 140B is the image captured by the image sensor 130 after movement. Image 140B is the image captured by the image sensor 130 at a position offset by only 1.5 × p [μm] in the X direction relative to image 140A. Figure 10A In the example shown, to easily distinguish between image 140A and image 140B, dotted shading lines are marked on image 140B, which was captured by the moved image sensor 130. Images 140A and 140B are examples of "multiple images" involved in the technology of this invention.

[0181] Image 140A contains multiple image pixels A, and image 140B contains multiple image pixels B. The multiple image pixels A correspond to multiple physical pixels 48 of the image sensor 130 before the movement, and the multiple image pixels B correspond to multiple physical pixels 48 of the image sensor 130 after the movement. Hereinafter, when identifying image pixels A and B by their colors, red image pixels A and B will be referred to as R image pixels A and B, green image pixels A and B as G image pixels A and B, and blue image pixels A and B as B image pixels A and B. Figure 10A The example shown illustrates a portion of multiple image pixels A obtained before the image sensor 130 moves, and a portion of multiple image pixels B obtained after the image sensor 130 moves, i.e., 4×3=12 image pixels A and 4×3=12 image pixels B. For example, image pixels A and B at address (1,1) correspond to physical pixels R at address (1,1), and image pixels A and B at address (2,1) correspond to physical pixels G at address (2,1). Furthermore, for ease of explanation, image pixels A and B will be referred to as "image pixels" without any symbol when there is no need to distinguish between them.

[0182] The storage processing unit 92 performs the processing of storing the images 140A and 140B captured by the image sensor 130 in the image memory 62.

[0183] As an example, such as Figure 10B As shown, the generation unit 94 performs super-resolution transformation on multiple low-resolution images (e.g., images 140A and 140B) acquired by the image sensor 130. Here, super-resolution transformation refers, for example, to the process of generating a high-resolution image (i.e., a higher-resolution image) by synthesizing multiple low-resolution images by staggering the positions of image pixels. The high-resolution image generated in this way is generally also referred to as a super-resolution image.

[0184] exist Figure 10B In the example shown, the generation unit 94 performs super-resolution on images 140A and 140B acquired by the image sensor 130 to generate a super-resolution image 150. Super-resolution using a color-type image sensor 130 with color filters arranged in a Bayer array is achieved, for example, by determining the arrangement order of four image pixels A and B arranged along the X-direction, obtained before and after the movement of the image sensor 130, based on their absolute positions before and after the movement of the image sensor 130. The absolute position is the position relative to the center of image pixel A and the center of image pixel B.

[0185] Next, the plurality of image pixels A contained in the image 140A obtained before the movement of the image sensor 130 and the plurality of image pixels B contained in the image 140B obtained after the movement of the image sensor 130 are arranged according to the above-described arrangement order. Thus, a super-resolution image 150 is obtained by super-resolution of images 140A and 140B. Figure 10B The example shown illustrates a portion of the super-resolution image 150, namely 12 × 2 = 24 image pixels A and B obtained before and after the movement of the image sensor 130.

[0186] As illustrated above, the camera control unit 90 controls the image sensor 130 to move along the X direction by a movement amount of 1.5 × p [μm].

[0187] Thus, as shown in the super-resolution image 150, when the image pixel A corresponding to any physical pixel 138 before the movement is set as the first image pixel A, and other image pixels A different from the first image pixel A are set as the second image pixel A, and the image pixel B corresponding to the aforementioned any physical pixel 138 after the movement is set as the first image pixel B, the first image pixel B and the second image pixel A are adjacent.

[0188] exist Figure 10A and Figure 10B In the example shown, for instance, if the R image pixel A at address (1,1) is set as the first image pixel A, the G image pixel A at address (2,1) is set as the second image pixel A, and the R image pixel B at address (1,1) is set as the first image pixel B, then the R image pixel B at address (1,1), i.e. the first image pixel B, is adjacent to the G image pixel A at address (2,1), i.e. the second image pixel A.

[0189] Each of the plurality of image pixels A and B corresponds to the position of each of the plurality of physical pixels 138 before and after the image sensor 130 moves. Therefore, the super-resolution image 150 represents the physical pixel configuration before and after the image sensor 130 moves. For example, when the physical pixel 138 corresponding to the first image pixel A and the first image pixel B is set as the first physical pixel, and the physical pixel 138 corresponding to the second image pixel A is set as the second physical pixel, the camera control unit 90 performs the following control: in the physical pixel configuration before and after the movement of the image sensor 130 represented by the super-resolution image 150, the image sensor 130 is moved to a position adjacent to the second physical pixel, which is different from the first physical pixel before the movement, where the first physical pixel after the movement is located.

[0190] In other words, the camera control unit 90 moves the image sensor 130 so that the physical pixel configuration of the image sensor 130 before and after the movement, as represented by the super-resolution image 150, is such that adjacent image pixels contained in the super-resolution image 150 become image pixels corresponding to mutually different physical pixels 138.

[0191] As an example of such control, in the above example, the camera control unit 90 controls the image sensor 130 to move along the X direction by a movement amount of 1.5 × p [μm]. As described above, the physical pixel configuration of the image sensor 130 before and after the movement is represented, for example, by the super-resolution image 150, and the above physical pixel configuration can be achieved regardless of whether there is a space equivalent to one physical pixel between the physical pixels of the image sensor 130.

[0192] In addition, Figure 10A and Figure 10B In the example shown, for instance, when the R physical pixel at address (1,1) is set as the first physical pixel before the image sensor 130 moves, the R physical pixel at address (1,1) is equivalent to an example of the "first physical pixel" involved in the technology of the present invention, and the second physical pixel, which is different from the R physical pixel at address (1,1) before the move, i.e., the G physical pixel at address (2,1), is equivalent to an example of the "second physical pixel" involved in the technology of the present invention.

[0193] Furthermore, for example, R image pixel A at address (1,1) is an example of "image pixel corresponding to the first physical pixel before the move" according to the technology of the present invention, G image pixel A at address (2,1) is an example of "image pixel corresponding to a second physical pixel different from the first physical pixel before the move" according to the technology of the present invention, and R image pixel B at address (1,1) is an example of "image pixel corresponding to the first physical pixel after the move" according to the technology of the present invention.

[0194] As an example, such as Figure 10C As shown, the detection unit 96 calculates the difference in pixel values ​​between the multiple image pixels A and multiple image pixels B contained in the super-resolution image 150 and the image pixels A and B corresponding to the second physical pixel before movement and the first physical pixel after movement, respectively. When the image sensor 130 moves along the X direction, it selects image pixels that are adjacent to each other in the X direction. Similarly, when the image sensor 130 moves along the Y direction, it selects image pixels that are adjacent to each other in the Y direction. The pixel value of image pixel A is proportional to the value of the electrical signal output from the physical pixel 138 corresponding to image pixel A, and the pixel value of image pixel B is proportional to the value of the electrical signal output from the physical pixel 138 corresponding to image pixel B.

[0195] The detection unit 96 detects defective physical pixels from a plurality of physical pixels 138 based on the calculated difference degree. A defective physical pixel refers to a physical pixel 138 that has a defect. As an example, the detection unit 96 sequentially calculates the difference degree of pixel values ​​between all image pixels A and B of the detection object (hereinafter referred to as the "detection object") that are defective physical pixels. Figure 10C In the example of image pixels A and B that are the objects of detection, it is shown that the G image pixel A at address (2,1) and the R image pixel B at address (1,1) are highlighted (displayed by circles) and the difference in pixel value between the G image pixel A at address (2,1) and the R image pixel B at address (1,1) is calculated.

[0196] Various methods can be applied to the method for detecting defective physical pixels based on the detection unit 96. For example, the detection unit 96 detects defective physical pixels using the following approach.

[0197] That is, for example, the detection unit 96 calculates the difference between the pixel values ​​of image pixels A and B that are the objects of detection as the difference between the pixel values ​​of image pixels A and B that are the objects of detection. When the calculated difference exceeds a preset threshold, the detection unit 96 acquires the pixel values ​​of each of the image pixels A and B that are the objects of the difference calculation and the pixel values ​​of each of the image pixels A and B surrounding them. For example, when the difference between the pixel values ​​of G image pixel A at address (2,1) and R image pixel B at address (1,1) exceeds the preset threshold, the detection unit 96 acquires the pixel value of G image pixel A at address (2,1) and the pixel value of R image pixel B at address (1,1), and acquires the pixel values ​​of each of the surrounding R image pixel A at address (1,1), R image pixel A at address (3,1), G image pixel B at address (2,1), R image pixel B at address (3,1), and G image pixel B at address (4,1).

[0198] Then, when the arrangement of the acquired multiple pixel values, i.e. the pixel value pattern, is consistent with the abnormal pixel value pattern that predefines the G image pixel A at address (2,1) and its corresponding G image pixel B at address (2,1) as white scratches or black scratches, the detection unit 96 detects that the G physical pixel at address (2,1) corresponding to the G image pixel A at address (2,1) and the G image pixel B at address (2,1) is a defective physical pixel.

[0199] Furthermore, in this case, when the acquired pixel value pattern matches an abnormal pixel value pattern containing white scratches, the detection unit 96 can detect that G image pixel A at address (2,1) and G image pixel B at address (2,1) are white scratches. And, when the acquired pixel value pattern matches an abnormal pixel value pattern containing black scratches, the detection unit 96 can detect that G image pixel A at address (2,1) and G image pixel B at address (2,1) are black scratches.

[0200] Furthermore, when the arrangement of the acquired multiple pixel values, i.e. the pixel value pattern, is consistent with the abnormal pixel value pattern that predefines the R image pixel B at address (1,1) and its corresponding R image pixel A at address (1,1) as white scratches or black scratches, the detection unit 96 detects that the R physical pixel at address (1,1) corresponding to the R image pixel B at address (1,1) and the R image pixel A at address (1,1) is a defective physical pixel.

[0201] Furthermore, in this case, when the acquired pixel value pattern matches an abnormal pixel value pattern containing white scratches, the detection unit 96 can detect that R image pixel B at address (1,1) and R image pixel A at address (1,1) are white scratches. And, when the acquired pixel value pattern matches an abnormal pixel value pattern containing black scratches, the detection unit 96 can detect that R image pixel B at address (1,1) and R image pixel A at address (1,1) are black scratches.

[0202] The correction unit 98 corrects the pixel value of the image pixel corresponding to the defective physical pixel detected by the detection unit 96 based on the pixel value of the image pixel adjacent to that image pixel. Various methods can be applied to correct the pixel value. For example, the correction unit 98 corrects the pixel value using the following method.

[0203] That is, the correction unit 98 performs correction by setting the pixel value of the first image pixel corresponding to the defective physical pixel detected by the detection unit 96 to the average or median value of the pixel values ​​of the second image pixel adjacent to the first image pixel.

[0204] For example, when a G physical pixel at address (2,1) is detected as a defective physical pixel, the correction unit 98 corrects it by setting the G image pixel A at address (2,1) to the average or median value of the pixel values ​​of image pixels A and B adjacent to the G image pixel A at address (2,1), and corrects it by setting the G image pixel B at address (2,1) to the average or median value of the pixel values ​​of image pixels A and B adjacent to the G image pixel B at address (2,1). Similarly, when a R physical pixel at address (1,1) is detected as a defective physical pixel, the correction unit 98 corrects it by setting the R image pixel B at address (1,1) to the average or median value of the pixel values ​​of image pixels A and B adjacent to the R image pixel B at address (1,1), and corrects it by setting the R image pixel A at address (1,1) to the average or median value of the pixel values ​​of image pixels A and B adjacent to the R image pixel A at address (1,1). Thus, by correcting the pixel values ​​of the image pixels corresponding to the defective physical pixels, a corrected super-resolution image 150 can be obtained. For example, when the image pixel corresponding to the defective physical pixel is a white scratch or a black scratch, a super-resolution image 150 can be obtained in a way that the white scratch or black scratch is not noticeable.

[0205] Furthermore, in this example, when the G physical pixel at address (2,1) is detected as a defective physical pixel, the G image pixel A and G image pixel B at address (2,1) correspond to an example of the "first image pixel corresponding to the defective physical pixel" according to the technology of this invention, and each image pixel A and B adjacent to the G image pixel A and G image pixel B at address (2,1) corresponds to an example of the "second image pixel adjacent to the first image pixel" according to the technology of this invention. Also, when the R physical pixel at address (1,1) is detected as a defective physical pixel, the R image pixel B and R image pixel A at address (1,1) correspond to an example of the "first image pixel corresponding to the defective physical pixel" according to the technology of this invention, and the image pixel A adjacent to the R image pixel B at address (1,1) and the image pixel B adjacent to the R image pixel A at address (1,1) correspond to an example of the "second image pixel adjacent to the first image pixel" according to the technology of this invention.

[0206] The output unit 100 performs the process of outputting the corrected super-resolution image 150 to the display 26. Thus, the super-resolution image 150 is displayed on the display 26 while the pixel values ​​of the image pixels corresponding to the defective physical pixels are corrected. For example, when the image pixel corresponding to the defective physical pixel is a white scratch or a black scratch, the super-resolution image 150, corrected in a way that makes the white or black scratch inconspicuous, is displayed on the display 26.

[0207] Furthermore, the operation of the camera device 10 in the second embodiment is the same as that in the first embodiment.

[0208] As explained above, in the imaging device 10 according to the second embodiment, control is performed such that, in the physical pixel configuration before and after the image sensor 130 having a plurality of physical pixels 138 is moved, the image sensor 130 is moved to a position adjacent to a second physical pixel that is different from any first physical pixel before the movement in any of the moved first physical pixels 138. Furthermore, the image sensor 130 is controlled by the imaging control unit 90 via the photoelectric conversion element driver 68 to capture images before and after the movement using the image sensor 130. Then, based on the difference in pixel values ​​between the image pixel A corresponding to the second physical pixel before the movement and the image pixel B corresponding to the first physical pixel after the movement in each of the plurality of image pixels A and B included in each of the images 140A and 140B captured by the image sensor 130, defective physical pixels are detected from the plurality of physical pixels 138. Therefore, defective physical pixels can be detected from the plurality of physical pixels 138 based on the images 140A and 140B captured by the image sensor 130 before and after the movement.

[0209] Furthermore, in the imaging device 10 according to the second embodiment, the images 140A and 140B captured by the image sensor 130 are stored in the image memory 62. Then, defective physical pixels are detected from the plurality of physical pixels 138 based on the difference in pixel values ​​between the plurality of image pixels A and the plurality of image pixels B contained in each of the images 140A and 140B stored in the image memory 62. Therefore, defective physical pixels can be detected from the plurality of physical pixels 138 based on the images 140A and 140B stored in the image memory 62.

[0210] Furthermore, in the imaging device 10 according to the second embodiment, the pixel value of the first image pixel corresponding to the defective physical pixel is corrected based on the pixel value of the second image pixel adjacent to the first image pixel. Therefore, the defect of the image pixel corresponding to the defective physical pixel can be corrected.

[0211] Furthermore, in the imaging device 10 according to the second embodiment, as an example, the image sensor 130 is a color-type image sensor in which color filters of different colors are distributed on a periodic substrate arrangement of a plurality of physical pixels 138. Therefore, defective physical pixels can be detected from the plurality of physical pixels 138 included in the color-type image sensor 130.

[0212] Furthermore, in the imaging device 10 according to the second embodiment, as an example, when the spacing between the plurality of physical pixels 138 is set to p, the imaging control unit 90 controls the image sensor 130 to move in the X direction by a movement amount of 1.5 × p [μm]. Therefore, in the physical pixel configuration before and after the movement of the color image sensor 130 with Bayer arrangement, it is possible to position the moved first physical pixel adjacent to a second physical pixel that is different from the moved first physical pixel.

[0213] Furthermore, the camera device 10 according to the second embodiment includes a controller 60 for detecting defective physical pixels, an image sensor 130, and a jitter correction mechanism 46 for moving the image sensor 130. Therefore, in the camera device 10, it is possible to continuously perform image capture based on the image sensor 130, move the image sensor 130, and detect defective physical pixels.

[0214] Furthermore, in the second embodiment described above, the camera control unit 90 moves the image sensor 130 along the X direction, but it can also move it along the Y direction. Moreover, the camera control unit 90 can move the image sensor 130 along both the X and Y directions. Furthermore, even when the image sensor 130 moves along the Y direction or along both the X and Y directions, the processing of detecting defective physical pixels from the plurality of physical pixels 138 and the processing of correcting the pixel value of the first image pixel corresponding to the defective physical pixel based on the pixel value of the second image pixel adjacent to the first image pixel can be performed using the same method as in the second embodiment described above.

[0215] Furthermore, in the second embodiment described above, the camera control unit 90 moves the image sensor 130 by a movement amount of 1.5 × p [μm]. However, when the spacing between the plurality of physical pixels 138 is set to 1, the camera control unit 90 can also move the image sensor 130 along the X or Y direction by a movement amount greater than 1, provided that the super-resolution image 150 is obtained. With this structure, in the physical pixel configuration before and after the movement of the image sensor 130, it is also possible to position the moved first physical pixel adjacent to a second physical pixel that is different from the first physical pixel before the movement.

[0216] Furthermore, in the second embodiment described above, when the natural number is set to n and the decimal number is set to d, the camera control unit 90 can move the image sensor 130 along the X or Y direction by a movement amount of (n+d)×p[μm] while obtaining the super-resolution image 150. With this structure, in the physical pixel configuration before and after the movement of the image sensor 130, it is also possible to position the moved first physical pixel adjacent to a second physical pixel that is different from the first physical pixel before the movement.

[0217] Furthermore, in the second embodiment described above, when the natural number is set to n and the number of physical pixels per cycle of the substrate arrangement is set to T, the camera control unit 90 can move the image sensor 130 along the X or Y direction by a movement amount of {(T-1)+0.5}×n×p[μm]. With this structure, in the physical pixel configuration before and after the movement of the image sensor 130, it is also possible to position the moved first physical pixel adjacent to a second physical pixel that is different from the first physical pixel before the movement.

[0218] Furthermore, if the camera control unit 90 moves the image sensor 130 by a movement amount of 1.5×p [μm], for example, compared to the case where the image sensor 130 is moved by a small amount greater than 1.5×p or a small amount less than 1.5×p and greater than 1.0×p [μm], a high-resolution super-resolution image 150 can be obtained.

[0219] Furthermore, in the second embodiment described above, the output unit 100 can output position information corresponding to the position of the defective physical pixel. This position information is, for example, information corresponding to the address of the physical pixel 138. According to this structure, the position of the defective physical pixel can be determined based on the position information corresponding to the output position of the defective physical pixel.

[0220] Furthermore, in the second embodiment described above, the detection unit 96 can extract image pixels whose pixel values ​​deviate from a predetermined range from a plurality of image pixels, and detect defective physical pixels based on the difference between the pixel value of the image pixel outside the predetermined range and the pixel value of the image pixels adjacent to the image pixel outside the predetermined range. According to this structure, compared to calculating the difference between the pixel values ​​of the image pixels corresponding to all the image pixels before and after the movement of the second physical pixel, the number of times the difference is calculated can be reduced.

[0221] Furthermore, in the second embodiment described above, the detection unit 96 detects defective physical pixels from a plurality of physical pixels 138 during the generation of the super-resolution image 150, but the technology of the present invention is not limited to this. For example, in a process different from the generation of the super-resolution image 150, the detection unit 96 may also detect defective physical pixels from a plurality of physical pixels 138 using the same method as described above, based on images 140A and 140B obtained by the image sensor 130 before and after movement.

[0222] In addition, Figure 17 The image shown is a super-resolution image 450 obtained through a comparative example relative to the second embodiment. This super-resolution image 450 is obtained by using the image sensor 130 described above (reference 2). Figure 10AThe image is obtained by moving the sensor 130 by a distance of 0.5 × p [μm] instead of 1.5 × p [μm]. In this case, in the physical pixel configuration before and after the movement of the image sensor 130, the first physical pixel after the movement is located adjacent to the first physical pixel before the movement. Therefore, the pixel values ​​between adjacent image pixels in the super-resolution image 450 are the same, and thus it is impossible to detect defective physical pixels from multiple physical pixels.

[0223] [Third Implementation]

[0224] Next, the third embodiment will be described. In the third embodiment, the structure of the camera device 10 is changed from that of the first embodiment as follows. Furthermore, in the third embodiment, the same reference numerals are used for elements and components that are the same as in the first embodiment, and detailed descriptions are omitted. Regarding the overall structure of the camera device 10 according to the third embodiment, please refer to... Figures 1-5 .

[0225] As an example, such as Figure 11 As shown, in the third embodiment, a color image sensor 160 is used. The image sensor 160 has a plurality of physical pixels 168. As an example, the plurality of physical pixels 168 are arranged along the X and Y directions. As an example, the X-direction spacing between the plurality of physical pixels 168 is the same, and the Y-direction spacing between the plurality of physical pixels 168 is the same. Furthermore, as an example, the X-direction spacing between the plurality of physical pixels 168 is the same as the Y-direction spacing between the plurality of physical pixels 168. Alternatively, the X-direction spacing between the plurality of physical pixels 168 may be different from the Y-direction spacing between the plurality of physical pixels 168.

[0226] exist Figure 11 In the example shown, a portion of the multiple physical pixels 168, namely 7×7=49 physical pixels 168, are illustrated. Different color filters are assigned to the multiple physical pixels 168 in a periodic basal arrangement. As an example of a basal arrangement, an X-Trans (trademarked) type arrangement is used in the image sensor 160.

[0227] The X-Trans (registered trademark) arrangement, for example, refers to an arrangement of 6×6=36 color filters as a whole. These 6×6=36 color filters consist of 8 filters corresponding to the red wavelength region, 20 filters corresponding to the green wavelength region, and 8 filters corresponding to the blue wavelength region. Hereinafter, when identifying multiple physical pixels 168 using color filters assigned to them, the physical pixel 168 assigned to the red wavelength region is referred to as the R physical pixel, the physical pixel 168 assigned to the green wavelength region is referred to as the G physical pixel, and the physical pixel 168 assigned to the blue wavelength region is referred to as the B physical pixel.

[0228] The 6 x 6 = 36 physical pixels (168) are arranged as follows: The first column (6 physical pixels) is arranged in the order of G physical pixel, R physical pixel, B physical pixel, G physical pixel, B physical pixel, and R physical pixel. The second column (6 physical pixels) is arranged in the order of B physical pixel, G physical pixel, G physical pixel, R physical pixel, G physical pixel, and G physical pixel. The third column (6 physical pixels) is arranged in the order of R physical pixel, G physical pixel, G physical pixel, B physical pixel, G physical pixel, and G physical pixel. The fourth column (6 physical pixels) is arranged in the order of G physical pixel, B physical pixel, R physical pixel, G physical pixel, R physical pixel, and B physical pixel. The fifth column (6 physical pixels) is arranged in the order of R physical pixel, G physical pixel, G physical pixel, B physical pixel, G physical pixel, and G physical pixel. The 6 physical pixels in column 6 are arranged in the order of B physical pixel, G physical pixel, G physical pixel, R physical pixel, G physical pixel, and G physical pixel.

[0229] exist Figure 11 The diagram shows the addresses of multiple physical pixels 168 in the X and Y directions. For example, address (1, 1) indicates that the address in the X direction is 1 and the address in the Y direction is 1, and address (2, 1) indicates that the address in the X direction is 2 and the address in the Y direction is 1. As an example, each of the multiple physical pixels 168 has a photodiode that performs photoelectric conversion on the received light and outputs an electrical signal corresponding to the amount of light received.

[0230] Next, the camera control unit 90, storage processing unit 92, generation unit 94, detection unit 96, and output unit 100 according to the third embodiment (see reference) Figure 6 (This will be explained.)

[0231] As an example, such as Figure 12AAs shown, the camera control unit 90 controls the actuator 52 of the shake correction mechanism 46 to move the image sensor 160 in the X direction. At this time, the camera control unit 90 controls the movement of the image sensor 160 to obtain the super-resolution image 180 (see reference 180) described later. Figure 12B ).

[0232] For example, corresponding to the image sensor 160 being a color image sensor with color filters of different colors arranged in an X-Trans (trademark) pattern, the camera control unit 90 controls the movement of the image sensor 160 as follows: That is, when the spacing between the plurality of physical pixels 168 is set to p [μm], the natural number is set to n, and the number of physical pixels per cycle of the X-Trans (trademark) arrangement is set to T, the camera control unit 90 controls the image sensor 160 to move in the X direction with a movement amount of {(T-1)+0.5}×n×p [μm] while keeping the position of the image sensor 160 fixed in the Y direction. For example, if the number of physical pixels T per cycle of the X-Trans (trademark) arrangement is 6, then when n=1, the movement amount is specified as 5.5×p. As an example, the camera control unit 90 controls the image sensor 160 to move with a movement amount of 5.5×p [μm].

[0233] Furthermore, the camera control unit 90 controls the photoelectric conversion element driver 68 to cause the image sensor 160 to capture images before and after movement. Thus, the camera control unit 90 acquires images 170A and 170B captured by the image sensor 160 before and after movement. Image 170A is the image captured by the image sensor 160 before movement, and image 170B is the image captured by the image sensor 160 after movement. Image 170B is the image captured by the image sensor 160 at a position offset by only 5.5 × p [μm] in the X direction relative to image 170A. Figure 12A In the example shown, to easily distinguish between image 170A and image 170B, dotted shading lines are marked on image 170B, which was captured by the moved image sensor 160. Images 170A and 170B are examples of "multiple images" involved in the technology of this invention.

[0234] Image 170A contains multiple image pixels A, and image 170B contains multiple image pixels B. The multiple image pixels A correspond to multiple physical pixels 168 of the image sensor 160 before the movement, and the multiple image pixels B correspond to the multiple physical pixels 168 of the image sensor 160 after the movement. Figure 12AThe example shown illustrates a portion of multiple image pixels A obtained before the image sensor 160 moves and a portion of multiple image pixels B obtained after the image sensor 160 moves, i.e., 7 × 3 = 21 image pixels A and 7 × 3 = 21 image pixels B. For example, image pixels A and B at address (1, 1) correspond to physical pixels G at address (1, 1), and image pixels A and B at address (6, 1) correspond to physical pixels B at address (6, 1). Furthermore, for ease of explanation, image pixels A and B will be referred to as "image pixels" without any symbol when there is no need to distinguish between them.

[0235] The storage processing unit 92 performs the processing of storing the images 170A and 170B captured by the image sensor 160 in the image memory 62.

[0236] As an example, such as Figure 12B As shown, the generation unit 94 performs super-resolution transformation on multiple low-resolution images (e.g., images 170A and 170B) acquired by the image sensor 160. Here, super-resolution transformation refers, for example, to the process of generating a high-resolution image (i.e., a higher-resolution image) by synthesizing multiple low-resolution images by staggering the positions of image pixels. The high-resolution image generated in this way is generally also referred to as a super-resolution image.

[0237] exist Figure 12B In the example shown, the generation unit 94 performs super-resolution on images 170A and 170B acquired by the image sensor 160 to generate a super-resolution image 180. Super-resolution using a color-type image sensor 160 with color filters of different colors arranged in an X-Trans (trademarked) pattern is achieved, for example, by determining the arrangement order of seven image pixels A and B arranged along the X-direction, obtained before and after the movement of the image sensor 160, based on their absolute positions before and after the movement of the image sensor 160. The absolute position is the position relative to the center of image pixel A and the center of image pixel B.

[0238] Next, the plurality of image pixels A contained in image 170A obtained before the movement of image sensor 160 and the plurality of image pixels B contained in image 170B obtained after the movement of image sensor 160 are arranged according to the above arrangement order. Thus, a super-resolution image 180 is obtained by super-resolution of images 170A and 170B. Figure 12B The example shown illustrates a portion of the super-resolution image 180, namely 42 × 2 = 84 image pixels A and B obtained before and after the movement of the image sensor 160.

[0239] As illustrated above, the camera control unit 90 controls the image sensor 160 to move along the X direction by a movement amount of 5.5 × p [μm]. Thus, as shown in the super-resolution image 180, when image pixel A corresponding to any physical pixel 168 before the movement is designated as the first image pixel A, other image pixels A different from the first image pixel A are designated as the second image pixel A, and image pixel B corresponding to the aforementioned random physical pixel 168 after the movement is designated as the first image pixel B, the first image pixel B and the second image pixel A are adjacent.

[0240] exist Figure 12A and Figure 12B In the example shown, for instance, if the G image pixel A at address (1,1) is set as the first image pixel A, the B image pixel A at address (6,1) is set as the second image pixel A, and the G image pixel B at address (1,1) is set as the first image pixel B, then the G image pixel B at address (1,1), i.e. the first image pixel B, is adjacent to the B image pixel A at address (6,1), i.e. the second image pixel A.

[0241] Each of the plurality of image pixels A and B corresponds to the position of each of the plurality of physical pixels 168 before and after the image sensor 160 moves. Therefore, the super-resolution image 180 represents the physical pixel configuration before and after the image sensor 160 moves. For example, when the physical pixel 168 corresponding to the first image pixel A and the first image pixel B is set as the first physical pixel, and the physical pixel 168 corresponding to the second image pixel A is set as the second physical pixel, the camera control unit 90 performs the following control: in the physical pixel configuration before and after the movement of the image sensor 160 represented by the super-resolution image 180, the image sensor 160 is moved to a position adjacent to the second physical pixel, which is different from the first physical pixel before the movement, where the first physical pixel after the movement is located.

[0242] In other words, the camera control unit 90 moves the image sensor 160 so that the physical pixel configuration of the image sensor 160 before and after the movement, as represented by the super-resolution image 180, is such that adjacent image pixels contained in the super-resolution image 180 become image pixels corresponding to mutually different physical pixels 168.

[0243] As an example of such control, in the above example, the camera control unit 90 controls the image sensor 160 to move along the X direction by a movement amount of 5.5 × p [μm]. As described above, the physical pixel configuration of the image sensor 160 before and after the movement is represented, for example, by the super-resolution image 180, and the above physical pixel configuration can be achieved regardless of whether there is a space equivalent to one physical pixel between the physical pixels of the image sensor 160.

[0244] In addition, Figure 12A and Figure 12B In the example shown, for instance, when the G physical pixel at address (1,1) is set as the first physical pixel before the image sensor 160 moves, the G physical pixel at address (1,1) is equivalent to an example of the "first physical pixel" involved in the technology of the present invention, and the second physical pixel, which is different from the G physical pixel at address (1,1) before the move, i.e., the B physical pixel at address (6,1), is equivalent to an example of the "second physical pixel" involved in the technology of the present invention.

[0245] Furthermore, for example, image pixel A at address (1,1) is an example of "image pixel corresponding to the first physical pixel before the move" according to the technology of the present invention, image pixel A at address (6,1) is an example of "image pixel corresponding to a second physical pixel different from the first physical pixel before the move" according to the technology of the present invention, and image pixel B at address (1,1) is an example of "image pixel corresponding to the first physical pixel after the move" according to the technology of the present invention.

[0246] As an example, such as Figure 12C As shown, the detection unit 96 calculates the difference in pixel values ​​between the multiple image pixels A and multiple image pixels B contained in the super-resolution image 180 and the image pixels A and B corresponding to the second physical pixel before movement and the first physical pixel after movement, respectively. When the image sensor 160 moves along the X direction, it selects image pixels that are adjacent to each other in the X direction. Similarly, when the image sensor 160 moves along the Y direction, it selects image pixels that are adjacent to each other in the Y direction. The pixel value of image pixel A is proportional to the value of the electrical signal output from the physical pixel 168 corresponding to image pixel A, and the pixel value of image pixel B is proportional to the value of the electrical signal output from the physical pixel 168 corresponding to image pixel B.

[0247] The detection unit 96 detects defective physical pixels from a plurality of physical pixels 168 based on the calculated difference degree. A defective physical pixel refers to a physical pixel 168 that has a defect. As an example, the detection unit 96 sequentially calculates the difference degree of pixel values ​​between all image pixels A and B of the detection object (hereinafter referred to as the "detection object") that constitute a defective physical pixel. Figure 12C In the example of image pixels A and B that are the objects of detection, it is shown that the B image pixel A at address (6,1) and the G image pixel B at address (1,1) are highlighted (displayed by circles) and the difference in pixel value between the B image pixel A at address (6,1) and the G image pixel B at address (1,1) is calculated.

[0248] Various methods can be applied to the method for detecting defective physical pixels based on the detection unit 96. For example, the detection unit 96 detects defective physical pixels using the following approach.

[0249] That is, for example, the detection unit 96 calculates the subtraction or division value between the pixel values ​​of image pixels A and B that are the objects of detection as the difference degree between the pixel values ​​of image pixels A and B that are the objects of detection. When the calculated difference degree exceeds a preset threshold, the pixel values ​​of image pixels A and B that are the objects of difference degree calculation and the pixel values ​​of image pixels A and B around them are obtained respectively.

[0250] That is, for example, the detection unit 96 calculates the subtraction or division value between the pixel values ​​of image pixels A and B that are the objects of detection as the difference degree between the pixel values ​​of image pixels A and B that are the objects of detection. When the calculated difference degree exceeds a preset threshold, the pixel values ​​of image pixels A and B that are the objects of difference degree calculation and the pixel values ​​of image pixels A and B around them are obtained respectively.

[0251] For example, when the difference in pixel value between B image pixel A at address (6,1) and G image pixel B at address (1,1) exceeds a preset threshold, the detection unit 96 acquires the pixel values ​​of B image pixel A at address (6,1) and G image pixel B at address (1,1), and acquires the pixel values ​​of each of the surrounding G image pixel A at address (1,1), B image pixel A at address (2,1), R image pixel A at address (3,1), G image pixel A at address (4,1), R image pixel A at address (5,1), G image pixel A at address (7,1), B image pixel B at address (2,1), R image pixel B at address (3,1), G image pixel B at address (4,1), R image pixel B at address (5,1), R image pixel B at address (6,1), and G image pixel B at address (7,1).

[0252] Then, when the arrangement of the acquired multiple pixel values, i.e. the pixel value pattern, is consistent with the abnormal pixel value pattern that predefines the B image pixel A at address (6,1) and its corresponding B image pixel B at address (6,1) as white scratches or black scratches, the detection unit 96 detects that the physical pixel B at address (6,1) corresponding to the B image pixel A at address (6,1) and the B image pixel B at address (6,1) is a defective physical pixel.

[0253] Furthermore, in this case, when the acquired pixel value pattern matches an abnormal pixel value pattern containing white scratches, the detection unit 96 can detect that B image pixel A at address (6,1) and B image pixel B at address (6,1) are white scratches. And, when the acquired pixel value pattern matches an abnormal pixel value pattern containing black scratches, the detection unit 96 can detect that B image pixel A at address (6,1) and B image pixel B at address (6,1) are black scratches.

[0254] Furthermore, when the arrangement of the acquired multiple pixel values, i.e. the pixel value pattern, is consistent with the abnormal pixel value pattern predetermined as a white scratch or a black scratch, the detection unit 96 detects that the physical pixel at address (1,1) corresponding to the physical pixel at address (1,1) of ...

[0255] Furthermore, in this case, when the acquired pixel value pattern matches an abnormal pixel value pattern containing white scratches, the detection unit 96 can detect that image pixel B at address (1,1) and image pixel A at address (1,1) are white scratches. And, when the acquired pixel value pattern matches an abnormal pixel value pattern containing black scratches, the detection unit 96 can detect that image pixel B at address (1,1) and image pixel A at address (1,1) are black scratches.

[0256] The correction unit 98 corrects the pixel value of the image pixel corresponding to the defective physical pixel detected by the detection unit 96 based on the pixel value of the image pixel adjacent to that image pixel. Various methods can be applied to correct the pixel value. For example, the correction unit 98 corrects the pixel value using the following method.

[0257] That is, the correction unit 98 performs correction by setting the pixel value of the first image pixel corresponding to the defective physical pixel detected by the detection unit 96 to the average or median value of the pixel values ​​of the second image pixels adjacent to the first image pixel. For example, when the B physical pixel at address (6,1) is detected as a defective physical pixel, the correction unit 98 performs correction by setting the B image pixel A at address (6,1) to the average or median value of the pixel values ​​of the image pixels A and B adjacent to the B image pixel A at address (6,1), and performs correction by setting the B image pixel B at address (6,1) to the average or median value of the pixel values ​​of the image pixels A and B adjacent to the B image pixel B at address (6,1). Furthermore, for example, when the G physical pixel at address (1,1) is detected as a defective physical pixel, the correction unit 98 performs correction by setting the G image pixel B at address (1,1) to the average or median value of the pixel values ​​of the image pixels A and B adjacent to the G image pixel B at address (1,1), and performs correction by setting the G image pixel A at address (1,1) to the average or median value of the pixel values ​​of the image pixels A and B adjacent to the G image pixel A at address (1,1). In this way, by correcting the pixel values ​​of the image pixels corresponding to the defective physical pixel, a corrected super-resolution image 180 can be obtained. For example, when the image pixel corresponding to the defective physical pixel is a white scratch or a black scratch, a super-resolution image 180 can be obtained in a way that the white scratch or black scratch is not noticeable.

[0258] Furthermore, in this example, when the physical pixel B at address (6,1) is detected as a defective physical pixel, the image pixel A at address (6,1) and the image pixel B at address (6,1) correspond to an example of the "first image pixel corresponding to the defective physical pixel" according to the technology of this invention. Each adjacent image pixel A and B of the image pixel A at address (6,1) and the image pixel B at address (6,1) corresponds to an example of the "second image pixel adjacent to the first image pixel" according to the technology of this invention. Also, when the physical pixel G at address (1,1) is detected as a defective physical pixel, the image pixel B at address (1,1) and the image pixel A at address (1,1) correspond to an example of the "first image pixel corresponding to the defective physical pixel" according to the technology of this invention. The image pixel A adjacent to the image pixel B at address (1,1) and the image pixel B adjacent to the image pixel A at address (1,1) correspond to an example of the "second image pixel adjacent to the first image pixel" according to the technology of this invention.

[0259] The output unit 100 performs the process of outputting the corrected super-resolution image 180 to the display 26. Thus, the super-resolution image 180 is displayed on the display 26 while the pixel values ​​of the image pixels corresponding to the defective physical pixels are corrected. For example, when the image pixel corresponding to the defective physical pixel is a white scratch or a black scratch, the super-resolution image 180, corrected in a way that makes the white or black scratch inconspicuous, is displayed on the display 26.

[0260] Furthermore, the operation of the camera device 10 in the third embodiment is the same as that in the first embodiment.

[0261] As explained above, in the imaging device 10 according to the third embodiment, control is performed such that, in the physical pixel configuration before and after the movement of the image sensor 160 having a plurality of physical pixels 168, the image sensor 160 is moved to a position adjacent to a second physical pixel that is different from any first physical pixel before the movement in any of the moved first physical pixels 168. Furthermore, the image sensor 160 is controlled by the imaging control unit 90 via the photoelectric conversion element driver 68 to capture images before and after the movement using the image sensor 160. Then, based on the difference in pixel values ​​between the image pixel A corresponding to the second physical pixel before the movement and the image pixel B corresponding to the first physical pixel after the movement in each of the plurality of image pixels A and B included in the images 170A and 170B captured by the image sensor 160, defective physical pixels are detected from the plurality of physical pixels 168. Therefore, defective physical pixels can be detected from the plurality of physical pixels 168 based on the images 170A and 170B captured before and after the movement of the image sensor 160.

[0262] Furthermore, in the imaging device 10 according to the third embodiment, the images 170A and 170B captured by the image sensor 160 are stored in the image memory 62. Then, defective physical pixels are detected from the plurality of physical pixels 168 based on the difference in pixel values ​​between the plurality of image pixels A and the plurality of image pixels B contained in each of the images 170A and 170B stored in the image memory 62. Therefore, defective physical pixels can be detected from the plurality of physical pixels 168 based on the images 170A and 170B stored in the image memory 62.

[0263] Furthermore, in the imaging device 10 according to the third embodiment, the pixel value of the first image pixel corresponding to the defective physical pixel is corrected based on the pixel value of the second image pixel adjacent to the first image pixel. Therefore, the defect of the image pixel corresponding to the defective physical pixel can be corrected.

[0264] Furthermore, in the imaging device 10 according to the third embodiment, as an example, the image sensor 160 is a color-type image sensor 160 in which color filters of different colors are distributed on a periodic substrate arrangement of a plurality of physical pixels 168. Therefore, defective physical pixels can be detected from the plurality of physical pixels 168 included in the color-type image sensor 160.

[0265] Furthermore, in the imaging device 10 according to the third embodiment, as an example, when the spacing between the plurality of physical pixels 168 is set to p, the imaging control unit 90 controls the image sensor 160 to move in the X direction by a movement amount of 5.5 × p [μm]. Therefore, in the physical pixel configuration before and after the movement of the color image sensor 160 with an X-Trans (registered trademark) type arrangement, it is possible to position the moved first physical pixel adjacent to a second physical pixel that is different from the moved first physical pixel.

[0266] Furthermore, the camera device 10 according to the third embodiment includes a controller 60 for detecting defective physical pixels, an image sensor 160, and a jitter correction mechanism 46 for moving the image sensor 160. Therefore, in the camera device 10, it is possible to continuously perform image capture based on the image sensor 160, move the image sensor 160, and detect defective physical pixels.

[0267] Furthermore, in the third embodiment described above, the camera control unit 90 moves the image sensor 160 along the X direction, but it can also move it along the Y direction. Moreover, the camera control unit 90 can move the image sensor 160 along both the X and Y directions. Furthermore, even when the image sensor 160 moves along the Y direction or along both the X and Y directions, the processing of detecting defective physical pixels from the plurality of physical pixels 168 and the processing of correcting the pixel value of the first image pixel corresponding to the defective physical pixel based on the pixel value of the second image pixel adjacent to the first image pixel can be performed using the same method as in the third embodiment described above.

[0268] Furthermore, in the third embodiment described above, the camera control unit 90 moves the image sensor 160 by a movement amount of 5.5 × p [μm]. However, when the spacing between the plurality of physical pixels 168 is set to 1, the camera control unit 90 can also move the image sensor 160 by a movement amount greater than 1 in the X or Y direction while obtaining the super-resolution image 180 described above. With this structure, in the physical pixel configuration before and after the movement of the image sensor 160, it is also possible to position the moved first physical pixel adjacent to a second physical pixel that is different from the first physical pixel before the movement.

[0269] Furthermore, in the third embodiment described above, when the natural number is set to n and the decimal number is set to d, the camera control unit 90 can move the image sensor 160 along the X or Y direction by a movement amount of (n+d)×p[μm] while obtaining the super-resolution image 180. With this structure, in the physical pixel configuration before and after the movement of the image sensor 160, it is also possible to position the moved first physical pixel adjacent to a second physical pixel that is different from the first physical pixel before the movement.

[0270] Furthermore, in the third embodiment described above, when the natural number is set to n and the number of physical pixels per cycle of the substrate arrangement is set to T, the camera control unit 90 can move the image sensor 160 along the X or Y direction by a movement amount of {(T-1)+0.5}×n×p[μm]. With this structure, in the physical pixel configuration before and after the movement of the image sensor 160, it is also possible to position the moved first physical pixel adjacent to a second physical pixel that is different from the first physical pixel before the movement.

[0271] Furthermore, if the camera control unit 90 moves the image sensor 160 by a movement amount of 5.5 × p [μm], for example, compared to the case where the image sensor 160 is moved by a small number greater than 5.5 × p or a small number less than 5.5 × p but greater than 5.0 × p, a high-resolution super-resolution image 180 can be obtained.

[0272] Furthermore, in the third embodiment described above, the output unit 100 can output position information corresponding to the position of the defective physical pixel. This position information is, for example, information corresponding to the address of the physical pixel 168. According to this structure, the position of the defective physical pixel can be determined based on the position information corresponding to the output position of the defective physical pixel.

[0273] Furthermore, in the third embodiment described above, the detection unit 96 can extract image pixels whose pixel values ​​deviate from a predetermined range from a plurality of image pixels, and detect defective physical pixels based on the difference between the pixel value of the image pixel outside the predetermined range and the pixel value of the image pixels adjacent to the image pixel outside the predetermined range. According to this structure, compared to calculating the difference between the pixel values ​​of the image pixels corresponding to all the image pixels before and after the movement of the second physical pixel, the number of times the difference is calculated can be reduced.

[0274] Furthermore, in the third embodiment described above, the detection unit 96 detects defective physical pixels from a plurality of physical pixels 168 during the generation of the super-resolution image 180, but the technology of the present invention is not limited to this. For example, in a process different from the generation of the super-resolution image 180, the detection unit 96 may also detect defective physical pixels from a plurality of physical pixels 168 using the same method as described above, based on images 170A and 170B obtained by the image sensor 160 before and after the movement.

[0275] In addition, Figure 18 The image shown is a super-resolution image 480 obtained through a comparative example relative to the third embodiment. This super-resolution image 480 is obtained by making the image sensor 160 (reference) Figure 12A The image is obtained by moving the sensor 160 by a distance of 0.5 × p [μm] (instead of moving it by 5.5 × p [μm]). In this case, in the physical pixel configuration before and after the movement of the image sensor 160, the first physical pixel after the movement is located adjacent to the first physical pixel before the movement. Therefore, the pixel values ​​between adjacent image pixels in the super-resolution image 480 are the same, and thus it is impossible to detect defective physical pixels from multiple physical pixels.

[0276] [Fourth Implementation]

[0277] Next, the fourth embodiment will be described. In the fourth embodiment, the structure of the camera device 10 is changed from that of the second embodiment as follows. Furthermore, in the fourth embodiment, the same symbols are used for the same components and parts as in the second embodiment, and detailed descriptions are omitted. Regarding the overall structure of the camera device 10 according to the fourth embodiment, please refer to... Figures 1-5 and Figure 9 .

[0278] In the fourth embodiment, as an example of a color-type image sensor 130, an image sensor 130 (see reference 130) uses color filters that are arranged in a Bayer arrangement for multiple physical pixels with different colors. Figure 9 ).

[0279] The fourth embodiment includes a camera control unit 90, a storage processing unit 92, a generation unit 94, a detection unit 96, and an output unit 100 (see reference). Figure 6 It consists of the following components.

[0280] As an example, such as Figures 13A-13C As shown, the camera control unit 90 controls the movement of the image sensor 130 and controls the image sensor 130 to capture images before and after the movement, so as to obtain multiple monochrome super-resolution images 200R, 200G, and 200B (see reference) described later. Figure 13B ).exist Figure 13Aand Figure 13B In the diagram, "R" represents a physical pixel assigned to a red filter and its corresponding red image pixel, "G" represents a physical pixel assigned to a green filter and its corresponding green image pixel, and "B" represents a physical pixel assigned to a blue filter and its corresponding blue image pixel.

[0281] As an example, such as Figure 13A As shown, the camera control unit 90 controls the image sensor 130 to capture an image before movement. This results in image 1-1 corresponding to the arrangement of multiple color filters. Next, the camera control unit 90 controls the image sensor 130 to move in the +X direction by an amount of 1.0 × p [μm], and controls the image sensor 130 to capture an image after movement. This results in image 1-2. Next, the camera control unit 90 controls the image sensor 130 to move in the -Y direction by an amount of 1.0 × p [μm], and controls the image sensor 130 to capture an image after movement. This results in image 1-3. Next, the camera control unit 90 controls the image sensor 130 to move in the -X direction by an amount of 1.0 × p [μm], and controls the image sensor 130 to capture an image after movement. This results in image 1-4. Figure 13A The images shown are portions of images 1-1 to 1-4, which are images with a total of 4×4=16 pixels obtained before and after the movement of image sensor 130.

[0282] Images 1-1 to 1-4 thus obtained represent the physical pixel configuration of the image sensor 130 before and after movement. As described above, the camera control unit 90 controls the image sensor 130 to move in the X or Y direction by a movement amount of 1.0 × p [μm]. Therefore, for example, in images 1-1 to 1-4, when any physical pixel before movement is set as the third physical pixel before movement, and a physical pixel before movement with a color filter having the same color as that third physical pixel before movement is set as the fourth physical pixel, the third physical pixel after movement overlaps with the fourth physical pixel before movement. For example, in... Figure 13A In the example shown, the green physical pixel at position 1-1 in image 1-1 overlaps with other green physical pixels at position 1-3 in image 1-3. Similarly, in Figure 13A In the example shown, the green physical pixels at position 1-2 in the image overlap with other green physical pixels at position 1-4 in the image.

[0283] Thus, the camera control unit 90 controls the movement of the image sensor 130 in the physical pixel configuration before and after the movement, so that the image sensor 130 is moved to a position where the third physical pixel after the movement overlaps with the fourth physical pixel which is assigned the same color as the third physical pixel before the movement, so as to obtain images 1-1 to 1-4.

[0284] exist Figure 13A In the example shown, when the green physical pixel at position 1-1 of the obtained image is set as an arbitrary physical pixel before the image sensor 130 moves, this green physical pixel corresponds to an example of the "third physical pixel" involved in the technology of this invention, and the other green physical pixels at position 1-3 of the obtained image correspond to an example of the "fourth physical pixel" involved in the technology of this invention. Furthermore, in Figure 13B In the example shown, when the green physical pixel at the position of image 1-2 is set as any physical pixel before the image sensor 130 moves, the green physical pixel is equivalent to an example of the "third physical pixel" involved in the technology of the present invention, and the other green physical pixels at the position of image 1-4 are equivalent to an example of the "fourth physical pixel" involved in the technology of the present invention.

[0285] Furthermore, for example, corresponding to the image sensor 130 being a color image sensor with color filters of different colors arranged in a Bayer array, the camera control unit 90 controls the movement of the image sensor 130 as follows: That is, when the spacing between multiple physical pixels is set to p [μm], a natural number greater than 2 is set to m, and a decimal number is set to d, the camera control unit 90 controls the image sensor 130 to move in the +X direction by an amount of (m+d)×p [μm] while keeping the image sensor 130 at a fixed position in the Y direction. As an example, when m = 2 and d = 0.5, the amount of movement is defined as 2.5×p. As an example, the camera control unit 90 controls the image sensor 130 to move in the +X direction by an amount of 2.5×p [μm]. Then, the camera control unit 90 controls the moved image sensor 130 to capture an image. Thus, image 2-1 is obtained. Furthermore, the camera control unit 90 controls the movement of the image sensor 130 and the capture of images using the same method as when obtaining images 1-2 to 1-4. Thus, images 2-2 to 2-4 are obtained.

[0286] Similarly, the camera control unit 90 controls the image sensor 130 to move in the -Y direction by a movement amount of 2.5 × p [μm] while keeping the image sensor 130 at a fixed position in the X direction. Then, the camera control unit 90 controls the moved image sensor 130 to take a picture. As a result, image 3-1 is obtained. Furthermore, the camera control unit 90 controls the movement of the image sensor 130 and the taking of pictures by the moved image sensor 130 using the same method as when obtaining images 1-2 to 1-4. As a result, images 3-2 to 3-4 are obtained.

[0287] Similarly, the camera control unit 90 controls the image sensor 130 to move in the -X direction by a movement amount of 2.5 × p [μm] while maintaining a fixed position of the image sensor 130 in the Y direction. Then, the camera control unit 90 controls the moved image sensor 130 to take a picture. As a result, image 4-1 is obtained. Furthermore, the camera control unit 90 controls the movement of the image sensor 130 and the taking of pictures by the moved image sensor 130 using the same method as when obtaining images 1-2 to 1-4. As a result, images 4-2 to 4-4 are obtained.

[0288] The storage processing unit 92 performs the process of storing multiple images 1-1 to 4-4 obtained by the image sensor 130 in the image memory 62.

[0289] As an example, such as Figure 13A As shown, the generation unit 94 generates multiple monochrome images R-1 to B-4 for each color of the color filter based on the multiple images 1-1 to 4-4. Monochrome image R-1 is a red image generated from the red pixels in images 1-1 to 1-4, monochrome image R-2 is a red image generated from the red pixels in images 2-1 to 2-4, monochrome image R-3 is a red image generated from the red pixels in images 3-1 to 3-4, and monochrome image R-4 is a red image generated from the red pixels in images 4-1 to 4-4.

[0290] Similarly, monochrome image G-1 is a green image generated from the green pixels in images 1-1 to 1-4, monochrome image G-2 is a green image generated from the green pixels in images 2-1 to 2-4, monochrome image G-3 is a green image generated from the green pixels in images 3-1 to 3-4, and monochrome image G-4 is a green image generated from the green pixels in images 4-1 to 4-4.

[0291] Similarly, monochrome image B-1 is a blue image generated from the blue pixels in images 1-1 to 1-4, monochrome image B-2 is a blue image generated from the blue pixels in images 2-1 to 2-4, monochrome image B-3 is a blue image generated from the blue pixels in images 3-1 to 3-4, and monochrome image B-4 is a blue image generated from the blue pixels in images 4-1 to 4-4.

[0292] The generation of monochrome images R-1 to B-4 based on multiple images 1-1 to 4-4 is achieved, for example, by the following method: For each color of the color filter, the arrangement order of multiple image pixels obtained before and after the movement of the image sensor 130 is determined according to their absolute positions before and after the movement of the image sensor 130. The absolute position is the position based on the center of image pixel A and the center of image pixel B. Then, the multiple image pixels contained in each of the multiple images obtained before and after the movement of the image sensor 130 are arranged according to the above arrangement order. For example, the pixel values ​​of overlapping image pixels before and after the movement of the image sensor 130 are taken as the average or center value of the overlapping image pixels. Thus, monochrome images R-1 to B-4 are obtained. Figure 13A The image shown is a portion of monochrome images R-1 to B-4, which is 4×4=16 pixels.

[0293] As an example, such as Figure 13B As shown, the generation unit 94 generates multiple monochrome super-resolution images 200R, 200G, and 200B for each color of the color filter by performing super-resolution transformation on multiple monochrome images R-1 to B-4 obtained by the image sensor 40. Monochrome super-resolution image 200R is a red super-resolution image generated from monochrome images R-1 to B-4, monochrome super-resolution image 200G is a green super-resolution image generated from monochrome images R-1 to B-4, and monochrome super-resolution image 200B is a blue super-resolution image generated from monochrome images R-1 to B-4.

[0294] The super-resolution of monochrome super-resolution images 200R, 200G, and 200B based on multiple monochrome images R-1 to B-4 is achieved, for example, by the following method: For each color of the color filter, the arrangement order of the multiple image pixels obtained before and after the movement of the image sensor 130 is determined according to the absolute positions before and after the movement of the image sensor 130. The absolute positions are positions based on the center of the image pixel and the center of the image pixel. Then, the multiple image pixels contained in each of the multiple images obtained before and after the movement of the image sensor 130 are arranged according to the above arrangement order. Thus, monochrome super-resolution images 200R, 200G, and 200B are obtained. Figure 13BThe image shown is a portion of monochrome super-resolution images 200R, 200G, and 200B, i.e., 8×8=64 image pixels.

[0295] As an example, in Figure 13C The image shown is a blue monochrome super-resolution image 200B. Figure 13C Figure 13 shows a portion of the blue monochrome super-resolution image 200B, specifically images with 13 × 12 = 156 pixels obtained before and after the movement of the image sensor 130. Furthermore, Figure 13 shows the X-direction and Y-direction addresses of the multiple image pixels. The X-direction address x and the Y-direction address y represent any natural number greater than 1.

[0296] As illustrated above, the camera control unit 90 repeatedly performs the following control: moves the image sensor 130 along the +X direction by a movement amount of 1.0 × p [μm], then moves the image sensor 130 along the -Y direction by a movement amount of 1.0 × p [μm], then moves the image sensor 130 along the -X direction by a movement amount of 1.0 × p [μm], and then moves the image sensor 130 along either the X or Y direction by a movement amount of 2.5 × p [μm]. Thus, as shown in the blue monochrome super-resolution image 200B, when the image pixel corresponding to any physical pixel before the movement is designated as the first image pixel, and other image pixels assigned a color filter of the same color as the first image pixel are designated as the second image pixels, and the image pixel corresponding to the aforementioned any physical pixel after the movement is designated as the first image pixel, the first image pixel and the second image pixel are adjacent.

[0297] exist Figure 13C In the example shown, for instance, when the image pixel at address (x, y) is set as the first image pixel, the image pixel at address (x+1, y) is set as the second image pixel, and the image pixel at address (x, y) is set as the first image pixel, the first image pixel, i.e. the image pixel at address (x, y), is adjacent to the second image pixel, i.e. the image pixel at address (x+1, y).

[0298] Each of the plurality of image pixels corresponds to the position of each of the plurality of physical pixels before and after the movement of the image sensor 130. Therefore, the monochrome super-resolution image 200B represents the physical pixel configuration before and after the movement of the image sensor 130. For example, when the physical pixel corresponding to the first image pixel is designated as the first physical pixel and the physical pixel corresponding to the second image pixel is designated as the second physical pixel, the camera control unit 90 performs the following control: in the physical pixel configuration before and after the movement of the image sensor 130 represented by the super-resolution image 120, the image sensor 130 is moved to a position adjacent to the second physical pixel, which is different from the first physical pixel before the movement. In other words, the camera control unit 90 moves the image sensor 130 so that the physical pixel configuration before and after the movement of the image sensor 130 represented by the monochrome super-resolution image 200B is such that the adjacent image pixels included in the monochrome super-resolution image 200B are physical pixel configurations corresponding to image pixels that are different from each other.

[0299] As an example of such control, in the above example, the camera control unit 90 repeatedly performs the following control: moves the image sensor 130 along the +X direction by a movement amount of 1.0 × p [μm], then moves the image sensor 130 along the -Y direction by a movement amount of 1.0 × p [μm], then moves the image sensor 130 along the -X direction by a movement amount of 1.0 × p [μm], and then moves the image sensor 130 along either the X or Y direction by a movement amount of 2.5 × p [μm]. As described above, the physical pixel configuration before and after the movement of the image sensor 40 is represented, for example, by the super-resolution image 120, and the above physical pixel configuration can be achieved regardless of whether there is a space equivalent to one physical pixel between the physical pixels of the image sensor 130.

[0300] In addition, Figure 13CIn the example shown, for instance, when the physical pixel at address (x, y) is set as the first physical pixel of the image sensor 130 before movement, the physical pixel at address (x, y) corresponds to an example of the "first physical pixel" according to the technology of this invention, and the physical pixel at address (x+1, y), which is different from the physical pixel at address (x, y) before movement, corresponds to an example of the "second physical pixel" according to the technology of this invention. Furthermore, for instance, the image pixel at address (x, y) before movement is an example of the "image pixel corresponding to the first physical pixel before movement" according to the technology of this invention, the image pixel at address (x+1, y) is an example of the "image pixel corresponding to the second physical pixel different from the first physical pixel before movement" according to the technology of this invention, and the image pixel at address (1, 1) after movement is an example of the "image pixel corresponding to the first physical pixel after movement" according to the technology of this invention.

[0301] As an example, such as Figure 13C As shown, the detection unit 96 calculates the difference in pixel values ​​among the multiple image pixels contained in the monochrome super-resolution image 200B, comparing them to image pixels of the same color corresponding to the second physical pixel before and the first physical pixel after the movement. As an example, the detection unit 96 sequentially calculates the difference in pixel values ​​among all image pixels that are the target. Figure 13C In the example of an image pixel that becomes an object, a case is shown where the image pixel at address (x, y) and the image pixel at address (x+1, y) are highlighted (displayed by circles) and the difference in pixel value between the image pixel at address (x, y) and the image pixel at address (x+1, y) is calculated.

[0302] Furthermore, the detection unit 96 detects defective physical pixels from multiple physical pixels based on the calculated difference degree. Various methods can be applied to detect defective physical pixels. For example, as described in the second embodiment above, the detection unit 96 uses a pattern matching method that compares a pattern of pixel values ​​of the image pixel for which the difference degree is calculated and those of the image pixels surrounding it with a predefined pattern of abnormal pixel values ​​to detect defective physical pixels.

[0303] Furthermore, similarly to the monochrome super-resolution image 200B, the detection unit 96 also detects defective physical pixels in the red monochrome super-resolution image 200R and the green monochrome super-resolution image 200G.

[0304] Furthermore, as described above, during the generation of monochrome images R-1 to B-4, when any physical pixel before the movement is set as the third physical pixel before the movement, and the physical pixel before the movement, which is assigned a color filter with the same color as that third physical pixel before the movement, is set as the fourth physical pixel, the third physical pixel after the movement overlaps with the fourth physical pixel before the movement. In such cases... Figure 13A During the generation of monochrome images R-1 to B-4, the detection unit 96 can detect defective physical pixels by utilizing the overlap between the moved third physical pixel and the original fourth physical pixel. That is, the detection unit 96 can detect defective physical pixels from multiple physical pixels based on the difference in pixel values ​​between the same color pixels corresponding to the original fourth physical pixel and the moved third physical pixel in each of the multiple images. Furthermore, as an example, the detection unit 96 can sequentially calculate the difference in pixel values ​​between all the image pixels that are the target. Then, the detection unit 96 can detect defective physical pixels from multiple physical pixels based on the calculated difference. Furthermore, defective physical pixel detection can also be performed on only one or two of the multiple monochrome super-resolution images 200B, 200R, and 200G.

[0305] The correction unit 98 corrects the pixel value of the image pixel corresponding to the defective physical pixel detected by the detection unit 96 based on the pixel values ​​of the image pixels adjacent to that image pixel. Various methods can be applied to correct the pixel values. For example, as described in the second embodiment above, the correction unit 98 corrects the pixel value by setting it to the average or median value of the pixel values ​​of the image pixels adjacent to that image pixel.

[0306] Furthermore, in this example, when the physical pixel detected as address (x, y) is a defective physical pixel, the image pixel at address (x, y) corresponds to an example of the "first image pixel corresponding to the defective physical pixel" in the technology of this invention, and the image pixel adjacent to the image pixel at address (x, y) corresponds to an example of the "second image pixel adjacent to the first image pixel" in the technology of this invention.

[0307] Furthermore, similarly to the monochrome super-resolution image 200B, for the red monochrome super-resolution image 200R and the green monochrome super-resolution image 200G, the detection unit 96 also corrects the pixel value of the image pixel corresponding to the defective physical pixel based on the pixel value of the image pixel adjacent to that image pixel.

[0308] The output unit 100 performs the process of synthesizing and correcting multiple monochrome super-resolution images 200R, 200G, and 200B and outputting the synthesized image to the display 26. Thus, a synthesized image formed by synthesizing and correcting multiple monochrome super-resolution images 200R, 200G, and 200B while the pixel values ​​of the image pixels corresponding to the defective physical pixels are corrected is displayed on the display 26. For example, when the image pixel corresponding to the defective physical pixel is a white scratch or a black scratch, a synthesized image corrected in a way that makes the white or black scratch inconspicuous is displayed on the display 26.

[0309] Furthermore, the operation of the camera device 10 in the fourth embodiment is the same as that in the first embodiment.

[0310] As explained above, in the imaging device 10 according to the fourth embodiment, control is performed such that, in the physical pixel configuration before and after the movement of the image sensor 130 having multiple physical pixels, the image sensor 130 is moved to a position adjacent to a second physical pixel that is different from any first physical pixel before the movement. Furthermore, control is performed to enable the image sensor 130 to capture images before and after the movement. Then, defective physical pixels are detected from the multiple physical pixels based on the difference in pixel values ​​between the image pixels included in each of the multiple images captured by the image sensor 130 and the image pixels corresponding to the second physical pixel before the movement and the first physical pixel after the movement, respectively. Therefore, defective physical pixels can be detected from the multiple physical pixels based on the multiple images captured by the image sensor 130 before and after the movement.

[0311] Furthermore, in the imaging device 10 according to the fourth embodiment, control is performed such that, in the physical pixel configuration before and after the movement of the image sensor 130, the image sensor 130 is moved to a position adjacent to a second physical pixel where the moved first physical pixel is assigned a color filter of the same color as any of the moved first physical pixels. Control is also performed to capture images with the image sensor 130 before and after the movement. Then, defective physical pixels are detected from multiple physical pixels based on the difference in pixel values ​​between the same color image pixels corresponding to the moved second physical pixel and the moved first physical pixel in each of the multiple images captured by the image sensor 130. Therefore, for example, the detection accuracy of defective physical pixels can be improved compared to the case where defective physical pixels are detected from multiple physical pixels based on the difference in pixel values ​​between image pixels of different colors.

[0312] Furthermore, in the imaging device 10 according to the fourth embodiment, for example, control is performed such that, in the physical pixel configuration before and after the movement of the image sensor 130, the image sensor 130 is moved to a position where the third physical pixel after the movement overlaps with the fourth physical pixel, which is assigned a color filter of the same color as the third physical pixel before the movement. Furthermore, control is performed to enable the image sensor 130 to capture images before and after the movement. Then, defective physical pixels are detected from multiple physical pixels based on the difference in pixel values ​​between the same color image pixels corresponding to the fourth physical pixel before the movement and the third physical pixel after the movement, respectively, in each of the multiple images captured by the image sensor 130. Therefore, compared to the case where defective physical pixels are detected solely based on the difference in pixel values ​​between the same color image pixels corresponding to the second physical pixel before the movement and the first physical pixel after the movement, respectively, in each of the multiple images captured by the image sensor 130, the detection accuracy of defective physical pixels can be improved.

[0313] Furthermore, in the imaging device 10 according to the fourth embodiment, a process is performed to store multiple images captured by the image sensor 130 in the image memory 62. Then, defective physical pixels are detected from multiple physical pixels based on the difference in pixel values ​​between the multiple image pixels contained in each of the multiple images stored in the image memory 62. Therefore, defective physical pixels can be detected from multiple physical pixels based on the multiple images stored in the image memory 62.

[0314] Furthermore, in the imaging device 10 according to the fourth embodiment, the pixel value of the first image pixel corresponding to the defective physical pixel is corrected based on the pixel value of the second image pixel adjacent to the first image pixel. Therefore, the defect of the image pixel corresponding to the defective physical pixel can be corrected.

[0315] Furthermore, in the imaging device 10 according to the fourth embodiment, multiple images are super-resolution converted to generate multiple monochrome super-resolution images 200R, 200G, and 200B for each color of the color filter, and the multiple monochrome super-resolution images 200R, 200G, and 200B are synthesized. Therefore, compared with the case of outputting an image that has not been super-resolution converted, a high-resolution image can be displayed on the display 26.

[0316] Furthermore, the camera device 10 includes a controller 60 for detecting defective physical pixels, an image sensor 130, and a jitter correction mechanism 46 for moving the image sensor 130. Therefore, the camera device 10 can continuously perform image capture based on the image sensor 130, move the image sensor 130, and detect defective physical pixels.

[0317] Furthermore, in the fourth embodiment described above, the camera control unit 90 repeatedly performs the following control: moving the image sensor 130 along the +X direction by a movement amount of 1.0×p[μm], then moving the image sensor 130 along the -Y direction by a movement amount of 1.0×p[μm], then moving the image sensor 130 along the -X direction by a movement amount of 1.0×p[μm], then moving the image sensor 130 along the X or Y direction by a movement amount of 2.5×p[μm]. However, the order and direction of moving the image sensor 130 can be other than those described above.

[0318] Furthermore, in the fourth embodiment described above, the camera control unit 90 moves the image sensor 130 by a movement amount of 2.5 × p [μm]. However, when the spacing between the multiple physical pixels is set to 1, the camera control unit 90 can also move the image sensor 130 along the X or Y direction by a movement amount greater than 1, provided that the aforementioned monochrome super-resolution images 200R, 200G, and 200B are obtained. With this structure, in the physical pixel configuration before and after the movement of the image sensor 130, it is also possible to position the moved first physical pixel adjacent to a second physical pixel that is different from the first physical pixel before the movement.

[0319] Furthermore, in the fourth embodiment described above, when the natural number is set to n and the decimal number is set to d, the camera control unit 90 can move the image sensor 130 along the X or Y direction by a movement amount of (n+d)×p[μm] while obtaining the monochrome super-resolution images 200R, 200G, and 200B. With this structure, in the physical pixel configuration before and after the movement of the image sensor 130, it is also possible to position the moved first physical pixel adjacent to a second physical pixel that is different from the first physical pixel before the movement.

[0320] Furthermore, in the fourth embodiment described above, when m is set to a natural number greater than or equal to 2 and the decimal number is set to d, the camera control unit 90 can move the image sensor 130 along the X or Y direction by a movement amount of (m+d)×p[μm] while obtaining the super-resolution image 120. With this structure, in the physical pixel configuration before and after the movement of the image sensor 130, it is also possible to position the moved first physical pixel adjacent to a second physical pixel that is different from the first physical pixel before the movement.

[0321] Furthermore, if the camera control unit 90 moves the image sensor 130 by a movement amount of 2.5×p [μm], for example, compared to the case where the image sensor 130 is moved by a small number greater than 2.5×p or a small number less than 2.5×p but greater than 2.0×p, a high-resolution super-resolution image 120 can be obtained.

[0322] Furthermore, in the fourth embodiment described above, the output unit 100 can output position information corresponding to the position of the defective physical pixel. This position information is, for example, information corresponding to the address of the physical pixel. According to this structure, the position of the defective physical pixel can be determined based on the position information corresponding to the output position of the defective physical pixel.

[0323] Furthermore, in the fourth embodiment described above, the detection unit 96 can extract image pixels whose pixel values ​​deviate from a predetermined range from a plurality of image pixels, and detect defective physical pixels based on the difference between the pixel value of the image pixel outside the predetermined range and the pixel value of the image pixels adjacent to the image pixel outside the predetermined range. According to this structure, compared to calculating the difference between the pixel values ​​of the image pixels corresponding to all the image pixels before and after the movement of the second physical pixel, the number of times the difference is calculated can be reduced.

[0324] Furthermore, in the fourth embodiment described above, the detection unit 96 detects defective physical pixels from multiple physical pixels during the generation of monochrome super-resolution images 200R, 200G, and 200B. However, in a different process than generating monochrome super-resolution images 200R, 200G, and 200B, defective physical pixels can also be detected from multiple physical pixels using the same method as described above, based on multiple images captured by the image sensor 130 before and after movement.

[0325] Furthermore, in the fourth embodiment described above, a color image sensor with a Bayer alignment is used as an example of the color image sensor 130. However, color image sensors with other alignments, such as bar alignment, triangular alignment, Bayer alignment, and X-Trans (registered trademark) alignment, can also be used. When using such a color image sensor with other alignments, defective physical pixels can be detected from multiple physical pixels based on multiple images captured by the image sensor 130 before and after movement using the same method as described above.

[0326] In addition, Figure 19 The image shown is a super-resolution image 500 obtained through a comparative example relative to the fourth embodiment. This super-resolution image 500 is obtained by adjusting the image sensor 130 (reference image)... Figure 13A The image is obtained by moving the sensor 130 by a distance of 0.5 × p [μm] (instead of moving it by 2.5 × p [μm]). In this case, in the physical pixel configuration before and after the movement of the image sensor 130, the first physical pixel after the movement is located adjacent to the first physical pixel before the movement. Therefore, the pixel values ​​between adjacent image pixels in the super-resolution image 500 are the same, and thus it is impossible to detect defective physical pixels from multiple physical pixels.

[0327] Next, common variations in the above embodiments will be described. In the above embodiments, an example of image processing being performed by the controller 60 within the imaging device 10 has been given, but the technology of the present invention is not limited thereto. For example, such as Figure 14 As shown, image processing can also be performed by a computer 314 within an external device 312 that is communicatively connected to the camera device 10 via a network 310 such as a LAN or WAN. Figure 14 In the example shown, computer 314 includes CPU 316, memory 318, and RAM 320. A classification database 322 is constructed in memory 318, and a camera processing program 88 is stored therein.

[0328] The camera device 10 requests the external device 312 to perform camera processing via the network 310. Correspondingly, the CPU 316 of the external device 312 reads the camera processing program 88 from the memory 318 and executes the camera processing program 88 on the memory 320. The CPU 316 performs camera processing according to the camera processing program 88 executed on the memory 320. Then, the CPU 316 provides the processing result obtained from performing the camera processing to the camera device 10 via the network 310.

[0329] Furthermore, the camera processing can be performed separately by the camera device 10 and the external device 312, or the camera processing can be performed separately by the camera device 10 and multiple devices including the external device 312.

[0330] Furthermore, in the above embodiment, the imaging device 10 is exemplified as a physical camera (hereinafter also referred to as a "physical camera"). However, the technology of the present invention is not limited to this, and a virtual camera may be used instead of a physical camera. This virtual camera virtually photographs the subject from a virtual viewpoint based on image data obtained by multiple physical cameras set at different locations, thereby generating virtual viewpoint image data. In this case, the image represented by the virtual viewpoint image data, i.e., the virtual viewpoint image, is an example of the "image" involved in the technology of the present invention.

[0331] Furthermore, in the above embodiment, an example of storing a camera processing program 88 in the memory 60B has been described. However, the technology of the present invention is not limited thereto. For example, such as Figure 15 As shown, the camera processing program 88 can also be stored in the storage medium 330. The storage medium 330 is a non-transitory storage medium. As an example of the storage medium 330, any portable storage medium such as an SSD or USB memory can be cited.

[0332] The camera processing program 88, stored in the storage medium 330, is installed in the controller 60. The CPU 60A executes camera processing according to the camera processing program 88.

[0333] Furthermore, the camera processing program 88 can also be stored in the storage unit of other computers or server devices connected to the controller 60 via a communication network (not shown), and the camera processing program 88 can be downloaded according to the request of the camera device 10 and installed in the controller 60.

[0334] In addition, it is not necessary to store the entire camera processing program 88 in the storage unit or memory 60B of other computers or server devices connected to the controller 60; a portion of the camera processing program 88 may also be stored.

[0335] exist Figure 15 The example shown illustrates a configuration in which the controller 60 is built into the camera device 10, but the technology of the present invention is not limited thereto. For example, the controller 60 may also be located outside the camera device 10.

[0336] exist Figure 15 In the example shown, CPU60A is a single CPU, but it can also be multiple CPUs. Furthermore, a GPU can be used instead of CPU60A.

[0337] exist Figure 15 The example shown illustrates a controller 60, but the technology of the present invention is not limited thereto, and devices including ASICs, FPGAs, and / or PLDs can be used instead of controller 60. Furthermore, a combination of hardware and software structures can be used instead of controller 60.

[0338] As the hardware resource for performing the image processing described in the above embodiments, various processors can be used, as shown below. For example, a general-purpose processor, i.e., a CPU, can function as the hardware resource for performing image processing by executing software, i.e., a program. Furthermore, as a processor, a dedicated circuit, i.e., a processor with a circuit structure specifically designed for performing a particular process, such as an FPGA, PLD, or ASIC, can be used. Memory is either built into or connected to any memory, and any processor performs image processing using memory.

[0339] The hardware resources for performing camera processing can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs or a combination of a CPU and an FPGA). Furthermore, the hardware resources for performing camera processing can also be a single processor.

[0340] As examples of processors, firstly, there are processors that combine one or more CPUs with software, functioning as hardware resources for performing camera processing. Secondly, there are processors, such as SoCs (System-on-a-Chip), that use a single IC chip to implement the overall system functionality, including multiple hardware resources for performing camera processing. Thus, camera processing is implemented using one or more of these various processors as hardware resources.

[0341] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits composed of semiconductor components and other circuit elements. Moreover, the aforementioned image processing is merely one example. Therefore, without departing from the intended purpose, unnecessary steps can certainly be removed, new steps added, or the processing order rearranged.

[0342] The descriptions and illustrations above constitute a detailed explanation of a portion of the technology involved in this invention, and are merely one example of the technology of this invention. For example, the descriptions related to the above-described structure, function, effect, and effect are examples of the structure, function, effect, and effect of the portion involved in the technology of this invention. Therefore, without departing from the technical spirit of this invention, unnecessary parts may be deleted from the descriptions and illustrations above, or new elements may be added or replaced. Furthermore, to avoid complications and to facilitate understanding of the portion involved in the technology of this invention, descriptions related to common technical knowledge that does not require special explanation in aspects enabling the implementation of this invention have been omitted from the descriptions and illustrations above.

[0343] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" can mean only A, only B, or a combination of A and B. Furthermore, in this specification, the same approach applies to situations where three or more cases are connected by "and / or".

[0344] All documents, patent applications and technical standards described in this specification, and the specific and separately described documents, patent applications and technical standards incorporated herein by reference, are incorporated herein by reference to the same extent.

[0345] Regarding the above implementation methods, the following notes are also disclosed.

[0346] (Postscript 1)

[0347] A detection device comprising:

[0348] Processor; and

[0349] Memory, connected to or built into the processor,

[0350] The processor performs the following:

[0351] Acquire multiple images captured by the image sensor before and after the image sensor moves to a location where, in the physical pixel configuration of the image sensor having multiple physical pixels arranged along a first direction and a second direction intersecting the first direction, the first physical pixel after the movement is located adjacent to a second physical pixel that is different from the first physical pixel before the movement.

[0352] Calculate the difference in pixel values ​​between the pixels in each of the acquired plurality of images and the pixels corresponding to the second physical pixel before the movement and the first physical pixel after the movement;

[0353] Defective physical pixels are detected from the plurality of physical pixels based on the degree of difference.

[0354] (Postscript 2)

[0355] A detection device comprising:

[0356] processor;

[0357] Memory, connected to or built into the processor,

[0358] The processor performs the following:

[0359] The control is performed as follows: in the physical pixel configuration before and after the movement of a color image sensor with multiple physical pixels arranged in a periodic base and having different color filters, the image sensor is moved to the position where the third physical pixel after the movement overlaps with the fourth physical pixel having a color filter of the same color as the third physical pixel before the movement.

[0360] The image sensor captures images before and after movement;

[0361] Defective physical pixels are detected from the plurality of physical pixels based on the difference in pixel values ​​between the image pixels of the same color corresponding to the fourth physical pixel before movement and the third physical pixel after movement, respectively, in each of the plurality of image pixels contained in the plurality of images captured by the image sensor.

[0362] (Note 3)

[0363] A detection device comprising:

[0364] Memory; and

[0365] Memory, connected to or built into the processor,

[0366] The processor performs the following:

[0367] The control is performed as follows: in the physical pixel configuration before and after the movement of a color image sensor with multiple physical pixels arranged on a periodic basis and having different color filters, the image sensor is moved to a position adjacent to a second physical pixel with a color filter of the same color as the first physical pixel before the movement, where the first physical pixel after the movement is located.

[0368] The image sensor captures images before and after movement;

[0369] Defective physical pixels are detected from the plurality of physical pixels based on the difference in pixel values ​​between the image pixels of the same color corresponding to the second physical pixel before movement and the first physical pixel after movement, respectively, in each of the plurality of image pixels contained in the plurality of images captured by the image sensor.

[0370] (Note 4)

[0371] A detection device comprising:

[0372] processor;

[0373] Memory, connected to or built into the processor,

[0374] The processor performs the following:

[0375] The control is performed as follows: in the physical pixel configuration of a color image sensor with multiple physical pixels arranged on a periodic basis and having different color filters, the image sensor is moved to the position of the moved first physical pixel adjacent to the second physical pixel having a color filter of the same color as the moved first physical pixel, and to the position of the moved third physical pixel overlapping with the fourth physical pixel having a color filter of the same color as the moved third physical pixel.

[0376] The image sensor captures images before and after movement;

[0377] Defective physical pixels are detected from the plurality of physical pixels based on the difference in pixel values ​​between the image pixels of the same color corresponding to the second physical pixel before movement and the first physical pixel after movement, respectively, in each of the plurality of image pixels contained in the plurality of images captured by the image sensor.

[0378] (Note 5)

[0379] According to the detection device described in Appendix 4, wherein...

[0380] The processor detects defective physical pixels from the plurality of physical pixels based on the difference in pixel values ​​between the image pixels in each of the plurality of images captured by the image sensor and the image pixels of the same color corresponding to the fourth physical pixel before the movement and the third physical pixel after the movement.

[0381] Symbol Explanation

[0382] 10-Camera device, 12-Camera device body, 14-Interchangeable lens, 18-Release button, 20-Turntable, 22-Touch panel display, 24-Indicator key, 26-Display, 28-Touch panel, 30-Camera lens, 30A-Objective lens, 30B-Focusing lens, 32-Sliding mechanism, 34-Motor, 36-Motor, 40-Image sensor, 40A-Light receiving surface, 42-Signal processing circuit, 44-Shake detection sensor, 46-Shake correction mechanism, 48-Physical pixel, 50 - Displacement driver, 52 Actuator, 54 Position detection sensor, 60 Controller, 60A CPU, 60B Memory, 60C RAM, 62 Image memory, 64 UI system device, 66 External I / F, 68 Photoelectric conversion element driver, 74 Mechanical shutter driver, 76 Mechanical shutter actuator, 78 Mechanical shutter, 80 Input / output interface, 82 Bus, 84 Receiver, 86 Hard key unit, 88 Camera processing program, 90 Camera control unit, 92 - Storage processing unit, 94 - Generation unit, 96 - Detection unit, 98 - Calibration unit, 100 - Output unit, 110A, 110B - Image, 120 - Super-resolution image, 130 - Image sensor, 138 - Physical pixel, 140A, 140B - Image, 150 - Super-resolution image, 160 - Image sensor, 168 - Physical pixel, 170 - Image, 170A, 170B - Image, 180 - Super-resolution image, 200R, 200G, 200B - Monochrome super-resolution Image, 310-Network, 312-External device, 314-Computer, 318-Memory, 320-RAM, 322-Classification database, 330-Storage medium, 420-Super-resolution image, 450-Super-resolution image, 480-Super-resolution image, 500-Super-resolution image, A, B-Image pixels, 1-1~1-4-Image, R-1~4-Red monochrome image, G-1~4-Green monochrome image, B-1~4-Blue monochrome image, OA-Optical axis.

Claims

1. A detection device comprising: Processor; and Memory, connected to or built into the processor, The processor performs the following: The control is performed such that, in the physical pixel configuration of an image sensor having a plurality of physical pixels arranged along a first direction and a second direction intersecting the first direction before and after the movement, the image sensor is moved to a position where the moved first physical pixel is adjacent to a second physical pixel that is different from the moved first physical pixel. The image sensor captures images before and after movement; Defective physical pixels are detected from the plurality of physical pixels based on the difference in pixel values ​​between the image pixels contained in each of the plurality of images captured by the image sensor and the image pixels corresponding to the second physical pixel before the movement and the first physical pixel after the movement.

2. The detection device according to claim 1, wherein, The processor performs the following: The process of storing the plurality of images captured by the image sensor in the memory is performed; The defective physical pixel is detected based on the difference in pixel values ​​among the multiple image pixels contained in each of the plurality of images stored in the memory.

3. The detection device according to claim 1 or 2, wherein, When the spacing between the plurality of physical pixels in the first direction or the second direction is set to 1, the processor controls the image sensor to move along the first direction or the second direction by a movement amount greater than 1.

4. The detection device according to claim 1 or 2, wherein, When the spacing between the plurality of physical pixels in the first direction or the second direction is set to p, and the natural number is set to n and the pure decimal is set to d, The processor controls the image sensor to move along the first direction or the second direction by a movement amount of (n+d)×p.

5. The detection device according to claim 1 or 2, wherein, The processor corrects the pixel value of the first image pixel corresponding to the defective physical pixel based on the pixel value of the second image pixel adjacent to the first image pixel.

6. The detection device according to claim 1 or 2, wherein, The processor outputs position information corresponding to the position of the defective physical pixel.

7. The detection device according to claim 1 or 2, wherein, The processor detects the defective physical pixel based on the difference between the pixel value of an image pixel that deviates from a predetermined range and the pixel value of an image pixel adjacent to the image pixel that deviates from the predetermined range.

8. The detection device according to claim 1 or 2, wherein, The image sensor is a monochrome image sensor.

9. The detection device according to claim 8, wherein, When the spacing between the plurality of physical pixels in the first direction or the second direction is set to p, the processor controls the image sensor to move along the first direction or the second direction by a movement amount of 1.5×p.

10. The detection device according to claim 1 or 2, wherein, The image sensor is a color-type image sensor in which color filters of different colors are assigned to the plurality of physical pixels in a periodic substrate arrangement. When the spacing between the plurality of physical pixels in the first direction or the second direction is set to p, the natural number is set to n, and the number of physical pixels in each cycle of the base arrangement is set to T, The processor controls the image sensor to move along the first direction or the second direction by a movement amount of {(T-1)+0.5}×n×p.

11. The detection device according to claim 10, wherein, The substrate arrangement is a Bayer arrangement. The specified movement amount is 1.5 × p.

12. The detection device according to claim 10, wherein, The base arrangement is an X-Trans type arrangement, where X-Trans is a registered trademark. The specified movement amount is 5.5 × p.

13. The detection device according to claim 1, wherein, The image sensor is a color-type image sensor in which color filters of different colors are assigned to the plurality of physical pixels in a periodic substrate arrangement. The processor performs the following: The control is performed such that, in the physical pixel configuration before and after the movement of the image sensor, the image sensor is moved to the position where the third physical pixel after the movement overlaps with the fourth physical pixel which is assigned a color filter with the same color as the third physical pixel before the movement. The image sensor captures images before and after movement; Defective physical pixels are detected from the plurality of physical pixels based on the difference in pixel values ​​between the image pixels of the same color contained in each of the plurality of images captured by the image sensor and the image pixels of the same color corresponding to the fourth physical pixel before the movement and the third physical pixel after the movement.

14. The detection device according to claim 1, wherein, The image sensor is a color-type image sensor in which color filters of different colors are assigned to the plurality of physical pixels in a periodic substrate arrangement. The processor performs the following: The control is performed such that, in the physical pixel configuration before and after the movement of the image sensor, the image sensor is moved to a position adjacent to the second physical pixel, which is assigned a color filter with the same color as the first physical pixel before the movement, in the position of the first physical pixel after the movement. The image sensor captures images before and after movement; Defective physical pixels are detected from the plurality of physical pixels based on the difference in pixel values ​​between the image pixels of the same color contained in each of the plurality of images captured by the image sensor and the image pixels of the same color corresponding to the second physical pixel before movement and the first physical pixel after movement.

15. The detection device according to claim 14, wherein, When the spacing between the plurality of physical pixels in the first direction or the second direction is set to p, and m is set to a natural number greater than 2 and a pure decimal is set to d, The processor controls the image sensor to move along the first direction or the second direction by a movement amount of (m+d)×p.

16. The detection device according to claim 15, wherein, The specified movement amount is 2.5 × p.

17. The detection apparatus according to any one of claims 13 to 16, wherein, The processor performs the following: Super-resolution is performed on the multiple images to generate multiple monochrome super-resolution images for each color of the color filter; The multiple monochrome super-resolution images are synthesized.

18. A camera device comprising: The detection device according to any one of claims 1 to 17; The image sensor; and A moving mechanism that moves the image sensor along at least one of the first direction and the second direction.

19. A detection method comprising the following steps: The control is performed such that, in the physical pixel configuration of an image sensor having a plurality of physical pixels arranged along a first direction and a second direction intersecting the first direction before and after the movement, the image sensor is moved to a position where the moved first physical pixel is adjacent to a second physical pixel that is different from the moved first physical pixel. The image sensor captures images before and after movement; and Defective physical pixels are detected from the plurality of physical pixels based on the difference in pixel values ​​between the image pixels contained in each of the plurality of images captured by the image sensor and the image pixels corresponding to the second physical pixel before the movement and the first physical pixel after the movement.

20. A storage medium storing a program for causing a computer to perform a process comprising the following steps: The control is performed such that, in the physical pixel configuration of an image sensor having a plurality of physical pixels arranged along a first direction and a second direction intersecting the first direction before and after the movement, the image sensor is moved to a position where the moved first physical pixel is adjacent to a second physical pixel that is different from the moved first physical pixel. The image sensor captures images before and after movement; and Defective physical pixels are detected from the plurality of physical pixels based on the difference in pixel values ​​between the image pixels contained in each of the plurality of images captured by the image sensor and the image pixels corresponding to the second physical pixel before the movement and the first physical pixel after the movement.

Images