Imaging method and imaging device

Abstract

Provided are an image capture method and an image capture device that allow a user to recognize processing status. An image capture method for use with an image capture device comprising an image capture element for capturing a subject image, and a moving mechanism enabling a change in the relative position between the subject image and the image capture element. The image capture method comprises: a changing step for changing the relative position a plurality of times; an image capture step for acquiring a plurality of first images by capturing the subject image using the image capture element in a plurality of relative positions; a synthesizing step for generating a second image by synthesizing the plurality of first images; and a display step for performing temporal display with regard to the image capture step or the synthesizing step.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to an imaging method and an imaging device. [Background technology]

[0002] Patent Document 1 discloses a digital camera that generates high-quality images through super-resolution processing. The digital camera described in Patent Document 1 acquires four captured images while moving the relative position of the image sensor with respect to the subject image, and performs super-resolution processing. Each time an image is captured, it detects whether there is a change in the subject image between the first captured image and the second or subsequent captured images. If a change in the subject image occurs, the super-resolution capture is restarted from the beginning.

[0003] Patent Document 2 discloses an image processing device capable of obtaining high-quality composite image data. An acquisition unit acquires multiple image data having RGB color elements. A division unit divides each image data into multiple regions. A calculation unit calculates the amount of deviation for each region of each image data. A color interpolation unit performs color interpolation for at least one of the RGB color elements based on the amount of deviation for each region of each image data.

[0004] Patent Document 3 discloses an image acquisition device that includes a pixel shifting means for shifting the relative position between the light beam incident on the imaging element and the imaging element, a photographing means for photographing when the relative position is at a specific position and when the relative position is at least one position moved from the specific position by a series of operations by the pixel shifting means, an image generation means for generating a new high-resolution image from multiple image data photographed by the photographing means during the series of operations, and a display control means for displaying an initial image photographed among multiple images photographed before the new image is generated. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2016-171511 [Patent Document 2] Japanese Patent Application Publication No. 2019-161564 [Patent Document 3] Japanese Patent Application Laid-Open No. 2003-283887 Summary of the Invention [Problem to be solved by the invention]

[0006] One embodiment of the technique of the present disclosure provides an imaging method and an imaging device that enable a user to recognize a processing status. [Means for solving the problem]

[0007] In order to achieve the above object, the imaging method of the present disclosure is an imaging method used in an imaging device that includes an imaging element that captures an image of a subject and a moving mechanism that can change the relative position between the subject image and the imaging element, and includes a changing process that changes the relative position multiple times, an imaging process that obtains multiple first images by capturing the subject image using the imaging element at multiple relative positions, a synthesis process that generates a second image by synthesizing the multiple first images, and a display process that performs a temporal display related to the imaging process or the synthesis process.

[0008] The time display is preferably a display that allows the execution time of the imaging process or the synthesis process to be recognized.

[0009] It is preferable that the method includes a determination step of determining whether or not the compositing step can be performed based on whether at least one of the multiple first images satisfies the first condition, and that the compositing step is performed if the determination is positive, and that the processing content of the imaging step is changed if the determination is negative.

[0010] It is preferable to include a first stopping step of stopping capturing of the subject image in the imaging step when the determination in the determining step is negative.

[0011] It is preferable to include a second stopping step of stopping the temporal display in the display step when the capturing of the subject image is stopped.

[0012] It is preferable to include a notification step of notifying the user of the reason why the capturing of the subject image or the temporal display has been stopped.

[0013] In the changing step, it is preferable to change the relative position by utilizing a vibration applied to the imaging device.

[0014] When the determination in the determining step is negative, it is preferable to capture the subject image more times in the imaging step than when the determination is positive.

[0015] It is preferable to include an updating step of updating the temporal display in the displaying step when the number of times the subject image is captured in the capturing step is increased.

[0016] It is preferable to include a notification step of notifying the user of the reason why the number of times of imaging has been increased in the imaging step or the reason why the temporal display has been updated.

[0017] In the changing step, it is preferable to change the relative position by moving the imaging element to a predetermined position using a moving mechanism.

[0018] The imaging device of the present disclosure includes an imaging element that captures an image of a subject, a moving mechanism that can change the relative position between the subject image and the imaging element, and a processor, and the processor executes a change process that changes the relative position multiple times, an imaging process that obtains multiple first images by capturing the subject image at multiple relative positions using the imaging element, a synthesis process that generates a second image by synthesizing the multiple first images, and a display process that performs a temporal display related to the imaging process or the synthesis process.

[0019] The processor is capable of selectively executing a first mode and a second mode, and in the first mode, in the change process, the relative position is changed by utilizing shaking applied to the imaging device, and in the second mode, in the change process, the relative position is changed by moving the imaging element to a predetermined position using a movement mechanism, and a determination process is executed to determine whether or not a synthesis process can be executed based on whether at least one of the multiple first images satisfies a first condition, and if the determination is negative in the determination process, it is preferable that the temporal display in the display process be changed to content different from each other in the first mode and the second mode. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 2 is a schematic perspective view showing an example of the front side of an imaging device. [Figure 2] FIG. 1 is a diagram illustrating an example of the internal configuration of an imaging device. [Figure 3] 10A to 10C are diagrams illustrating an example of an imaging process and a composition process in a multi-shot composition mode. [Figure 4] FIG. 10 is a diagram showing an example of a time display performed in a multi-shot synthesis mode. [Figure 5] FIG. 10 is a diagram showing an example of a notification performed in a multi-shot synthesis mode. [Figure 6] 10 is a flowchart showing an example of a series of operations in a multi-shot synthesis mode. [Figure 7] 10A and 10B are diagrams illustrating an example of an imaging process in a pixel shift multi-shot composition mode. [Figure 8] 10 is a flowchart showing an example of a series of operations in a pixel shift multi-shot composition mode. [Figure 9] 10A and 10B are diagrams illustrating an example of updating the temporal display when the number of times a subject image is captured is increased. [Figure 10] FIG. 10 is a diagram illustrating an example in which the imaging device is connected to an external device. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of the technology of the present disclosure will be described with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following explanation, "IC" is an abbreviation for "Integrated Circuit." "CPU" is an abbreviation for "Central Processing Unit." "ROM" is an abbreviation for "Read Only Memory." "RAM" is an abbreviation for "Random Access Memory." "CMOS" is an abbreviation for "Complementary Metal Oxide Semiconductor."

[0024] "FPGA" is an abbreviation for "Field Programmable Gate Array." "PLD" is an abbreviation for "Programmable Logic Device." "ASIC" is an abbreviation for "Application Specific Integrated Circuit." "OVF" is an abbreviation for "Optical View Finder." "EVF" is an abbreviation for "Electronic View Finder." "JPEG" is an abbreviation for "Joint Photographic Experts Group." DSP is an abbreviation for "Digital Signal Processor."

[0025] The technology of the present disclosure will be described using an interchangeable lens digital camera as an example of an embodiment of an imaging device. Note that the technology of the present disclosure is not limited to interchangeable lens digital cameras, and can also be applied to digital cameras with an integrated lens.

[0026] FIG. 1 shows an example of the front side of an imaging device 10. As shown in FIG. 1, the imaging device 10 is a digital camera with interchangeable lenses. The imaging device 10 is composed of a main body 11 and an imaging lens 12 that is interchangeably attached to the main body 11. The imaging lens 12 is attached to the front side of the main body 11 via a camera-side mount 11A and a lens-side mount 12A (see FIG. 2). The imaging lens 12 is an example of a lens according to the technology of the present disclosure.

[0027] A dial 13 and a release button 14 are provided on the top surface of the main body 11. The dial 13 is operated when setting the operation mode, etc. The operation modes of the imaging device 10 include, for example, a still image capturing mode, a video capturing mode, and an image display mode. The release button 14 is operated by the user when starting to capture a still image or a video.

[0028] The still image capture mode also includes a "multi-shot synthesis mode" for obtaining a super-resolution image. The multi-shot synthesis mode of this embodiment is a mode in which multiple images are acquired and synthesized by changing the relative position between the subject image and the image sensor 20 (see FIG. 2) by utilizing the shaking of the image capture device 10 caused by the user's hand movement or the like. By synthesizing multiple images in which the relative positions between the subject image and the image sensor 20 are different, a super-resolution image with a resolution exceeding that of a single image can be obtained.

[0029] The main body 11 is also provided with a viewfinder 17. Here, the viewfinder 17 is a hybrid viewfinder (registered trademark). A hybrid viewfinder is a viewfinder that selectively uses, for example, an optical viewfinder (hereinafter referred to as "OVF") and an electronic viewfinder (hereinafter referred to as "EVF").

[0030] A viewfinder eyepiece 18 is provided on the rear side of the main body 11. The viewfinder eyepiece 18 selectively displays an optical image visible through the OVF, or a live view image, which is an electronic image visible through the EVF. The user can observe the optical image or live view image of the subject through the viewfinder eyepiece 18.

[0031] A display 15 (see FIG. 2) is provided on the rear side of the main body 11. The display 15 displays an image based on an image signal obtained by imaging, various menu screens, and the like.

[0032] Z axis A shown in Figure 1 Z corresponds to the optical axis of the imaging lens 12. X-axis A X and Y-axis A Y are perpendicular to each other and are parallel to the Z axis A Z It is perpendicular to the X axis A. X and Y-axis A Y corresponds to the pitch axis and yaw axis according to the technology of the present disclosure. In the following description, the Z axis A Z The rotation direction around the X axis is called the roll direction. X The rotation direction around the Y axis is called the pitch direction. Y The rotation direction around the X axis is called the yaw direction. X The direction is called the X direction, and the Y axis A Y The direction is called the Y direction.

[0033] 2 shows an example of the internal configuration of the imaging device 10. The main body 11 and imaging lens 12 are electrically connected by electrical contacts 11B provided on the camera-side mount 11A coming into contact with electrical contacts 12B provided on the lens-side mount 12A.

[0034] The imaging lens 12 includes an objective lens 30, a focus lens 31, a rear lens 32, and an aperture 33. The components are arranged along the optical axis (i.e., the Z axis AZ) of the imaging lens 12 in the order of objective lens 30, aperture 33, focus lens 31, and rear lens 32 from the objective side. teeth, constitute the imaging optical system. The type, number and arrangement order of the lenses that constitute the imaging optical system are not limited to the example shown in FIG.

[0035] The imaging lens 12 also has a lens drive control unit 34 and a memory. The lens drive control unit 34 is configured with, for example, a CPU, RAM, and ROM. The lens drive control unit 34 is electrically connected to a processor 40 in the main body 11 via electrical contacts 12B and 11B.

[0036] The lens drive control unit 34 drives the focus lens 31 and the diaphragm 33 based on a control signal transmitted from the processor 40. The lens drive control unit 34 controls the drive of the focus lens 31 based on a control signal for focus control transmitted from the processor 40 in order to adjust the focus position of the imaging lens 12.

[0037] The diaphragm 33 has an aperture whose diameter is variable around the optical axis. The lens drive control unit 34 controls the drive of the diaphragm 33 based on an aperture adjustment control signal sent from the processor 40 in order to adjust the amount of light incident on the light receiving surface 20A of the image sensor 20.

[0038] The main body 11 is provided with an imaging sensor 20, a processor 40, an image processing unit 41, an operation unit 42, a mechanical vibration isolation mechanism 43, a shake detection sensor 44, a memory 45, and a display 15. The operations of the imaging sensor 20, the image processing unit 41, the operation unit 42, the mechanical vibration isolation mechanism 43, the shake detection sensor 44, the memory 45, and the display 15 are controlled by the processor 40. The processor 40 is composed of, for example, a CPU, RAM, ROM, etc. In this case, the processor 40 executes various processes based on an operating program 45A stored in the memory 45. The processor 40 may be composed of a collection of multiple IC chips.

[0039] The image sensor 20 is, for example, a CMOS image sensor. The image sensor 20 is aligned along a Z axis A Z is perpendicular to the light receiving surface 20A and is aligned with the Z axis AZ is arranged so as to be located at the center of the light receiving surface 20A. Light (subject image) that has passed through the imaging lens 12 is incident on the light receiving surface 20A. A plurality of pixels that generate image signals by performing photoelectric conversion are formed on the light receiving surface 20A. The imaging sensor 20 photoelectrically converts the light that has entered each pixel to generate and output an image signal. The imaging sensor 20 is an example of an "imaging element" according to the technology of the present disclosure.

[0040] A color filter array in a Bayer pattern is arranged on the light receiving surface of the image sensor 20, with a color filter of either R (red), G (green), or B (blue) arranged opposite each pixel. Therefore, each pixel of an image before color interpolation contains color information of either R, G, or B. The arrangement of the color filter array is not limited to the Bayer pattern and can be changed as appropriate.

[0041] The image sensor 20 is held by a mechanical vibration isolation mechanism 43. The mechanical vibration isolation mechanism 43 supports the image sensor 20 along the X-axis A. X and Y-axis A Y The mechanical vibration isolation mechanism 43 is held so as to be able to translate in the up direction and to rotate in the roll direction. The configuration of the mechanical vibration isolation mechanism 43 is publicly known, for example, from JP 2016-171511 A. The mechanical vibration isolation mechanism 43 is an example of a "moving mechanism that can change the relative position between the subject image and the imaging element" according to the technology of the present disclosure. The mechanical vibration isolation mechanism 43 may also be a mechanism that changes the relative position between the subject image and the imaging sensor 20 by driving a part of the lens that constitutes the imaging optical system of the imaging lens 12.

[0042] The shake detection sensor 44 detects shake applied to the main body 11 that houses the imaging sensor 20. The shake detection sensor 44 is, for example, a five-axis shake detection sensor that detects shake in the roll direction, yaw direction, pitch direction, X direction, and Y direction. Hereinafter, shake in the roll direction will be referred to as rotational shake, shake in the yaw direction and pitch direction will be referred to as angular shake, and shake in the X direction and Y direction will be referred to as translational shake.

[0043] The shake detection sensor 44 is composed of, for example, a gyro sensor and an acceleration sensor. The gyro sensor detects rotational shake and angular shake, and the acceleration sensor detects translational shake.

[0044] Processor 40 controls the driving of mechanical vibration isolation mechanism 43 based on the shake of imaging device 10 (vibration applied to imaging device 10) detected by shake detection sensor 44. Specifically, processor 40 changes the relative position between the subject image and imaging sensor 20 so as to offset the displacement of the subject image caused by the user's hand shake.

[0045] The mechanical vibration isolation mechanism 43 may be provided with a position detection sensor for detecting the position of the image sensor 20. This position detection sensor may be, for example, a Hall sensor. In this case, the processor 40 controls the drive of the mechanical vibration isolation mechanism 43 based on the shake information detected by the shake detection sensor 44 and the position information of the image sensor 20 detected by the position detection sensor. Note that the processor 40 may stop the drive control of the mechanical vibration isolation mechanism 43 in the above-mentioned multi-shot composition mode.

[0046] In the multi-shot composition mode, the processor 40 causes the image sensor 20 to perform a predetermined number of imaging operations (for example, four times) in response to the user's operation of the release button 14. That is, in the multi-shot composition mode, the image sensor 20 performs an imaging process to acquire multiple images in which the relative positions of the subject image and the image sensor 20 are different. Note that the images acquired by the image sensor 20 are an example of a "first image" according to the technology of the present disclosure.

[0047] The image processing unit 41 is configured by, for example, a DSP. The image processing unit 41 performs various image processing such as color interpolation processing on the image signal to generate image data in a predetermined file format (for example, JPEG format).

[0048] In addition, in the multi-shot synthesis mode, the image processing unit 41 performs synthesis processing to generate a super-resolution image by synthesizing multiple images acquired by the imaging processing. Note that the super-resolution image is an example of the "second image" according to the technology of the present disclosure.

[0049] Specifically, the image processing unit 41 derives the amount of misalignment for each region of each image based on the blur information detected by the blur detection sensor 44 and the position information of the image sensor 20 detected by the position detection sensor, and combines the multiple images based on the derived amount of misalignment. Note that the image processing unit 41 can also calculate the amount of misalignment for each region of each image using block matching technology instead of the blur information and position information. The combining process used in the multi-shot combining mode is known, for example, from JP 2019-161564 A.

[0050] The display 15 displays images based on the image data generated by the image processing unit 41. The images include still images, videos, and live view images. The live view images are images that are displayed in real time on the display 15 by sequentially outputting the image data generated by the image processing unit 41 to the display 15.

[0051] The image data generated by the image processing unit 41 can be stored in an internal memory (not shown) built into the main body 11 or in a storage medium (for example, a memory card) that is detachable from the main body 11.

[0052] The operation unit 42 includes the aforementioned dial 13, release button 14, and command keys (not shown). The command keys are provided, for example, on the rear side of the main body 11. The processor 40 controls each part in the main body 11 and the lens drive control unit 34 in the imaging lens 12 in response to operations on the operation unit 42.

[0053] In the multi-shot composition mode, the processor 40 performs a determination process to determine whether composition processing by the image processing unit 41 is possible based on whether at least one of the multiple images acquired by the imaging process satisfies a predetermined quality condition. The quality condition is a condition determined based on the brightness, blur, or degree of blur of the image. The quality condition may be a condition based on the brightness, blur, or degree of blur of a single image, or a condition based on the difference in brightness, blur, or degree of blur between multiple images, or the difference in subject position. The quality condition may also be a comprehensive condition using multiple indicators such as brightness, blur, and degree of blur. Furthermore, the quality condition may be a different condition for each of the multiple images acquired by the imaging process. The quality condition is an example of a "first condition" according to the technology of the present disclosure.

[0054] If the determination in the determination process is negative, the processor 40 changes the processing content of the imaging process. In this embodiment, a first cancellation process is performed to cancel the imaging of the subject image by the imaging sensor 20. Examples of the changed processing content of the imaging device include a mode in which the imaging process is canceled, a mode in which part of the imaging process is executed again, and a mode in which the imaging condition settings (shutter speed, aperture value of the imaging lens, sensitivity of the imaging element, sensitivity of the movement mechanism, etc.) are changed.

[0055] In the multi-shot synthesis mode, the processor 40 also performs a display process for displaying a time series of the image capturing process or the synthesis process. The processor 40 controls the display 15 to display a time series of the image capturing process or the synthesis process on the display 15. More preferably, the time series of the image capturing process or the synthesis process is displayed on the display 15. process or synthetic processThe time display is a display that allows the user to recognize the execution time of the above. The time display is, for example, a remaining time display that shows the time until the image capture process and the compositing process are completed. The time display may display the remaining time of the image capture process and the remaining time of the compositing process separately, or may display the remaining time until the super-resolution process that combines the image capture process and the compositing process is completed. Furthermore, the time display is not limited to a display mode that directly displays the remaining time, but may also be a display mode that displays the number of remaining images, etc.

[0056] In addition, in the multi-shot synthesis mode, when the judgment is negative in the judgment process, the processor 40 performs a first stop process that stops the imaging of the subject image by the imaging sensor 20, as well as a second stop process that stops the temporal display related to the imaging process or the synthesis process.

[0057] Furthermore, if the determination in the determination process is negative, the processor 40 performs a notification process to notify the user of the reason why the capturing of the subject image or the temporal display was stopped. For example, the processor 40 controls the display 15 to display the reason for the stop on the display 15, thereby notifying the user of the reason for the stop.

[0058] 3 illustrates an example of the imaging process and composition process in the multi-shot composition mode. As shown in Fig. 3, in the imaging process, N images P are acquired by the imaging sensor 20. During the imaging process, if the imaging device 10 is shaken by a user's hand or the like, a shift occurs in the position of the subject image SI captured in each image P.

[0059] Due to the shift in the position of the subject image SI in each image P, each region constituting the subject image SI contains multiple color information. In the example shown in Fig. 3, the region PA in the subject image SI contains all of the color information of R, G, and B. In this way, since each region constituting the subject image SI contains multiple color information, a high-quality super-resolution image PS can be obtained by combining N images P.

[0060] The super-resolution image PS is stored in the memory 45 and then displayed on the display 15 under the control of the processor 40 .

[0061] Fig. 4 shows an example of a time display performed in the multi-shot synthesis mode. In the example shown in Fig. 4, the ratio of the time that has been completed to the total time required for each of the imaging process and synthesis process is displayed on the display 15 (i.e., percentage display). In addition to the percentage display, the remaining time for each process may also be displayed.

[0062] FIG. 5 shows an example of a notification made in the multi-shot composition mode. In the example shown in FIG. 5, when the processor 40 stops image capture during the image capture process, it displays on the display 15 a message indicating that the image capture process has been stopped and the reason for the stoppage. The reason for the stoppage may be, for example, a change in brightness during the image capture process. For example, if the brightness of the subject image changes due to a change in lighting caused by fluorescent light flicker, a difference in brightness may occur between the multiple images to be combined, potentially making it impossible to perform the composition process. Other factors that may cause the image capture process or composition process to be stopped include unacceptable large shaking of the image capture device 10 or large movement of the subject.

[0063] Fig. 6 is a flowchart showing an example of a series of operations in the multi-shot synthesis mode. As shown in Fig. 6, processor 40 determines whether or not a capture instruction has been issued by the user operating release button 14 (step S10). If processor 40 determines that a capture instruction has been issued (step S10: YES), it starts the above-mentioned time-lapse display (step S11) and causes image sensor 20 to capture an image of the subject (step S12).

[0064] The processor 40 determines whether the image acquired by the image sensor 20 satisfies the above-mentioned quality condition (step S13). If the quality condition is satisfied (step S13: YES), the processor 40 determines whether the image sensor 20 has completed a predetermined number of imaging attempts (here, N times) (step S14). If the image sensor 20 has not completed N imaging attempts (step S14: NO), the processor 40 returns the process to step S12 and causes the image sensor 20 to capture the subject image again.

[0065] When N images have been captured (step S14: YES), the processor 40 causes the image processing unit 41 to perform the above-mentioned synthesis process (step S15), and when the synthesis process by the image processing unit 41 has ended, causes the processor 40 to end the temporal display (step S16). Then, the processor 40 stores the super-resolution image generated by the image processing unit 41 in the memory 45 and causes the display 15 to display it (step S17).

[0066] If the processor 40 determines in step S13 that the quality condition is not satisfied (step S13: NO), the processor 40 stops capturing the subject image by the image sensor 20 (step S18). The processor 40 also stops capturing the subject image and stops the temporal display (step S19). The processor 40 then notifies the user of the reason for the stop by displaying the reason for the stop on the display 15 (step S20).

[0067] Note that step S12 is an example of an "imaging process" according to the technology of the present disclosure. Step S15 is an example of a "combining process" according to the technology of the present disclosure. Step S11 is an example of a "displaying process" according to the technology of the present disclosure. Step S14 is an example of a "changing process" according to the technology of the present disclosure. Step S13 is an example of a "determining process" according to the technology of the present disclosure. Step S18 is an example of a "first stopping process" according to the technology of the present disclosure. Step S19 is an example of a "second stopping process" according to the technology of the present disclosure. Step S20 is an example of a "notifying process" according to the technology of the present disclosure.

[0068] As described above, the imaging device 10 of this embodiment displays the image over time while super-resolution processing is being performed in multi-shot composition mode. Because super-resolution processing involves both imaging and composition processing and therefore requires a long processing time, the user may have doubts about whether the processing is proceeding normally. By displaying the image over time, the imaging device 10 of this embodiment allows the user to recognize the processing status, thereby eliminating the user's doubts.

[0069] [Variations] Various modifications of the above embodiment will be described below.

[0070] In the above embodiment, the multi-shot composition mode is a mode in which a change process is performed to change the relative position between the subject image and the image sensor 20 by utilizing the shaking of the image capture device 10 caused by the user's hand shake or the like (hereinafter referred to as "hand shake-utilizing multi-shot composition mode"). Alternatively, the multi-shot composition mode may be a mode in which a change process is performed to change the relative position between the subject image and the image sensor 20 multiple times by actively moving the image sensor 20 using the mechanical vibration isolation mechanism 43 (hereinafter referred to as "pixel-shifted multi-shot composition mode"). That's what they say. ) may also be used.

[0071] In the pixel-shift multi-shot composition mode, the processor 40 causes the image sensor 20 to capture images multiple times while slightly moving the image sensor 20 in a direction perpendicular to the optical axis. The image processing unit 41 generates a super-resolution image by combining the multiple images acquired by the image sensor 20. The pixel-shift multi-shot composition mode is known from Japanese Patent Application Laid-Open No. 2016-171511, Japanese Patent Application Laid-Open No. 2019-161564, etc.

[0072] In the pixel shift multi-shot composition mode, it is undesirable for the imaging device 10 to be shaken by the user's hand or the like, so it is preferable that the imaging device 10 is used in a fixed state using a tripod or the like.

[0073] Fig. 7 shows an example of imaging processing in pixel-shift multi-shot composition mode. The example shown in Fig. 7 is an example in which a total of four images P are acquired while the imaging sensor 20 is shifted by one pixel in the X direction or Y direction. In pixel-shift multi-shot composition mode, each region constituting the subject image SI contains all of the color information of R, G, and B, so that a high-quality super-resolution image PS can be obtained by combining the four images P. Note that the number of images acquired is not limited to four and can be changed as appropriate, such as to nine.

[0074] Fig. 8 is a flowchart showing an example of a series of operations in the pixel-shift multi-shot composition mode. Steps S30 to S37 shown in Fig. 8 are the same as steps S10 to S17 shown in Fig. 6. However, in this modification, the processor 40 moves the image sensor 20 by one pixel, as shown in Fig. 7, each time it causes the image sensor 20 to capture an image in step S32. Also, in step S34, the processor 40 determines whether or not image capture has been completed for the four positions shown in Fig. 7.

[0075] In this modification, if the processor 40 determines in step S33 that the quality condition is not satisfied (step S33: NO), it causes the image sensor 20 to capture an image again at the same position (step S38). This re-imaging increases the number of times of imaging compared to the initially planned number of times (four times), and the time required for the imaging process increases. Therefore, the time display related to the imaging process is updated (step S3 9 Then, the processor 40 notifies the user of the reason for updating the temporal display by displaying the reason for updating the temporal display on the display 15 (step S40). After this, the processor 40 returns the process to step S33, and determines whether the image acquired by re-imaging satisfies the quality condition. Note that in step S3 9 is an example of an "update process" according to the technique of the present disclosure.

[0076] As described above, in the pixel-shift multi-shot composition mode according to this modified example, the relative position between the subject image and the image sensor 20 is changed by moving the image sensor 20 to a predetermined position using the mechanical vibration isolation mechanism 43. When the determination process results in a negative judgment, the processor 40 increases the number of times the subject image is captured in the image capture process compared to when the determination process results in a positive judgment. When the number of times the subject image is captured is increased, the processor 40 updates the temporal display in the display process. Then, a notification process is performed to notify the user of the reason for increasing the number of times the image is captured in the image capture process or the reason for updating the temporal display.

[0077] Fig. 9 shows an example of updating the temporal display when the number of times a subject image is captured is increased. When the number of times an image is captured is increased by re-imaging, the processor 40 increases the remaining time for the imaging process, and therefore, as shown in Fig. 9, changes the percentage display so that the user can recognize that the remaining time has increased. In addition, in the example shown in Fig. 9, the processor 40 displays on the display 15 that the reason for updating the temporal display is that brightness has changed.

[0078] 9, when it is determined in step S33 that the quality condition is not satisfied, the processor 40 causes the image sensor 20 to re-image, but re-imaging is not required. In this case, the processor 40 may cause the image processing unit 41 to perform synthesis processing using only images that satisfy the quality condition.

[0079] Furthermore, the user may be able to select between a camera shake utilizing multi-shot composition mode (hereinafter referred to as the first mode) and a pixel shift multi-shot composition mode (hereinafter referred to as the second mode) using the operation unit 42. That is, the processor 40 may be able to selectively execute the first mode and the second mode. In this case, when the determination process results in a negative judgment, the processor 40 changes the temporal display in the display process to different content in the first mode and the second mode. For example, the processor 40 stops the temporal display in the first mode and updates the temporal display in the second mode.

[0080] In the above embodiment and each of the above modifications, whether or not the image obtained by imaging satisfies the quality condition is determined each time the image sensor 20 captures an image of a subject. Alternatively, the image sensor 20 may capture images multiple times, and after a predetermined number of images have been acquired, whether or not each image satisfies the quality condition may be determined.

[0081] Furthermore, in the above embodiment and each of the above variants, the processor 40 displays the time and reason on the display 15, but instead of or in addition to the display 15, the processor 40 may display the time and reason on the viewfinder 17.

[0082] 10, the imaging device 10 may be capable of tethered imaging, in which imaging is performed while connected to a personal computer 50 via a wired or wireless connection. In this case, the above-described composition process may be performed within the personal computer 50. The above-described time display and reason display may also be performed on a display 52 provided in the personal computer 50. The imaging device 10 may be connectable to an external device other than the personal computer 50.

[0083] In the above embodiment, the hardware structure of the control unit, for example, the processor 40, can be any of the following various processors: In addition to a CPU, which is a general-purpose processor that functions by executing software (programs), the various processors can also be a processor such as an FPGA, whose circuit configuration can be changed after manufacturing. PLD Includes: The above processors This includes dedicated electrical circuits such as PLDs or ASICs, which are processors having circuitry specifically designed to perform specific processing.

[0084] The control unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple control units may be configured with a single processor.

[0085] There are several possible examples of configuring multiple control units with a single processor. A first example is a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple control units. A second example is a form in which a processor is used to realize the functions of an entire system including multiple control units on a single IC chip, as typified by system-on-chip (SOC). In this way, the control unit can be configured as a hardware structure using one or more of the various processors described above.

[0086] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0087] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0088] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0089] 10. Imaging device 11 Main unit 11A Camera side mount 11B Electrical contacts 12 Imaging lens 12A Lens side mount 12B Electrical Contact 13 Dial 14 Release button 15 Display 17 Finder 18 Finder eyepiece 20 Image sensor 20A light receiving surface 30 objective lenses 31 Focus Lens 32 Rear lens 34 Lens drive control unit 40 processors 41 Image processing section 42 Operation section 43 Mechanical vibration isolation mechanism 44 Shake detection sensor 45 memory 45A Operation Program 50 Personal Computers 52 Display A X X axis A Y Y-axis A Z Z-axis P Image PA area PS super-resolution image SI subject image

Claims

1. An imaging method used in an imaging device including an imaging element that captures an image of a subject, and a movement mechanism that can change a relative position between the subject image and the imaging element, a changing step of changing the relative position a plurality of times; an imaging step of capturing a plurality of first images by capturing the subject image using the imaging element at a plurality of the relative positions; a combining step of generating a second image by combining the plurality of first images; a display step of performing a time display relating to the imaging step or the combining step; Including, The first mode and the second mode are selectively executable; In the first mode, in the changing step, the relative position is changed by utilizing a shake applied to the imaging device; In the second mode, in the changing step, the relative position is changed by moving the imaging element to a predetermined position using the moving mechanism; performing a determination step of determining whether the combining step is executable based on whether at least one of the plurality of first images satisfies a predetermined quality condition; If the determination in the determining step is negative, the temporal display in the display step is changed to different contents in the first mode and the second mode. Imaging method.

2. The quality conditions are conditions determined based on brightness, blur, or degree of blur. The imaging method according to claim 1 .

3. The temporal display is a display that allows the execution time of the imaging process or the synthesis process to be recognized. The imaging method according to claim 1 or 2.

4. When the judgment in the judgment step is affirmative, the synthesis step is executed, and when the judgment is negative, the processing content of the imaging step is changed. The imaging method according to claim 1 .

5. When the determination is negative in the determining step, the number of times of capturing the subject image in the imaging step is increased compared to when the determination is positive. The imaging method according to claim 4 .

6. an updating step of updating the temporal display in the displaying step when the number of times the subject image is captured in the imaging step is increased; The imaging method according to claim 5 .

7. a notification step of notifying a user of the reason for increasing the number of times of imaging in the imaging step or the reason for updating the temporal display, The imaging method according to claim 6.

8. an imaging device that captures an image of a subject; a movement mechanism that can change the relative position of the subject image and the imaging device; and a processor; The processor: a change process for changing the relative position multiple times; an imaging process of acquiring a plurality of first images by capturing the subject image using the imaging element at a plurality of the relative positions; a synthesis process for generating a second image by synthesizing the plurality of first images; a display process for performing a time display relating to the imaging process or the composition process; An imaging device that performs The processor: The first mode and the second mode are selectively executable; In the first mode, in the change process, the relative position is changed by utilizing a shake applied to the imaging device; In the second mode, in the change process, the relative position is changed by moving the imaging element to a predetermined position using the movement mechanism; performing a determination process of determining whether the combining process is executable based on whether at least one of the plurality of first images satisfies a predetermined quality condition; When the determination is negative in the determination process, the temporal display in the display process is changed to different contents in the first mode and the second mode. Imaging device.

9. The quality conditions are conditions determined based on brightness, blur, or degree of blur. The imaging device according to claim 8 .

Images