Imaging device, imaging device control method, and computer program product

Abstract

An imaging device according to one embodiment is a digital imaging device configured to enable a zero shutter lag (ZSL) operation that generates an image at a time point of execution of a shooting instruction of a user by using a captured image obtained before the time point of detection of the shooting instruction of the user. The imaging device includes a shooting prediction unit and an imaging control unit. The shooting prediction unit predicts the shooting instruction of the user. In a case where the shooting prediction unit predicts a shooting state in which the shooting instruction of the user can be executed at a time point before detection of the shooting instruction, the imaging control unit changes control of an image sensor in the ZSL operation.

Description

[Technical Field]

[0001] Embodiments of this disclosure relate to imaging devices, imaging device control methods, and computer program products. [Background Technology]

[0002] In imaging devices such as smartphones, there are already known techniques for synthesizing multiple captured images to achieve high image quality, such as HDR (High Dynamic Range) synthesis and NR (Noise Reduction).

[0003] For example, in scenes where the subject is moving, the movement of the subject can produce artifacts called motion blur during the acquisition of multiple images for composition. Motion blur can be improved by making the synthesized images as close as possible in time. Therefore, techniques are known to acquire multiple images in short time intervals by driving the imaging device at a high frame rate in a preview state before shooting.

[0004] [Existing Technical Documents]

[0005] [Patent Document 1]

[0006] U.S. Patent Application Publication No. 2020 / 0221008

[0007] [Disclosure of the Invention]

[0008] [The problem this invention aims to solve]

[0009] However, there is a problem with increased power consumption due to driving the imaging device at a high frame rate in advance. From the perspective of heat dissipation and battery capacity, smaller imaging devices such as smartphones are more susceptible to the effects of increased power consumption.

[0010] The purpose of this disclosure is to obtain an image with less motion blur while suppressing power consumption in preview mode.

[0011] [Solutions to the problem]

[0012] An imaging apparatus according to one aspect of an embodiment is a digital imaging apparatus configured to perform zero shutter lag (ZSL) operation, which generates an image at the time the user's shooting command is executed by using a captured image obtained before the time the user's shooting command is detected. The imaging apparatus includes a shooting prediction unit and an imaging control unit. The shooting prediction unit predicts the user's shooting command. If the shooting prediction unit predicts a shooting state in which the user's shooting command can be executed at the time before the shooting command is detected, the imaging control unit changes the control of the image sensor in the ZSL operation.

[0013] According to another aspect of this embodiment, the control method is executed by a digital imaging device configured to perform zero shutter lag (ZSL) operation, which generates an image at the time the user's shooting command is executed by using a captured image obtained before the time the user's shooting command is detected. The method includes: predicting the user's shooting command; and, if a shooting state is predicted that the user's shooting command can be executed at the time before the shooting command is detected, changing the control of the image sensor in the ZSL operation.

[0014] According to another aspect of this embodiment, a computer program product stores a program executed by a computer of a digital imaging device configured to perform zero shutter lag (ZSL) operation, which generates an image at the time the user's shooting command is executed by using a captured image obtained before the time the user's shooting command is detected. The program causes the computer to: predict the user's shooting command; and, if it predicts a shooting state in which the user's shooting command can be executed at the time before the shooting command is detected, change the control of the image sensor in the ZSL operation.

[0015] [Effects of the Invention]

[0016] According to embodiments of this disclosure, images with less motion blur can be obtained while suppressing power consumption in preview mode.

[0017] [Brief description of the attached diagram]

[0018] Figure 1 This is a diagram illustrating an example configuration of an imaging device according to a first embodiment;

[0019] Figure 2 This is an example diagram illustrating the functional configuration of the controller according to the first embodiment;

[0020] Figure 3 This is a diagram illustrating the image sensor's image capture prediction and control changes in the control processing according to the first embodiment;

[0021] Figure 4 This is a flowchart example illustrating the control processing flow according to the first embodiment;

[0022] Figure 5 This is a diagram explaining the shooting prediction in the control processing according to the second embodiment;

[0023] Figure 6 This is a diagram illustrating an example configuration of an imaging device according to a third embodiment;

[0024] Figure 7 This is a diagram explaining the shooting prediction in the control processing according to the third embodiment;

[0025] Figure 8 This is a diagram illustrating the control change of the image sensor in the control processing according to the fourth embodiment;

[0026] Figure 9 This is a diagram explaining the threshold for motion detection in the control processing according to the fifth embodiment;

[0027] Figure 10 This is a diagram illustrating the control change of the image sensor in the control processing according to the sixth embodiment;

[0028] Figure 11 This is a diagram illustrating the control change of the image sensor in the control processing according to the seventh embodiment; and

[0029] Figure 12 This is a diagram illustrating the control changes of the image sensor in the control processing according to the eighth embodiment.

[0030] [Examples of Implementing the Invention]

[0031] In the following, an imaging apparatus, an imaging apparatus control method, a program, and a computer program product according to embodiments will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments.

[0032] In the description of this embodiment, components having the same or substantially the same function as those previously described in the previous figures have the same reference numerals, and their descriptions may be appropriately omitted. Furthermore, even if the same or substantially the same components are shown, the dimensions and scale of the components may vary depending on the drawing. Additionally, from the perspective of ensuring the visibility of the figures, for example, reference numerals may be assigned only to the main components used to describe each figure, and components having the same or substantially the same function as those previously described in the previous figures may not have reference numerals.

[0033] In imaging devices such as smartphones, there are already known techniques for synthesizing multiple captured images to achieve high image quality, such as HDR (High Dynamic Range) synthesis and NR (Noise Reduction).

[0034] For example, in scenes where the subject is moving, the movement of the subject can produce artifacts called motion blur during the acquisition of multiple images for composition. Motion blur can be improved by making the synthesized images as close as possible in time. Therefore, techniques are known to acquire multiple images in short time intervals by driving the imaging device at a high frame rate in a preview state before shooting.

[0035] However, there is a problem with increased power consumption due to driving the imaging device at a high frame rate in advance. From the perspective of heat dissipation and battery capacity, smaller imaging devices such as smartphones are more susceptible to the effects of increased power consumption.

[0036] For example, power consumption can be reduced when an imaging device operates at a low frame rate in preview mode before shooting, and then at a higher frame rate before shooting. In this case, many smartphones employ Zero Shutter Latency (ZSL) to reduce the delay between the user's shooting command and the controller's detection of the shooting command. In ZSL operation, images captured during preview are temporarily stored, and when the controller detects the shooting command, a composite image is generated based on the time point at which the user's shooting command was executed, using the stored captured image. Therefore, in smartphones employing ZSL, it is difficult to reduce power consumption simply by changing the frame rate to a high frame rate before shooting.

[0037] Furthermore, there is a problem in smartphones using ZSL: even if the frame rate is changed at the time the system detects the user's shooting command, the image captured at the time the user's shooting command is executed has already been taken at a low frame rate, so motion blur in the synthesized image cannot be suppressed.

[0038] Therefore, the following embodiments will describe an imaging device, imaging device control method, program, and computer program product that are capable of obtaining images with less motion blur while suppressing power consumption during preview.

[0039] Note that the following embodiments illustrate a digital imaging device configured to perform ZSL (Zero-Simulation Lens) from the user's shooting command. Furthermore, the following embodiments illustrate an imaging device configured to perform high-image-quality processing, such as HDR (High Dynamic Range) synthesis and NR (Noise Reduction), which obtains a synthesized image using multiple captured images acquired during ZSL operation.

[0040] First Embodiment

[0041] Figure 1 This is a diagram illustrating an example configuration of an imaging device according to a first embodiment. (See diagram for example.) Figure 1 As shown, the imaging device 1 includes an imaging unit 10, a controller 21, a digital signal processor 23, a memory 25, an input interface 27, and a gyroscope sensor 29. The imaging unit 10 is electrically connected to the DSP (Digital Signal Processor) 23. For example, the controller 21, the digital signal processor 23, the memory 25, the input interface 27, and the gyroscope sensor 29 are connected to communicate with each other via signal lines such as bus 31.

[0042] Imaging unit 10 images the subject area to generate a captured image (image data). For example... Figure 1 As shown, the imaging unit 10 includes an optical system 11, an image sensor 13, and an analog front end (AFE) 15.

[0043] The optical system 11 includes optical elements configured to form an image of a beam of light from a subject on the imaging surface 131 of the image sensor 13. Note that... Figure 1 For example, a single lens is used as an optical element in optical system 11, but this embodiment is not limited to this. Optical system 11 can have the desired imaging performance by using at least one optical element with power. In other words, optical system 11 can consist of a compound lens including at least one single lens, or it can consist of a combination of a lens system and a reflection system.

[0044] Image sensor 13 images the subject area to generate an image signal. Image sensor 13 is arranged on the optical axis of optical system 11. Image sensor 13 is arranged at the position where optical system 11 forms an image of the light beam from the subject. Image sensor 13 can suitably employ solid-state imaging devices such as CCD (electrically coupled device) and CMOS (complementary metal-oxide-semiconductor).

[0045] AFE 15 is an analog signal processor configured to perform analog processing, such as amplification, on image signals read from image sensor 13.

[0046] Optionally, AFE 15 includes a correlated dual sampler (CDS), a gain-controlled amplifier (GCA), and an analog-to-digital converter (ADC). The CDS is positioned after the image sensor 13 and performs noise suppression on the image signal from the image sensor 13. The GCA is positioned after the CDS and amplifies the image signal whose noise has been removed by the CDS, according to the control of the controller 21. The ADC is positioned after the GCA and converts the GCA-amplified image signal into digital image data (the captured image). Note that some or all of the functions of AFE 15 can be implemented internally within the image sensor 13.

[0047] Note that the imaging unit 10 is configured to be able to change the focus position. Here, "able to change the focus position" means that for each of at least two object points located at different positions along the optical axis of the optical system 11, the image formed on the imaging surface 131 can be smaller than the diameter of an allowable circle of confusion. For example, the diameter of the allowable circle of confusion is defined based on the pixel pitch of the image sensor 13 or the imaging performance of the optical system 11. In other words, the imaging unit 10 is configured to be able to focus or blur (defocus) any subject. Optionally, the imaging unit 10 is configured to be able to move at least one of the image-side focus position of the optical system 11, the object-side focus position of the optical system 11, and the imaging surface 131 of the image sensor 13 along the optical axis of the optical system 11.

[0048] The controller 21 controls each component of the imaging device 1 according to a program stored in the internal memory or memory 25. The controller 21 includes a processor and memory as hardware resources. The processor can suitably employ various processors, such as CPU (Central Processing Unit), DSP (Digital Signal Processor), ASIC (Application-Specific Integrated Circuit), and FPGA (Field-Programmable Gate Array). Furthermore, the memory can suitably employ various types of memory, such as ROM (Read-Only Memory), flash memory, and RAM (Random Access Memory). Note that the controller 21 can be a microcomputer.

[0049] The DSP 23 is located behind the AFE 15 and performs various image processing operations required for displaying and recording images from the AFE 15. Image processing includes, for example, optical black (OB) subtraction, white balance (WB) correction, demosaicing, color conversion, gamma conversion, noise reduction, magnification / reduction, compression, etc.

[0050] Memory 25 stores the programs required for the operation of imaging device 1. Furthermore, memory 25 stores information required for various processes of imaging device 1. This information includes, for example, information indicating the correspondence between the motion magnitude of imaging device 1 and the frame rate. Additionally, memory 25 temporarily stores various data therein, such as images output from DSP 23 and processing data from controller 21. Memory 25 includes non-volatile memory as hardware resources, such as ROM and flash memory, as well as volatile memory, such as DRAM (Dynamic RAM), SDRAM (Synchronous DRAM), and SRAM (Static RAM).

[0051] Input interface 27 accepts input operations from the user in imaging device 1 via various input devices, such as touch panel displays, switches, buttons, keyboards, and microphones that receive the user's voice input. As an example, input interface 27 accepts a release operation (shooting command) instructing the user to take a picture.

[0052] The gyroscope sensor 29 is configured to output a detection signal corresponding to the magnitude of motion, such as jitter, of the imaging device 1.

[0053] Figure 2This is an example diagram illustrating the functional configuration of the controller 21 according to the first embodiment. The controller 21 implements the functions of the imaging control module 211, the image prediction module 213, and the image command detection module 215 by executing a program developed in internal memory or RAM of memory 25 via a processor. Here, the controller 21 implementing the function of the imaging control module 211 is an example of an imaging control unit. Furthermore, the controller 21 implementing the function of the image prediction module 213 is an example of an image prediction unit. Furthermore, the controller implementing the function of the image command detection module 215 is an example of a command detection unit.

[0054] Note that modules 211, 213, and 215 can be implemented by a single processor or by a combination of multiple independent processors. Furthermore, each of modules 211, 213, and 215 can be implemented by integration or distribution across multiple processors.

[0055] Based on the AE evaluation value indicating the brightness of the subject in the captured image, the imaging control module 211 performs automatic exposure (AE) processing to set imaging conditions including aperture and shutter speed values. The imaging control module 211 performs AE processing, for example, by using a user's first release operation as a trigger.

[0056] Furthermore, based on focus information acquired from images, the imaging control module 211 performs autofocus (AF) adjustment processing to control the driving of at least one of the focusing lens and image sensor 13 included in the optical system 11. The focus information is, for example, an AF evaluation value (contrast value) calculated from the captured image. Additionally, when the image sensor 13 is configured to have focus detection pixels, the focus information can be the amount of defocus calculated from the output of the focus detection pixels.

[0057] Furthermore, the imaging control module 211 performs the shooting process of the imaging unit 10 to acquire a captured image. The imaging control module 211 performs the shooting process, for example, by using a second release operation (shooting command) from the user as a trigger. Here, the first and second release operations include operations such as tapping any object on the touch panel display (not shown) during the preview display. Note that the preview display can also be referred to as a live view display.

[0058] Furthermore, the imaging control module 211 performs the imaging process of controlling the imaging unit 10 and the memory 25 to obtain a composite image, i.e., imaging process utilizing ZSL operation. In ZSL operation, the imaging control module 211 temporarily stores the captured images obtained during preview in the memory 25. When the shooting command detection module 215 detects a user operation (shooting command), the imaging control module 211 reads multiple captured images obtained before a predetermined time from the detection point from the memory 25. The imaging control module 211 performs high image quality processing, synthesizing the read multiple captured images using the DSP 23 to obtain a composite image.

[0059] Furthermore, in the case of the shooting state predicted by the shooting prediction module 213, the imaging control module 211 changes the control of the image sensor 13 in ZSL operation. In this shooting state, the user's shooting command can be executed at a time point before the shooting command detection module 215 detects the user's shooting command.

[0060] As an example, the imaging control module 211 changes the frame rate used to image the pixel signals from the image sensor 13 based on the shooting state predicted by the shooting prediction module 213. Details of the control changes to the image sensor 13 will be described below.

[0061] In the shooting process with ZSL operation, the shooting prediction module 213 performs shooting prediction to predict the shooting state. Here, the shooting state prediction in the shooting prediction includes directly or indirectly determining what shooting state the imaging device is in. In addition, the shooting prediction includes directly or indirectly determining the temporal proximity of the shooting state to the shooting state.

[0062] As an example, the shooting prediction module 213 performs shooting prediction based on the magnitude of motion of the imaging device 1. For instance, the shooting prediction module 213 detects the magnitude of motion of the imaging device 1 based on the detection signal output from the gyroscope sensor 29. Then, the shooting prediction module 213 predicts the shooting state based on whether the magnitude of motion of the imaging device 1 is within a predetermined threshold range. The details of the shooting prediction will be described below.

[0063] Note that the image prediction module 213 can perform image prediction by using a threshold for the detection signal from the gyroscope sensor 29 instead of a threshold for the magnitude of motion of the imaging device 1. Alternatively, the image prediction module 213 can perform image prediction by using a threshold for the convergence rate of motion magnitude, which indicates how much the motion magnitude has converged relative to the motion of the imaging device 1 during the first period of preview display.

[0064] The shooting instruction detection module 215 detects the shooting instruction executed by the user based on the output of the input interface 27, for example, according to the second release operation (shooting instruction).

[0065] Here, the imaging process with ZSL operation in the imaging device 1 according to the embodiment will be described. Figure 3 This is a diagram illustrating the image sensor's image capture prediction and control changes in the control processing according to the first embodiment.

[0066] Here, we will refer to Figure 3 This describes a scenario where the user is holding the imaging device 1 and the object is being photographed. Note that it is assumed that the imaging device 1 has already started previewing via the imaging control module 211 and started photographing prediction via the shooting prediction module 213.

[0067] First period

[0068] In this embodiment, the period during which the composition has not yet been determined is referred to as the first period. During the first period, for example, while holding the imaging device 1, the user considers the composition for the subject while viewing the preview display. At this time, since the composition is not yet determined, the user moves the imaging device 1 to fit the subject within the viewing angle while viewing the preview display, or moves the imaging device 1 to try various compositions of the subject. Therefore, as... Figure 3 As shown, the imaging device 1 moves significantly during the first time period.

[0069] When the imaging device 1 moves significantly, the position of the object relative to the viewpoint changes between the multiple captured images 601, 602, and 603 acquired at a frame rate 411 of the first time period. Therefore, when the imaging device 1 moves significantly, motion blur 501 appears in the composite image 611 generated by synthesizing the multiple captured images 601, 602, and 603.

[0070] Second period

[0071] The period following the first period, in which the composition is determined, is called the second period. The second period is the time when the user's shooting command is close to the shooting state. For example, the second period is approximately 1 to 3 seconds. During the second period, the user adjusts the orientation and position of imaging device 1 while viewing the preview display, for example, to fix the composition of the object. Therefore, as... Figure 3 As shown, the motion of imaging device 1 in the second time period is less than that in the first time period.

[0072] As an example, when the magnitude of motion of imaging device 1 is within a predetermined range of threshold 401 in the second time period, the shooting prediction module 213 predicts that the current state is the shooting state in the second time period when the shooting command is approaching.

[0073] As an example, when the shooting prediction module 213 predicts that the current state is the shooting state of the second time period when the shooting command is approaching, the imaging control module 211 changes the control to drive the image sensor 13 at the frame rate 413 of the second time period, which is higher than the frame rate 411 of the first time period.

[0074] As described above, when the current state approaches the shooting state of the third time period, a higher frame rate is set for imaging the pixel signals from the image sensor 13. As a result, since setting a higher frame rate increases the likelihood of the user executing shooting instructions, even if the user executes shooting instructions at this stage, motion blur can be reduced while lowering power consumption compared to controlling the image sensor 13 at a high frame rate, for example, at the time when the preview display begins.

[0075] Note that multiple frame rates 413 for the second time period can be set according to the magnitude of the movement of the imaging device 1. In other words, the frame rate 413 for the second time period can have a higher value as the current state approaches the shooting state of the third time period. In this case, each of the multiple frame rates 413 for the second time period can have a higher value as the current state approaches the shooting state of the third time period. For example, based on multiple predetermined thresholds 401 for the second time period, the shooting prediction module 213 predicts the temporal proximity to the shooting state of the third time period as the shooting state at its time point. At this time, the imaging control module 211 can select the frame rate 413 to be set for the second time period from a table of multiple predetermined frame rates indicating the various thresholds 401 of the second time period. Alternatively, the imaging control module 211 can seamlessly set the frame rate 413 for the second time period according to the proximity to the shooting state by using a relational expression such as the operating magnitude of the imaging device 1, which indicates the relationship between the frame rate and the proximity to the shooting state of the third time period.

[0076] Third period

[0077] The period following the second period, which essentially determines the composition, is called the third period. The third period is the time during which the user's shooting instructions can be executed. For example, the third period is approximately one second. During the third period, for example, the user may time the shooting or fine-tune the direction and position of the imaging device 1 while viewing the preview display. Therefore, as... Figure 3 As shown, the motion of imaging device 1 in the third time period is smaller than that in the second time period.

[0078] As an example, when the magnitude of motion of imaging device 1 is within a predetermined range of threshold 403 in the third time period, the shooting prediction module 213 predicts that the current state is the shooting state in the third time period in which the user's shooting command can be executed.

[0079] As an example, in a shooting state where the shooting prediction module 213 predicts that a user's shooting command may be executed in the third shooting period, the imaging control module 211 sets the frame rate for imaging the pixel signals from the image sensor 13 to a higher time point than before the third shooting state, i.e., the first and second shooting periods. Here, the frame rate 415 set in the third shooting state where the user's shooting command is predicted to be executed is an example of the first frame rate. Furthermore, the imaging control module 211 initiates ZSL operation. In other words, for example, the imaging control module 211 acquires the captured image at the frame rate 415 of the third shooting period and stores it sequentially in the memory 25.

[0080] After the third period

[0081] This will describe the situation where the user's shooting instructions are executed after the third time period.

[0082] As an example, the shooting instruction detection module 215 detects the user's shooting instruction based on the output of the input interface 27.

[0083] The imaging control module 211 reads from the memory 25 a plurality of captured images 621, 622, and 623 acquired before the time point at which the user's shooting command is detected. Then, by using the plurality of captured images 621, 622, and 623 read, the imaging control module 211 generates image 631 at the time point at which the user's shooting command is executed.

[0084] A higher frame rate is set in the third time period than in the first and second time periods. Therefore, the position of the subject relative to the viewpoint changes very little between the multiple captured images 621, 622, and 623 obtained at the frame rate 415 of the third time period. As a result, the composite image 631 generated by combining the multiple captured images 621, 622, and 623 has a small motion blur 503.

[0085] Furthermore, after setting a high frame rate during the shooting state of the third time period when the user's shooting instruction can be executed, for example, when the shooting instruction detection module 215 detects the user's shooting instruction, the imaging control module 211 sets a frame rate 417 lower than the frame rate 415 of the third time period. For example, the low frame rate 417 set after the shooting state can be the same frame rate as a time point before the shooting state of the third time period, such as the first and second time periods, or it can be a frame rate different from that time point. Here, frame rate 417 is an example of the second frame rate.

[0086] As described above, in the imaging device 1 according to this embodiment, during the step of starting preview display, i.e., during the first time period, control of driving the image sensor 13 is performed at a frame rate 411, which is the normal frame rate of the first time period. Furthermore, when a shooting state in the second time period, where a shooting command is predicted to be approaching, is predicted, the control of the image sensor 13 is changed to drive the image sensor 13 at a frame rate 413, which is higher than the frame rate 411 of the first time period. Furthermore, when a shooting state in the third time period, where a shooting command can be executed, is predicted, the control of the image sensor 13 is changed to drive the image sensor 13 at a frame rate 415, which is higher than the frame rate 413 of the second time period. As a result, compared to control of driving the image sensor 13 at a high frame rate at the time of starting preview display, for example, motion blur can be reduced while lowering power consumption.

[0087] Furthermore, for example, when the imaging device 1 is a smartphone, each module according to this embodiment, such as the shooting command detection module 215, is implemented through a shooting application installed in the smartphone. Therefore, for example, the shooting command detection module 215 detects the output of the input interface 27 based on the user's shooting command via the smartphone's OS (operating system). As described above, the imaging device 1, such as a smartphone, has a system delay from the time the user executes the shooting command to the time the shooting command detection module 215 detects the shooting command. Therefore, even if frame rate control is performed at the time the shooting command is detected, the image captured at the time the shooting command was executed can be acquired at the frame rate prior to the change.

[0088] In this scenario, according to the imaging device 1 of this embodiment, when the shooting state of the third time period in which the shooting command can be executed is predicted, the ZSL operation is initiated, and the control of the image sensor 13 is changed to drive the image sensor 13 at a frame rate 415 higher than that of the first and second time periods. Therefore, according to the imaging device 1 of this embodiment, even with system delays, the control of the image sensor 13 can be changed at an appropriate time, neither too early nor too late, based on the shooting prediction. Optionally, according to the imaging device 1 of this embodiment, since low frame rate control is performed when the preview display begins, but high frame rate control is performed at the time of executing the shooting command, the captured image stored in the ZSL operation can be acquired at the changed high frame rate while reducing power consumption.

[0089] Note that ZSL operation can begin in either the first or second time period. Even in this case, motion blur can be reduced while lowering power consumption compared to starting the preview display and setting a high frame rate simultaneously.

[0090] Note that, based on the time sequence of the motion of imaging device 1, the second time period may not exist.

[0091] Here, an operational example of the imaging device 1 according to this embodiment will be described. Figure 4 This is a flowchart example illustrating the control processing flow according to the first embodiment. For example, Figure 4 The process shown is implemented in a scene where moving objects are being filmed, but it can also be implemented in another scene, or it can be implemented regardless of the scene.

[0092] First, the preview display of the imaging control module 211 and the shooting prediction of the shooting prediction module 213 are started (S101).

[0093] As an example, the imaging control module 211 initiates imaging by the image sensor 13 using imaging conditions such as a shutter speed (exposure time) for a first time period and a predetermined normal frame rate 411, and begins displaying a preview of the captured image on a display (not shown). Note that the captured image used for the preview display may be a low-quality image, with its exposure time, pixel count, etc., reduced to a lower level than the captured image according to the shooting instructions used by the user for image generation, or some image processing may be omitted or suppressed. Note that the preview display may be referred to as a live view display.

[0094] As an example, the shooting prediction module 213 begins shooting prediction based on the magnitude of motion of the imaging device 1. According to this embodiment, the shooting prediction module 213 performs shooting prediction of the shooting state by comparing the magnitude of motion of the imaging device 1 acquired based on the detection signal from the gyroscope sensor 29 with a predetermined range of thresholds 401 and 403.

[0095] When the shooting prediction module 213 predicts that the shooting state is close to the shooting command (S102: Yes), the imaging control module 211 changes the control of the image sensor 13 (S103).

[0096] As an example, when the motion magnitude of imaging device 1 is within a predetermined range of threshold 401 in the second time period, the shooting prediction module 213 predicts that the current state is a shooting state approaching a shooting command. At this time, the imaging control module 211 sets the previously defined frame rate 413 of the second time period in association with the threshold 401 in the second time period. In other words, when the current state is predicted to be a shooting state in the second time period approaching a shooting command, the imaging control module 211 changes the control of image sensor 13 using the frame rate 411 of the first time period to the control of image sensor 3 using the frame rate 413 of the second period, which is higher than the frame rate 411 of the first time period.

[0097] When the shooting prediction module 213 predicts a shooting state in which the shooting command can be executed (S104: Yes), the imaging control module 211 starts ZSL operation and simultaneously changes the control of the image sensor 13 (S105).

[0098] As an example, when the motion magnitude of imaging device 1 is within a predetermined range of threshold 403 in the third time period, the shooting prediction module 213 predicts that the current state is a shooting state in which a shooting command can be executed. At this time, the imaging control module 211 sets the previously defined frame rate 415 of the third time period in association with the threshold 403 in the third time period. In other words, when the current state is predicted to be a shooting state in the third time period in which a shooting command can be executed, the imaging control module 211 changes the control of image sensor 13 using frame rate 413 in the second time period to the control of image sensor 3 using frame rate 415 in the third period, which is higher than the frame rate 413 in the second time period. In other words, the imaging control module 211 changes the control of image sensor 13 in ZSL operation to the control of frame rate 415 in the third time period.

[0099] After executing step S105, or when the shooting prediction module 213 does not predict a shooting state that is close to the shooting command or a shooting state that can execute the shooting command (S102, S104: No), Figure 4 The process shown proceeds to step S106.

[0100] When the shooting instruction detection module 215 detects the output of the input interface 27 according to the user's shooting instruction (S106: Yes), the imaging control module 211 generates an image based on the captured image (S107).

[0101] As an example, in the case of a first or second period before the shooting state of the third period when the shooting command can be executed, the imaging control module 211 generates an image by using the captured image obtained at the time when the user's shooting command is detected.

[0102] As an example, in the case of starting the ZSL operation in the third period and the next period based on the shooting state of the third period when the shooting command can be executed, the imaging control module 211 generates an image corresponding to the time point when the user executes the shooting command by using the captured image obtained when the user executes the shooting command through the ZSL operation.

[0103] When the changed control returns (S108: Yes), the imaging control module 211 returns the control from the frame rate 415 of the third time period to the frame rate 417, such as the frame rate 411 of the first time period and the frame rate 413 of the second time period (S109). On the other hand, when the changed control does not return (S108: No), step S109 is skipped. Note that the case of the changed control returning refers to the situation where the control of the image sensor 13 is changed in step S103 and / or step S105, and a user's shooting command is detected in step S106.

[0104] Then, if filming has not ended (S110: No), Figure 4 The process shown returns to step S102, and terminates when shooting is stopped (S110: Yes). For example, shooting is stopped when input interface 27 receives an operation instructing the user to stop shooting.

[0105] As described above, the imaging device 1 according to this embodiment is configured to detect motion of the imaging device 1, and predict a longer time for the user's shooting command when the motion is large, and a shorter time when the motion is small. Furthermore, compared to frame rates 411 and 413 in the shooting state during the first or second period when the motion of the imaging device 1 is large, the imaging device 1 is configured to set a higher frame rate 415 in the shooting mode during the third period when the motion of the imaging device 1 is small.

[0106] According to this configuration, the shooting time can be predicted, and the control of the image sensor 13 can be changed from a low frame rate to a high frame rate at an appropriate time, not too early before the predicted shooting time. Therefore, according to the technology of this embodiment, images with less motion blur can be obtained while suppressing power consumption during preview. Furthermore, according to the technology of this embodiment, even when shooting with ZSL widely used in smartphones, especially before shooting, the frame rate can be changed from a low frame rate to a high frame rate.

[0107] Second Embodiment

[0108] Here, the differences from the first embodiment will be described in detail, and explanations of repeated content will be omitted as appropriate. Figure 5 This diagram explains the image prediction in the control processing according to the second embodiment. The imaging device 1 according to this embodiment may not include the gyroscope sensor 29. On the other hand, as... Figure 5 As shown, imaging device 1 is configured to detect motion of imaging device 1 based on the captured image. For example, as Figure 5 As shown, the image prediction module 213 calculates motion vectors 505, 507, and 509 of the object P in image 645 based on at least two captured images 641 and 642, and detects the motion of the imaging device 1 based on the calculated motion vectors 505, 507, and 509. Similarly, the image prediction module 213 calculates motion vectors 511 and 513 of the subject P in image 647 based on at least two captured images 642 and 643, and detects the motion of the imaging device 1 based on the calculated motion vectors 511 and 513. Note that the motion vectors can be calculated based on features such as the outline, eyes, nose, and mouth of the object P.

[0109] As described above, the shooting prediction according to this embodiment detects the motion of the imaging device 1 based on the motion vector of the subject calculated from at least two captured images obtained using the image sensor 13. With this configuration, the same effect as in the embodiment described above is achieved without installing the gyroscope sensor 29.

[0110] Third Embodiment

[0111] Here, the differences from the first embodiment will be described in detail, and explanations of repeated content will be omitted as appropriate. Figure 6 This is a configuration example diagram of the imaging device 1 according to the third embodiment. The imaging device 1 according to this embodiment may not include the gyroscope sensor 29. On the other hand, the main body 101 of the imaging device 1 is provided with a main camera 10a and a front-facing camera 10b having different shooting directions as an imaging unit 10. The main camera 10a is similar to the imaging unit 10 according to the above embodiment. The image sensor 13 of the front-facing camera 10b is arranged to be able to capture images on the side opposite to the image sensor 13 of the main camera 10a in the optical axis direction.

[0112] Figure 7 This diagram explains the shooting prediction in the control processing according to the third embodiment. The shooting prediction module 213 detects the state of the user (subject P) as the photographer from the captured image acquired by the image sensor 13 using the front-facing camera 10b, and predicts the shooting state based on the photographer's state. For example, when the subject P is not shown as in captured image 651, when the subject P is cut off as in captured image 652, when the subject P is tilted as in captured image 653, or when the subject P is located on the periphery as in captured image 654, the shooting prediction module 213 predicts that the current state is the shooting state of the first or second time period, where the photographer does not have a pose for taking pictures using the imaging device 1, for example, the composition has not been decided. On the other hand, when the photographer is directly in front of the center portion as in captured image 657, the shooting prediction module 213 predicts that the current state is the shooting state of the third time period, where the shooting command can be executed. Note that the shooting prediction module 213 can perform shooting prediction based on the photographer's line of sight, or it can perform shooting prediction by calculating the motion vector of the photographer or the line of sight. Then, the imaging control module 211 changes the control of the image sensor 13 of the main camera 10a according to the predicted shooting state.

[0113] As described above, the shooting prediction according to this embodiment predicts whether the current state is the same as the state before shooting by observing the photographer's state without detecting the movement of the imaging device 1. With this configuration, the same effect as the above embodiment can be achieved even without installing the gyroscope sensor 29, and even when the imaging target is not in the field of view. The shooting prediction according to this embodiment is particularly useful when using a tripod 3, etc.

[0114] Fourth embodiment

[0115] Here, the differences from the first embodiment will be described in detail, and explanations of repeated content will be omitted as appropriate. Figure 8 This is a diagram illustrating the control changes of the image sensor 13 in the control processing according to the fourth embodiment. (See diagram for example.) Figure 8 As shown in (a), the imaging device 1 according to the fourth embodiment is configured to return the frame rate to a frame rate 417, which is lower than the frame rate 415 of the third time period, when the system side detects a shooting command. On the other hand, there may be a situation where the system side does not detect a shooting command. Therefore, as Figure 8 As shown in (b), the moment when the frame rate returns to a low frame rate can be the moment when the motion magnitude of the imaging device 1 increases from state 515 to state 517 and exceeds a predetermined range of threshold 403. Or, as Figure 8 As shown in (c), the time when the frame rate returns to a low frame rate can be the time elapsed after the control is changed to frame rate 415 in the third time period. Note that a different threshold than threshold 403 can be used. Furthermore, multiple thresholds can be used to return to a low frame rate in multiple steps.

[0116] As described above, when the shooting command detection module 215 detects a user's shooting command, if the movement of the imaging device 1 exceeds a predetermined threshold 403, or if a frame rate 415 for the third time period is set and a predetermined time has elapsed, the imaging device 1 according to this embodiment sets the frame rate 417 to be lower than the frame rate 415 for the third time period. With this configuration, even if a shooting command is not executed, control can be returned to a low frame rate at an appropriate time, thus reducing power consumption.

[0117] Fifth Embodiment

[0118] Here, the differences from the first embodiment will be described in detail, and explanations of repeated content will be omitted as appropriate. Figure 9This is a diagram explaining the threshold for motion detection in the control processing according to the fifth embodiment. The shooting prediction module 213 records and learns the time series of motion magnitude from the imaging device 1 and information about the time of the user's shooting command for each photographer in the memory 25. Then, based on the learned information, the shooting prediction module 213 determines thresholds 403a, 403b, and 403c for each photographer. For example, the shooting prediction module 213 can determine statistical values ​​of the motion magnitude at the time of the shooting command, such as the average and median, as the threshold for each photographer. Alternatively, the shooting prediction module 213 can determine the threshold by using the time series indicating motion and the information about the shooting command time on the input side, and by using a preset threshold and a current threshold on the output side, using the output of a machine learning model that determines the parameters. Note that the technique according to this embodiment can be applied to the threshold 401 for the second time period, the threshold at frame rate return, and the frequency used for motion detection.

[0119] As described above, according to this embodiment, the imaging device 1 learns the motion detected by the imaging device 1 for each photographer, and sets a threshold for predicting the shooting state for each photographer based on the learned information. According to this configuration, the control of the image sensor 13 can be changed at a time suitable for each photographer, depending on the magnitude of the motion of the imaging device 1 (which varies depending on the photographer).

[0120] Sixth Embodiment

[0121] Here, the differences from the first embodiment will be described in detail, and explanations of repeated content will be omitted as appropriate. Figure 10 This diagram illustrates the control changes of the image sensor 13 during the control processing according to the sixth embodiment. In the first and second time periods, the imaging control module 211 controls the driving of the image sensor 13 in a single-sampling mode 421, which performs one sample per frame. Conversely, in the third time period, the imaging control module 211 controls the driving of the image sensor 13 in a multi-sampling mode 425, which performs multiple samples per frame.

[0122] As described above, the imaging device 1 according to this embodiment changes the number of sampling operations performed on the pixel signals from the image sensor 13 based on the predicted shooting state. With this configuration, the increase in power consumption due to multiple sampling operations can be suppressed, while noise is reduced through multiple sampling operations.

[0123] Seventh Embodiment

[0124] Here, the differences from the first embodiment will be described in detail, and explanations of repeated content will be omitted as appropriate. Figure 11This diagram illustrates the control change of the image sensor 13 in the control processing according to the seventh embodiment. In the first and second time periods, the imaging control module 211 does not change the exposure time for each frame. In this case, even if multiple captured images obtained without changing the exposure time are synthesized, such as composite image 661, where the exposure is suitable for the bird object 523, the exposure is too high and saturated for the cloud object 521, and the exposure is insufficient for the human object 525, proper exposure cannot be achieved for each object 521, 523, and 525. On the other hand, when performing an HDR operation that pre-changes the exposure time for each frame, power consumption increases, and at the time the user shooting command is detected, the system cannot switch to HDR operation due to a delay. In this case, in the third time period, the imaging control module 211 according to this embodiment changes the control of the image sensor 13 to HDR mode, where different exposure times are applied to each frame. In other words, the imaging control module 211 temporarily stores multiple captured images 671, 672, and 673 with different exposure times in the memory 25 and implements the HDR mode in the ZSL operation that generates the composite image 681 via HDR.

[0125] As described above, the imaging device 1 according to this embodiment executes an HDR mode that changes the exposure time of the image sensor 13 for each frame based on the predicted shooting state. According to this configuration, ZSL operation can be performed simultaneously with proper exposure of each of objects 531, 533, and 535 via the HDR mode, similar to the composite image 681.

[0126] Eighth embodiment

[0127] Here, the differences from the first embodiment will be described in detail, and explanations of repeated content will be omitted as appropriate. Figure 12 This diagram illustrates the control changes of the image sensor 13 in the control processing according to the eighth embodiment. The shooting prediction module 213 can perform shooting prediction based on object recognition. For example, in the state of a captured image 691 where no object is detected, the shooting prediction module 213 predicts that the current state is the shooting state of the first time period. When a target object is detected, the shooting prediction module 213 predicts that the current state is the shooting state of the third time period. The time when the target object is detected is, for example, the time point T1 when the captured image 692 detects object 541 (e.g., a face). Note that when assuming object 543 is the target object, the time when the target object is detected can be, for example, the time point T2 when object 543 enters the field of view in captured image 694, the time T3 when object 543 fully enters the field of view in captured image 695, or the time T4 when another object 541 is no longer in the field of view in captured image 696. At this time, the imaging control module 211 increases the frame rate to the frame rate 413 of the third time period.

[0128] Furthermore, the imaging control module 211 may return the frame rate to a low frame rate at time point T5 (case 1) after a predetermined time has elapsed, at time point T3 (case 2) after the target object 541 is no longer in the field of view and at time point T2 (case 3) after another object 543 (e.g., a dog) enters the field of view and enters the captured image 694. Alternatively, the imaging control module 211 may reset the modified elapsed time count to a high frame rate at time point T2 after another target object 543 (e.g., a dog) enters the field of view and return the frame rate to a low frame rate at time point T6 after a predetermined time has elapsed from that time point (case 4).

[0129] As described above, the shooting prediction according to this embodiment predicts the shooting state of the third time period when a target object is detected in the captured image obtained using image sensor 13. Even with this configuration, the same effect as the embodiment described above can be achieved.

[0130] In the above embodiments, for example, when a first release operation such as a user selecting (tapping) a target object on a touch panel display is detected, a shooting prediction can be performed to predict whether the current state is a shooting state in the second or third time period.

[0131] Note that the techniques according to the above embodiments can be appropriately combined. For example, in shooting prediction, the motion of the imaging device 1 can be detected in various ways. For example, shooting prediction can be performed based on at least two of the motion of the imaging device 1, the state of the photographer, and the state of the subject. For example, in the control of the image sensor 13, at least two of the frame rate, the number of samples, and the exposure time can be changed. In other words, at least one of the above-described shooting prediction and at least one control of the image sensor 13 can be appropriately combined.

[0132] Note that the techniques according to the above embodiments can be applied to other processes for obtaining synthetic images, such as video capture and burst shooting.

[0133] Note that some or all of the processing performed by the imaging device 1 according to this embodiment can be implemented by software.

[0134] The program executed by the computer of the imaging device 1 according to this embodiment is recorded and provided in an installable or executable format in a computer-readable non-transient recording medium (computer program product), such as flash memory (semiconductor memory), USB (Universal Serial Bus) memory, SSD (Solid State Drive), and HDD (Hard Disk Drive).

[0135] Furthermore, the program executed by the imaging device 1 according to this embodiment can be configured to be provided via a computer connected to a network such as the Internet and downloaded over the network. Additionally, the program executed by the imaging device 1 according to this embodiment can be configured to be provided or distributed via a network such as the Internet.

[0136] Furthermore, the program executed by the imaging device 1 according to this embodiment can be configured to be pre-integrated into ROM or the like and provided.

[0137] According to at least one of the above embodiments, images with less motion blur can be obtained while suppressing power consumption during preview.

[0138] While certain embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the invention. In fact, the novel embodiments described herein can be embodied in various other forms; furthermore, various omissions, substitutions, and modifications can be made to the forms of the embodiments described herein without departing from the spirit of the invention. The appended claims and their equivalents are intended to cover forms or modifications falling within the scope and spirit of the invention.

[0139] (Supplementary Explanation)

[0140] (1) An imaging device configured for digital operation and capable of zero shutter lag (ZSL) operation, said operation generating an image at the time of executing the user's shooting command by using a captured image obtained before the time of detecting a user's shooting command, said device comprising:

[0141] The shooting prediction unit is configured to predict the user's shooting commands; and

[0142] The imaging control unit is configured to change the control of the image sensor in ZSL operation when the shooting prediction unit predicts that the shooting state can execute the user's shooting command at a time point before the shooting command is detected.

[0143] (2) In the imaging device according to (1), the imaging control unit is configured to set the frame rate for imaging the pixel signals from the image sensor to a time point higher than the time point before the shooting state in a predicted shooting state.

[0144] (3) In the imaging device according to (1) or (2), the imaging control unit is configured to set a higher frame rate for imaging the pixel signals from the image sensor as the state of the imaging device approaches the shooting state.

[0145] (4) In the imaging device according to (2) or (3), the imaging control unit is configured to set a first frame rate higher than a time point before the predicted shooting state in the shooting state, and then set a second frame rate lower than the first frame rate.

[0146] (5) In the imaging device according to (4), the imaging control unit is configured to set the second frame rate to be lower than the first frame rate when a user’s shooting instruction is detected, when the motion of the imaging device is greater than a predetermined threshold, or when a predetermined time has elapsed since the first frame rate was set.

[0147] (6) In the imaging apparatus according to any one of (1) to (5), the change in control of the image sensor is a change in the number of sampling operations for sampling the pixel signals from the image sensor.

[0148] (7) In the imaging apparatus according to any one of (1) to (6), the imaging control unit is configured to change the exposure time of the image sensor in a predicted shooting state.

[0149] (8) In the imaging device according to any one of (1) to (7), the shooting prediction unit is configured to detect the motion of the imaging device and predict that the state of the imaging device is a shooting state when the detected motion magnitude of the imaging device is less than a predetermined threshold.

[0150] (9) In the imaging device according to (8), the imaging device further includes a gyroscope sensor; and

[0151] The image prediction unit is configured to detect the motion of the imaging device based on the output of the gyroscope sensor.

[0152] (10) In the imaging device according to (8) or (9), the image prediction unit is configured to calculate the motion vector of an object from at least two captured images obtained by the image sensor, and detect the motion of the imaging device based on the motion vector.

[0153] (11) In the imaging device according to any one of (8) to (10), the shooting prediction unit is configured to learn the motion of the imaging device detected for each photographer and set a threshold for each photographer to predict the state of the imaging device as a shooting state based on the learned information.

[0154] (12) In the imaging apparatus according to any one of (1) to (11), the imaging apparatus further includes another image sensor arranged to image a side opposite to the image sensor in the optical axis direction of the image sensor used for imaging, and

[0155] The shooting prediction unit is configured to detect the state of the photographer using the captured image obtained by the other image sensor, and to predict the shooting state based on the state of the photographer.

[0156] (13) In the imaging device according to any one of (1) to (12), the shooting prediction unit is configured to predict the state of the imaging device as a shooting state when a target object is detected in a captured image acquired from the image sensor.

[0157] (14) A method for controlling a user digital imaging device, the digital imaging device being configured to perform zero shutter lag (ZSL) operation, the operation of generating an image at the time of executing the user's shooting command by using a captured image obtained before the time of detecting the user's shooting command, the method comprising:

[0158] Predicting the user's shooting instructions; and

[0159] If the shooting state is predicted to execute the user's shooting command at a time point before the shooting command is detected, the control of the image sensor in ZSL operation is changed.

[0160] (15) A program executed by a computer of a digital imaging device, the digital imaging device being configured to perform zero shutter lag (ZSL) operation, the operation of generating an image at the time of executing the user's shooting command by using a captured image obtained before the time of detecting the user's shooting command, the program causing the computer to perform:

[0161] Predicting the user's shooting instructions; and

[0162] If the shooting state is predicted to execute the user's shooting command at a time point before the shooting command is detected, the control of the image sensor in ZSL operation is changed.

[0163] (16) A computer program product storing a program executed by a computer of an imaging device, the program being based on (15).

[0164] [Explanation of letters or numbers]

[0165] 1: Imaging equipment

[0166] 10: Imaging Unit

[0167] 10a: Main camera

[0168] 10b: Front-facing camera

[0169] 101: Main Body

[0170] 11: Optical System

[0171] 13: Image Sensor

[0172] 131: Imaging surface

[0173] 15: Simulate the front end

[0174] 21: Controller

[0175] 211: Imaging Control Module

[0176] 213: Shooting Prediction Module

[0177] 215: Shooting Command Detection Module

[0178] 23: Digital Signal Processor

[0179] 25: Memory

[0180] 27: Input Interface

[0181] 29: Gyroscope sensor

[0182] 31: Bus

Claims

1. An imaging device configured for digital operation and capable of zero shutter lag (ZSL) operation, said operation generating an image at the time of executing the user's shooting command by using a captured image obtained before the time of detecting a user's shooting command, said device comprising: The shooting command detection module is configured to detect the user's shooting commands; A shooting prediction unit is configured to predict the user's shooting state, wherein it detects motion of the imaging device and predicts that the state of the imaging device is a shooting state when the detected motion magnitude of the imaging device is less than a predetermined threshold; and An imaging control unit is configured to change the control of the image sensor in ZSL operation when the shooting prediction unit predicts a shooting state in which the user's shooting command can be executed at a time point before the shooting command is detected, wherein, in the predicted shooting state, the frame rate for imaging the pixel signals from the image sensor is set to a higher frame rate than at the time point before the shooting state.

2. The imaging device according to claim 1, wherein the imaging control unit is configured to set a higher frame rate for imaging the pixel signals from the image sensor during a second period when the shooting command is close to the shooting state.

3. The imaging device according to claim 2, wherein in the shooting state of a third time period in which a user's shooting command is predicted to be executed, the imaging control unit is configured to set a first frame rate higher than the frame rate of the second time period, and then, after the shooting state of the third time period, set a second frame rate lower than the first frame rate.

4. The imaging device according to claim 3, wherein the imaging control unit is configured to set the second frame rate to be lower than the first frame rate when a user's shooting instruction is detected, when the movement of the imaging device is greater than a predetermined threshold, or when a predetermined time has elapsed since the first frame rate was set.

5. The imaging apparatus of claim 1, wherein the change in control of the image sensor is a change in the number of sampling operations for sampling pixel signals from the image sensor.

6. The imaging device of claim 1, wherein the imaging control unit is configured to change the exposure time of the image sensor in a predicted shooting state.

7. The imaging device according to claim 1, wherein: The imaging device also includes a gyroscope sensor; and The image prediction unit is configured to detect the motion of the imaging device based on the output of the gyroscope sensor.

8. The imaging device of claim 1, wherein the image prediction unit is configured to calculate motion vectors of objects from at least two captured images acquired by the image sensor, and to detect motion of the imaging device based on the motion vectors.

9. The imaging device of claim 1, wherein the shooting prediction unit is configured to learn the detected motion of the imaging device for each photographer and set a threshold for each photographer to predict the state of the imaging device as a shooting state based on the learned information.

10. A method for controlling a user digital imaging device, the digital imaging device being configured to perform zero shutter lag (ZSL) operation, the operation of generating an image at the time of executing the user's shooting command by using a captured image obtained before the time of detecting a user's shooting command, the method comprising: Detect the user's shooting command; Predicting the user's shooting state, wherein motion of the imaging device is detected, and when the detected motion magnitude of the imaging device is less than a predetermined threshold, the state of the imaging device is predicted to be a shooting state; and If a shooting state is predicted that the user's shooting command can be executed at a time point before the shooting command is detected, the control of the image sensor in the ZSL operation is changed, wherein, in the predicted shooting state, the frame rate for imaging the pixel signals from the image sensor is set to a time point higher than the time point before the shooting state.

11. A computer program product storing a program executed by a computer of a digital imaging device, causing the digital imaging device to perform the control method of the user digital imaging device of claim 10.

Images