Method and device for image acquisition

Abstract

Methods and apparatus for image acquisition are provided. The method for image acquisition includes: acquiring, via a sensor, an image frame including a shutter closing frame corresponding to shutter closing by performing shutter closing during continuous photographing; acquiring a measurement signal corresponding to the target image frame; based on a target shutter-off frame corresponding to the target image frame, removing a first residual signal corresponding to the target shutter-off frame from a measurement signal corresponding to a shutter-off period frame, the shutter-off period frame including an image frame between the target image frame and the target shutter-off frame; and generating a target image frame from which the second residual signal is removed based on one or more of the shutter closing period frames from which the first residual signal is removed.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2023-0150106 filed on November 2, 2023, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field

[0002] The following disclosure relates to methods and apparatus having image acquisition. Background Art

[0003] The optical camera system can convert light incident to the imaging sensor through the lens into an electrical signal through a photodiode and construct an image by measuring the electrical signal. In the process of converting photons into electrical signals through a photodiode, the converted charge can be accumulated in the pixel during the exposure time.

[0004] High-speed continuous shooting can speed up the operation speed of ordinary cameras and use the same techniques as general shooting to acquire images. In high-speed shooting, the exposure time can be reduced due to the fast image acquisition rate per frame to acquire multiple images at high speed. The short exposure time can reduce the amount of light received by the camera sensor system and therefore can reduce the size of the acquired signal. As the speed of high-speed shooting is accelerated, its effect can be intensified, resulting in degradation of image quality.

[0005] Analog / digital amplification that increases the size of the electrical signal can be used to compensate for the reduced amount of light during high-speed shooting, but analog / digital amplification has the problem of noise amplification during processing. In order to increase the amount of received light, the area per unit pixel can be increased by increasing the size of the sensor, however, this has limitations such as the impact on the size of the form factor. Summary of the invention

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0007] In one or more general aspects, a method implemented by a processor with image acquisition includes: acquiring image frames including shutter-closed frames corresponding to shutter closure via a sensor by performing shutter closure during continuous shooting; acquiring a measurement signal corresponding to a target image frame; based on a target shutter-closed frame corresponding to the target image frame, removing a first residual signal corresponding to the target shutter-closed frame from the measurement signal corresponding to a shutter-closed period frame, the shutter-closed period frame including image frames between the target image frame and the target shutter-closed frame; and generating a target image frame from which a second residual signal is removed based on one or more of the shutter-closed period frames from which the first residual signal is removed.

[0008] The removing of the first residual signal may include: estimating the first residual signal based on the target shutter-closed frame and parameter information of the sensor; and removing the first residual signal from the measurement signal corresponding to the shutter-closed period frame.

[0009] The estimating of the first residual signal may include estimating a residual signal due to a signal generated before shutter closing corresponding to the target shutter closing frame based on the target shutter closing frame and information about a photodiode forming the sensor.

[0010] The removing of the first residual signal may include removing the first residual signal from the measurement signal corresponding to the shutter-closed period frame by inputting the target shutter-closed frame into the first artificial neural network model.

[0011] The step of generating the target image frame from which the second residual signal is removed may include generating the target image frame from which the second residual signal is removed by inputting a shutter closing period frame from which the first residual signal is removed into a second artificial neural network model.

[0012] The method may include restoring an image frame corresponding to the target shutter-closed frame by inputting a predetermined number of previous and next image frames based on the target shutter-closed frame into a third artificial neural network model.

[0013] Shutter-closed framing can construct an image using the residual charge of the photodiodes forming the sensor without acquiring a signal during the time corresponding to the shutter being closed.

[0014] The generating of the target image frame from which the second residual signal is removed may include generating the target image frame from which a residual signal due to a signal after shutter closing corresponding to the target shutter closing frame is removed.

[0015] The sensor may include an organic photodiode (OPD).

[0016] The sensor may include a hybrid image sensor including an OPD and a silicon photodiode.

[0017] The sensor may include an image sensor and an optical structure for improving light sensitivity.

[0018] The step of acquiring the image frame may include: determining a frequency and a timing of shutter closing; and acquiring the image frame by performing the shutter closing according to the determined frequency and timing.

[0019] The step of acquiring the image frame may include acquiring the image frame by periodically performing shutter closing.

[0020] The removing of the first residual signal may include: acquiring a residual signal between shutter-closed period frames; and acquiring a shutter-closed period frame from which the first residual signal is removed based on the residual signal between the shutter-closed period frames.

[0021] The step of generating a target image frame from which the second residual signal is removed may include: generating a residual signal between shutter-closed period frames from which the first residual signal is removed; and generating a target image frame from which the second residual signal is removed based on the residual signal between shutter-closed period frames from which the first residual signal is removed.

[0022] The method may include acquiring a short-term image frame corresponding to an exposure time shorter than an exposure time of the image frame; and generating a high dynamic range (HDR) image based on the short-term image frame.

[0023] In one or more general aspects, a non-transitory computer-readable storage medium may store instructions that, when executed by one or more processors, configure the one or more processors to perform any one, any combination, or all of the operations and / or methods described herein.

[0024] In one or more general aspects, an electronic device includes: an image signal acquisition device, including: a sensor, configured to: acquire image frames including shutter-closed frames corresponding to shutter closure by performing shutter closure during continuous shooting, and acquire measurement signals corresponding to target image frames; and an image signal recovery device, configured to: remove a first residual signal corresponding to the target shutter-closed frame from the measurement signal corresponding to the shutter-closed period frame based on a target shutter-closed frame corresponding to the target image frame, and generate a target image frame from which a second residual signal is removed based on one or more of the shutter-closed period frames from which the first residual signal is removed, the shutter-closed period frames including image frames between the target image frame and the target shutter-closed frame.

[0025] To remove the first residual signal, the image signal restoration apparatus may be configured to estimate the first residual signal based on the target shutter-closed frame and parameter information of the sensor, and remove the first residual signal from the measurement signal corresponding to the shutter-closed period frame.

[0026] To estimate the first residual signal, the image signal restoration apparatus may be configured to estimate a residual signal due to a signal generated before shutter closing corresponding to the target shutter closing frame based on the target shutter closing frame and information about a photodiode forming the sensor.

[0027] In order to remove the first residual signal, the image signal restoration apparatus may be configured to remove the first residual signal from the measurement signal corresponding to the shutter-closed period frame by inputting the target shutter-closed frame into the first artificial neural network model.

[0028] In order to generate the target image frame, the image signal restoration apparatus may be configured to generate the target image frame from which the second residual signal is removed by inputting the shutter closing period frame from which the first residual signal is removed into the second artificial neural network model.

[0029] The image signal restoration device may be configured to restore an image frame corresponding to the target shutter-closed frame by inputting a predetermined number of preceding and succeeding image frames based on the target shutter-closed frame into the third artificial neural network model.

[0030] The shutter closed frame constructs an image using the residual charge of the photodiodes forming the sensor without acquiring a signal during the time corresponding to the shutter being closed.

[0031] In order to generate the target image frame, the image signal restoration apparatus may be configured to generate the target image frame from which a residual signal due to a signal after shutter closing corresponding to the target shutter closing frame is removed.

[0032] The sensor may include an organic photodiode (OPD).

[0033] The sensor may include a hybrid image sensor including an OPD and a silicon photodiode.

[0034] The sensor may include an image sensor and an optical structure for improving light sensitivity.

[0035] In order to acquire the image frame, the image signal acquisition device may be configured to: determine the frequency and timing of shutter closing, and acquire the image frame by performing shutter closing according to the determined frequency and timing.

[0036] In order to acquire image frames, the image signal acquisition device may be configured to acquire image frames by periodically performing shutter closing.

[0037] In order to remove the first residual signal, the image signal restoration apparatus may be configured to acquire a residual signal between shutter-closed period frames, and acquire a shutter-closed period frame from which the first residual signal is removed based on the residual signal between the shutter-closed period frames.

[0038] In order to generate a target image frame, the image signal restoration device may be configured to: generate a residual signal between shutter closed period frames from which the first residual signal is removed, and generate a target image frame from which the second residual signal is removed based on the residual signal between shutter closed period frames from which the first residual signal is removed.

[0039] The image signal acquisition device may be configured to acquire a short-term image frame corresponding to an exposure time shorter than that of the image frame, and the image signal restoration device may be configured to generate a high dynamic range (HDR) image based on the short-term image frame.

[0040] In one or more general aspects, a method implemented by a processor having image acquisition includes: generating a recovered signal by the following steps: removing a first residual signal corresponding to a shutter closed frame from a measurement signal corresponding to an image frame acquired after a shutter closed frame, and removing a second residual signal from the measurement signal corresponding to the image frame acquired after the shutter closed frame, the second residual signal corresponding to one or more image frames acquired after the shutter closed frame and before the image frame; the image frame is acquired after the shutter closed frame; and generating a target image frame corresponding to the image frame acquired after the shutter closed frame based on the recovered signal.

[0041] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 An example of an electronic device is shown.

[0043] Figure 2 An example of a shutter-closed photography technique is shown.

[0044] Figure 3A An example of a method of removing a residual signal before the shutter is closed is shown.

[0045] Figure 3B An example of a method of removing a residual signal before the shutter is closed is shown.

[0046] Figure 4 An example of a method of removing a residual signal after a shutter is closed is shown.

[0047] Figure 5 An example of a method of restoring a shutter-closed frame is shown.

[0048] Figure 6 An example of an image acquisition method is shown.

[0049] Fig. 7A and Figure 7B An example of an image sensor is shown.

[0050] Figure 8 An example of a non-periodic shutter-off photography technique is shown.

[0051] Fig. 9 An example of a method of restoring an image using a residual signal is shown.

[0052] Fig.10 An example of a method of acquiring a short exposure image is shown.

[0053] Fig.11 An example of a configuration of an electronic device is shown.

[0054] Throughout the drawings and detailed description, unless otherwise described or provided, the same figure reference numerals will be understood to refer to the same or similar elements, features and structures. The drawings may not be to scale, and the relative sizes, proportions and depictions of the elements in the drawings may be exaggerated for clarity, illustration and convenience. DETAILED DESCRIPTION

[0055] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the method, device and / or system described herein. However, after understanding the disclosure of the application, various changes, modifications and equivalents of the method, device and / or system described herein will be clear. For example, the order in the operation described herein and / or the order of operation are only examples, and are not limited to those orders set forth herein, but except for the order in the operation that must occur in a specific order and / or the order of operation, it can be changed as it will be clear after understanding the disclosure of the application. As another example, except that at least a part of the order in the order of operation and / or the order in the operation must occur in an order (e.g., a specific order), the order in the order of operation and / or the order in the operation can be performed in parallel. In addition, for greater clarity and simplicity, the description of features known after understanding the disclosure of the application can be omitted.

[0056] Although terms such as "first", "second" and "third", or A, B, (a), (b), etc. may be used herein to describe various components, components, regions, layers or parts, these components, components, regions, layers or parts should not be limited by these terms. Each of these terms is not used to define, for example, the essence, order or sequence of the corresponding component, component, region, layer or part, but is only used to distinguish the corresponding component, component, region, layer or part from other components, components, regions, layers or parts. Therefore, without departing from the teaching of the example, the first component, first component, first region, first layer or first part referred to in the example described herein may also be referred to as the second component, second component, second region, second layer or second part.

[0057] Throughout the specification, when a component or element is described as being "on", "connected to", "bonded to" or "joined to" another component, element or layer, it may be directly (e.g., in contact with), "on", "directly connected to", "directly joined to" or "directly joined to" another component, element or layer, or one or more other components, elements or layers may reasonably be present in between. When a component or element is described as being "directly on", "directly connected to", "directly joined to" or "directly joined to" another component, element or layer, there may not be other components, elements or layers in between. Similarly, expressions such as "between" and "immediately between" and "adjacent to" and "immediately adjacent to" may also be interpreted as described above.

[0058] The terms used herein are only used to describe various examples and are not used to limit the present disclosure. Unless the context clearly indicates otherwise, singular terms are also intended to include plural forms. As non-limiting examples, the terms "comprise", "include" and "have" illustrate the existence of stated features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the existence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof, or the existence of alternative features, quantities, operations, components, elements and / or combinations thereof that replace stated features, quantities, operations, components, elements and / or combinations thereof. Additionally, although an embodiment may set forth such terms "comprise", "include" and "have" that specify the existence of stated features, quantities, operations, components, elements and / or combinations thereof, there may be: other embodiments in which one or more of the stated features, quantities, operations, components, elements and / or combinations thereof are not present.

[0059] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs and based on the understanding of the disclosure of the present application. Unless explicitly defined as such herein, it will also be understood that terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the disclosure of the relevant field and the present application, and should not be interpreted in an idealized or overly formal manner.

[0060] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more. Unless the corresponding description and embodiments require such a list (e.g., "at least one of A, B, and C") to be interpreted as having a connecting meaning, the phrases "at least one of A, B, and C", "at least one of A, B, or C", etc. are intended to have a separate meaning, and these phrases "at least one of A, B, and C", "at least one of A, B, or C", etc. also include examples in which there may be one or more of each of A, B, and / or C (e.g., any combination of one or more of each of A, B, and C).

[0061] The features described herein may be embodied in different forms and should not be construed as being limited to the examples described herein. On the contrary, the examples described herein have been provided to illustrate only some of the many possible ways in which it will be clear to implement the methods, devices and / or systems described herein after understanding the disclosure of the present application. The use of the term "may" with respect to an example or embodiment herein (e.g., what an example or embodiment may include or implement) indicates that there is at least one example or embodiment that includes or implements such a feature, and all examples are not limited thereto. The use of the term "example" or "embodiment" herein has the same meaning (e.g., the phrase "in one example" has the same meaning as "in one embodiment", and "one or more examples" has the same meaning as "in one or more embodiments").

[0062] Examples may be implemented as various types of products (such as, for example, personal computers (PCs), laptop computers, tablet computers, smart phones, televisions (TVs), smart home appliances, smart vehicles, autonomous service terminals, and wearable devices). Hereinafter, examples will be described in detail with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals are used for the same elements.

[0063] Figure 1 An example of an electronic device is shown.

[0064] Reference Figure 1 , the electronic device 100 according to one or more embodiments may include an image signal acquisition device 110 and an image signal restoration device 150. However, not all of the components shown are required. The electronic device 100 may be implemented by more components than the components shown, and the electronic device 100 may be implemented by fewer components.

[0065] The image signal acquisition device 110 according to one or more embodiments may generate a plurality of image frames by continuous shooting. The image signal acquisition device 110 may include various optical image acquisition devices capable of high-speed continuous shooting (e.g., ultra-high-speed cameras, medical imaging devices, semiconductor measurement devices, vehicle cameras, etc.).

[0066] The image signal acquisition device 110 may include a sensor (e.g., Fig.11 Sensor 1105). For example, the image signal acquisition device 110 may include an image sensor that converts an optical signal into an electrical signal. The image sensor may include a lens and a photodiode. The image sensor may include a contact image sensor (CIS) and a charge coupled device (CCD) image sensor. However, the sensor is not limited to the aforementioned examples, and may include various types of image sensors (such as a laser radar (LiDAR) sensor, a quantum dot (QD) image sensor, a one-dimensional (1D), two-dimensional (2D) and three-dimensional (3D) image sensor, etc.). For example, the sensor may also include a sensor for acquiring a signal in a form other than an image.

[0067] Light incident to the imaging sensor through the lens may be converted into an electrical signal by a photodiode, and the image signal acquisition device 110 may construct or generate an image by measuring the electrical signal. In the process of converting photons into electrical signals by a photodiode, the converted charges may be accumulated in the pixels during the exposure time.

[0068] When continuously capturing images at high speed, it may be difficult for a typical electronic device to ensure a signal-to-noise ratio because the image signal is small due to a short exposure time for image acquisition. In contrast to a typical electronic device, the electronic device 100 according to one or more embodiments can acquire (e.g., generate) a high-quality image with a high signal-to-noise ratio even in high-speed shooting using the image signal acquisition device 110 and the image signal restoration device 150.

[0069] For example, even in high-speed shooting, the image signal acquisition device 110 according to one or more embodiments may use an image sensor including a photodiode with high sensitivity to obtain a high-quality image with a high signal-to-noise ratio. For example, the image signal acquisition device 110 may use an image sensor including an organic photodiode (OPD). For example, the image signal acquisition device 110 may increase the sensitivity by using an OPD, thereby obtaining a high-quality image with a high signal-to-noise ratio in high-speed shooting (such as 960 frames per second (fps) slow-motion video). As described in detail in the examples below, the OPD may replace part or all of the photodiodes in the CIS, and thereby increase the sensitivity while maintaining the same CIS structure.

[0070] When OPD is used to increase sensitivity during high-speed shooting, high-quality images with a high signal-to-noise ratio can be acquired due to the high sensitivity for the same acquisition time, and when the same signal-to-noise ratio signal is acquired, the acquisition time can be reduced and therefore images can be acquired at a high frame rate.

[0071] However, when the OPD has a low charge reduction of the photodiode compared to the existing silicon photodiode (SiPD), the residual signal may remain in the subsequent frames. In typical electronic devices, the residual signal caused by the slow light response may cause image degradation (such as motion blur, halo, and color distortion due to signal saturation, etc.).

[0072] Compared to a typical electronic device, the electronic device 100 according to one or more embodiments may use a shutter-closed shooting technique that reduces image degradation due to residual signals and easily estimates and removes residual signals. The image signal acquisition device 110 may construct an image using a signal retained in a photodiode without acquiring a signal at a corresponding time by closing the shutter during continuous shooting.

[0073] The shutter closing of one or more embodiments may not acquire a signal at a corresponding time, and thus may prevent a residual signal due to the corresponding signal from remaining in a subsequent image frame, thereby reducing degradation caused by the residual signal.

[0074] Thereafter, the image signal restoration device 150 may acquire the remaining signal information generated before the shutter is closed from the corresponding frame, and estimate and compensate for the image degradation that occurs in the image after the shutter is closed due to the previous signal. The degraded image may be restored by specializing the image acquired by the image signal acquisition device 110 using an artificial neural network-based model via the image signal restoration device 150, and the image signal acquisition device 110 is configured with a high-sensitivity photodiode and a shutter-closed high-speed shooting technology. Figure 5 An example of a detailed operation of the image signal restoration device 150 according to one or more embodiments is described. In a non-limiting example, the electronic device 100 may be or may include Fig.11 The electronic device 1100, the image signal acquisition device 110 may include Fig.11 The sensor 1105, and the image signal restoration device 150 may include Fig.11 processor 1101 and memory 1103.

[0075] Figure 2 An example of a shutter-closed photography technique is shown.

[0076] Reference Figure 1 The description provided can also be applied to Figure 2 . Reference Figure 2According to one or more embodiments, the shutter-closed shooting technique may be a shooting technique for performing shutter closing during continuous shooting, wherein the shutter closing may be an operation of not acquiring a signal of a corresponding time. Therefore, during the shutter closing period, the signal of the corresponding time may not be acquired, and an image may be constructed using a signal retained in a photodiode (e.g., a signal generated before the corresponding time). An image generated using a signal retained in a photodiode during the shutter closing period may be referred to as a shutter-closed frame or a residual signal image.

[0077] The electronic device 100 according to one or more embodiments may perform shutter closing at a predetermined time and frequency during continuous shooting. For example, the electronic device 100 may perform shutter closing periodically for every N frames. Alternatively, the electronic device 100 may perform shutter closing aperiodically.

[0078] The electronic device 100 of one or more embodiments may minimize the residual signal remaining in subsequent frames due to the slow light response decay of a high-sensitivity photodiode (eg, OPD) through a shutter-off shooting technique.

[0079] The image signal restoration device 150 can use the shutter closing frame to estimate the residual signal before the shutter is closed, and remove the residual signal from the image frame. Figure 3A and Figure 3B An example of a method of removing a residual signal before the shutter is closed is described.

[0080] Figure 3A An example of a method of removing a residual signal before the shutter is closed is shown.

[0081] The image signal restoration device 150 may estimate a residual signal remaining in the image frame after the shutter is closed using a shutter-closed frame acquired during the shutter-closed period and parameters of the sensor, and remove the estimated residual signal from the image frame.

[0082] Reference Figure 3A In one example, the first shutter closes at t=t s , and the second shutter closes at t = t s +(N+1)Δt, where t represents time, Δt represents a time interval between two adjacent frames, and N is a natural number. The first shutter-closed frame 310 may be generated when the first shutter is closed, and the second shutter-closed frame 330 may be generated when the second shutter is closed.

[0083] N image frames may be generated between the first shutter-closed frame 310 and the second shutter-closed frame 330, and the image frames between the shutter-closed frames may be referred to as shutter-closed period frames 320. The image signal restoration device 150 may estimate a residual signal due to a signal before the first shutter is closed using the first shutter-closed frame 310, and remove the residual signal due to the signal before the first shutter is closed from the shutter-closed period frame 320. Therefore, the first shutter-closed frame 310, which is a shutter-closed frame for removing the residual signal in the shutter-closed period frame 320, may be referred to as a target shutter-closed frame of the shutter-closed period frame 320.

[0084] In the shutter closing period frame 320, with time t>t s The corresponding image frame may be referred to as a target image frame. The image signal restoration device 150 may remove a residual signal due to a slow light response of a photodiode from the target image frame.

[0085] When the ideal signal of the target image frame is represented by x t , the measurement signal measured or generated by the sensor is denoted by y t , the residual signal due to the signal before the shutter closes is expressed as And the remaining signal due to the signal after the shutter is closed is expressed as When , the measurement signal can be expressed by, for example, the following equation 1.

[0086] Equation 1:

[0087]

[0088] In the following, the residual signal (eg, ) may be referred to as a first residual signal, and a residual signal due to a signal after the shutter is closed (eg, ) can be called the second residual signal.

[0089] When an image is constructed using residual charges of a photodiode without acquiring a signal at a corresponding time in the first shutter-closed frame 310 , the first shutter-closed frame 310 may be a remaining signal image before the shutter is closed.

[0090] The image signal recovery device 150 may use the first shutter closed frame 310 and the photoresponse information 340 of the photodiode to estimate the influence of the measured afterimage on the image after the shutter is closed. Thereafter, the image signal recovery device 150 may remove the residual signal of the signal before the shutter is closed that affects the image after the shutter is closed. For example, the image signal recovery device 150 may use the first shutter closed frame 310 and the photoresponse information 340 of the photodiode to estimate and remove the residual signal of the signal before the shutter is closed from the measurement signal corresponding to the shutter closed period frame 320. The first shutter-closed frame 310 and the light response information 340 of the photodiode are residual signal information acquired during the shutter-closed period, and therefore, the image signal restoration device 150 may restore the signal y′ represented by the following equation 2, for example, through the process t For example, the image signal restoration device 150 may estimate the image signal by performing an operation (eg, a multiplication operation) between the signal corresponding to the first shutter-closed frame 310 and the light response information 340 of the photodiode.

[0091] Equation 2:

[0092]

[0093] Figure 3B An example of a method of removing a residual signal before the shutter is closed is shown.

[0094] Reference Figure 3B The image signal restoration device 150 may use an artificial neural network to estimate and remove the first residual signal. For example, the image signal restoration device 150 may remove the first residual signal from the measurement signal corresponding to the shutter closing period frame by inputting the target shutter closing frame 310 into the first artificial neural network model 350.

[0095] The first artificial neural network model 350 according to one or more embodiments may be trained to remove the first residual signal from the measurement signal corresponding to the shutter closed period frame using the shutter closed frame. The training of the artificial neural network model may be performed to determine the parameters that minimize the loss function. The loss function may be used as a representation of determining the optimal parameters in the process of training the artificial neural network model. The loss function according to one or more embodiments may be defined as the difference between the ground truth data output from the first artificial neural network model 350 (the shutter closed period frame from which the first residual signal is actually removed) and the shutter closed period frame from which the first residual signal is removed, and the parameters that minimize the loss function may be determined. The method of estimating the first residual signal is not limited to the above method, and various methods may be adopted.

[0096] Figure 4 An example of a method of removing a residual signal after a shutter is closed is shown.

[0097] Reference Figure 4 Although the first residual signal is removed from the shutter closing period frame 320 , the residual signal may be generated again during the image acquisition process resumed after the shutter is closed.

[0098] When signal acquisition is resumed after the first shutter is closed, a residual signal (e.g., a second residual signal) caused by signal acquisition after the first shutter is closed may remain in the target image frame. For example, when the target image frame is the fourth image frame after the first shutter is closed, the target image frame may include a residual signal caused by acquisition of the first image frame to the third image frame after the first shutter is closed.

[0099] The image signal restoration device 150 may obtain a target image frame from which the second residual signal is removed based on the artificial neural network model. For example, the image signal restoration device 150 may obtain a target image frame from which the second residual signal is removed by inputting a shutter closing period frame from which the first residual signal is removed into the second artificial neural network model 410. The image signal restoration device 150 may, for example, obtain a signal y″ from which the second residual signal is removed through this process as expressed in the following equation 3 t Alternatively, the image signal restoration device 150 may remove a second residual signal corresponding to one or more image frames acquired after the first shutter-closed frame 310 and before the target image frame from the measurement signal corresponding to the target image frame acquired after the first shutter-closed frame 310 .

[0100] Equation 3:

[0101] y″ t =x t

[0102] The second artificial neural network model 410 according to one or more embodiments may be trained to output a target image frame from which the second residual signal is removed using a shutter closed period frame. A loss function according to one or more embodiments may be defined as a difference between ground truth data output from the second artificial neural network model 410 (a target image frame from which the second residual signal is actually removed) and the target image frame from which the second residual signal is removed, and a parameter that minimizes the loss function may be determined.

[0103] Since the residual signal information changes based on the first shutter closing frame 310, the electronic device 100 of one or more embodiments may consider the shutter closing period frame 320 without considering the entire image series for image restoration, which is efficient. For example, when the image signal restoration device 150 compensates for the first residual signal, the image signal restoration device 150 of one or more embodiments may restore the image using only the shutter closing period frame 320 instead of the entire image series to compensate for the second residual signal, thereby reducing the computing resources used for image restoration. However, the method of removing the residual signal after the shutter is closed is not limited to the above method. For example, the image signal restoration device 150 may estimate and remove the second residual signal by a signal processing method.

[0104] Figure 5 An example of a method of restoring a shutter-closed frame is shown.

[0105] Reference Figure 5 , the image signal restoration device 150 may have removed the residual signal in the acquired image through the residual signal removal method, but may not have acquired information about the shutter-closed frame.

[0106] Therefore, the image signal restoration device 150 can restore the shutter closed frame using the adjacent frame information. For example, the image signal restoration device 150 can restore the image frame corresponding to the shutter closed frame by inputting a predetermined number of front and back image frames based on the shutter closed frame into the third artificial neural network model 510. In one example, the front and back image frames include an image frame before the shutter closed frame and / or an image frame after the shutter closed frame.

[0107] For example, the image signal restoration device 150 may restore the image frame 540 corresponding to the first shutter closed frame 310 by inputting two previous image frames 520 and two subsequent image frames 530 of the first shutter closed frame 310 into the third artificial neural network model 510. However, the number of image frames used to restore the shutter closed frame is not limited to the above example.

[0108] At this time, the third artificial neural network model 510 for restoring the shutter-closed frame may be integrated with the second artificial neural network model 410, or sequential processing may be performed.

[0109] Figure 6 An example of an image acquisition method is shown.

[0110] For the convenience of description, operations 610 to 650 are described as follows: Figure 1 However, operations 610 to 650 may also be performed by any suitable electronic device and in any suitable system.

[0111] also, Figure 6 The operations may be performed in the order and manner shown. However, without departing from the spirit and scope of the illustrated examples, the order of one or more of the operations may be changed, one or more of the operations may be omitted, and the operations may be performed in parallel or simultaneously. Figure 6 Two or more of the operations shown in .

[0112] In operation 610, the electronic device 100 according to one or more embodiments may acquire an image frame including a shutter-closed frame corresponding to shutter closing via a sensor by performing shutter closing during continuous shooting. The shutter-closed frame may construct an image using residual charges of a photodiode forming the sensor without acquiring a signal during a time corresponding to shutter closing.

[0113] In operation 620 , the electronic device 100 according to one or more embodiments may acquire a measurement signal corresponding to a shutter-closed period frame.

[0114] In operation 630 , the electronic device 100 according to one or more embodiments may remove a first residual signal corresponding to a target shutter-closed frame from a measurement signal corresponding to a shutter-closed period frame based on a target shutter-closed frame corresponding to a target image frame.

[0115] The image signal restoration device 150 may estimate a first residual signal corresponding to a target shutter closed frame based on a target shutter closed frame corresponding to a target image frame and parameter information of the sensor. The image signal restoration device 150 may estimate a residual signal due to a signal before shutter closing corresponding to the target shutter closed frame based on the target shutter closed frame and information about a photodiode forming the sensor.

[0116] The electronic device 100 may remove the first residual signal from the measurement signal corresponding to the shutter closing period frame including the image frame between the target image frame and the target shutter closing frame. The image signal recovery device 150 may obtain the target image frame from which the residual signal caused by the signal after the shutter closing corresponding to the target shutter closing frame is removed.

[0117] Alternatively or additionally, the image signal restoration device 150 may remove the first residual signal from the measurement signal corresponding to the shutter-closed period frame by inputting the target shutter-closed frame into the first artificial neural network model.

[0118] In operation 640 , the electronic device 100 according to one or more embodiments may acquire a target image frame from which the second residual signal is removed, based on at least one of the shutter-closed period frames from which the first residual signal is removed.

[0119] The image signal restoration device 150 may obtain a target image frame from which the second residual signal is removed by inputting a shutter-closed period frame from which the first residual signal is removed into a second artificial neural network model.

[0120] In operation 650, the electronic device 100 according to one or more embodiments may restore an image frame corresponding to a target shutter closed frame. The image signal restoration device 150 may obtain a target image frame from which the second residual signal is removed by inputting a predetermined number of preceding and succeeding image frames based on the target shutter closed frame into a third artificial neural network model.

[0121] Fig. 7A and Figure 7B An example of an image sensor is shown.

[0122] Reference Fig. 7A, in addition to the image sensor 710 (eg, as described above with reference to Figures 1 to 6 In addition to any image sensor and / or sensor described herein, the sensor 700 according to one or more embodiments may further include an optical structure 720 for improving sensitivity.

[0123] The image sensor 710 may include an OPD or may be a hybrid image sensor including an OPD and a SiPD.

[0124] By introducing the optical structure 720 for improving sensitivity into the image sensor 710, the sensor 700 can reduce the exposure time without losing the signal-to-noise ratio and ensure the reset time.

[0125] Figure 7B Graph 730 in FIG. 7 shows an exposure time for acquiring a signal during continuous shooting and a reset time that is not used for acquiring a signal. Referring to graph 740, a first graph 741 is a graph showing a reduction in a photodiode residual charge during a reset time and an exposure time when only the image sensor 710 is used, and a second graph 742 is a graph showing a reduction in a photodiode residual charge during a reset time and an exposure time when the sensor 700 having an optical structure 720 for improving sensitivity is used, wherein the horizontal axis represents time and the vertical axis represents normalized charge.

[0126] Comparing the first graph 741 and the second graph 742 , it can be learned that since the residual charge of the photodiode is reduced during the reset time additionally ensured when the sensor 700 having the optical structure 720 for improving sensitivity is used, the residual signal can be reduced.

[0127] Figure 8 An example of a non-periodic shutter-off photography technique is shown.

[0128] As described above, the electronic device 100 may perform shutter closing at a predetermined time and frequency during continuous photographing. For example, the electronic device 100 may perform shutter closing periodically for every N frames.

[0129] Optionally, refer to Figure 8 , the electronic device 100 may perform shutter closing non-periodically, and may adjust the effect of shutter closing and the resources used for image acquisition and recovery by variably adjusting the frequency and timing of shutter closing. In other words, the electronic device 100 may acquire image frames by performing shutter closing according to the determined frequency and timing.

[0130] For example, the first shutter closing period frame 810 may include N i-1 frames, the second shutter closing period frame 820 may include N i frames, the third shutter closing period frame 830 may include Ni+1 frames, and the fourth shutter closing period frame 840 may include N i+2 frames, where N i-1 To N i+2 is a positive integer.

[0131] In addition, the electronic device 100 may optimize the image signal restoration device 150 to be specifically used for a non-periodic image acquisition method.

[0132] Fig. 9 An example of a method of restoring an image using a residual signal is shown.

[0133] When a plurality of image frames are acquired by high-speed continuous shooting, the difference between adjacent frames may not be significant. Therefore, the image signal restoration device 150 according to one or more embodiments may restore the image using the residual signal of the image frame instead of using the acquired signal of the image frame as it is. The residual signal of the i-th image frame may be the difference between the i-th image frame and the (i-1)-th image frame.

[0134] Reference Fig. 9 The image signal restoration device 150 can obtain a residual signal of a shutter closing period frame from which the first residual signal is removed, and obtain a target image frame from which the second residual signal is removed by inputting the residual signal into the second artificial neural network model 910.

[0135] For example, the residual signal y of the target image frame after removing the first residual signal tr1 It can be expressed by, for example, the following equation 4, where y′ t represents the measurement signal corresponding to the target image frame from which the first residual signal is removed, and y′ t-Δt The measured signal corresponding to the image frame before the target image frame from which the first residual signal is removed is represented.

[0136] Equation 4:

[0137] y tr1 =y′ t -y′ t-Δt

[0138] Due to y′ t Entered into Figure 4 In the second artificial neural network model 410, y′ t Small y tr1 Entered into Fig. 9 In the second artificial neural network model 910, therefore, when the residual signal is used to restore the image, less signal processing resources (such as memory) can be used.

[0139] Similarly, the image signal restoration device 150 may acquire a residual signal between shutter-closed period frames, and acquire a shutter-closed period frame from which the first residual signal is removed based on the residual signal between the shutter-closed period frames.

[0140] For example, the residual signal y of the target image frame after removing the first residual signal tr2 It can be expressed by, for example, the following equation 5, where y″ t represents the measurement signal corresponding to the target image frame from which the first residual signal is removed, and y″ t-Δt The measured signal corresponding to the image frame before the target image frame from which the first residual signal is removed is represented.

[0141] Equation 5:

[0142] y tr2 =y″ t -y″ t-Δt The image signal restoration device 150 may obtain a target image frame from which the second residual signal is removed by inputting a residual signal between shutter-closed period frames into a second artificial neural network model.

[0143] Fig.10 An example of a method of acquiring a short exposure image is shown.

[0144] The image signal acquisition device 110 according to one or more embodiments may acquire an image using a short exposure time instead of shutter closing.

[0145] Reference Fig.10 , the image signal acquisition device 110 may use a short exposure time instead of shutter closing to periodically or non-periodically acquire images 1010. Short exposure may be an exposure having an exposure time shorter than the exposure time 1020 used to acquire a signal in normal continuous shooting.

[0146] The image signal restoration device 150 may generate a high dynamic range (HDR) image based on a short-term image frame (eg, an image 1010 with a short exposure time). In this case, the image signal restoration device 150 may also use the image 1010 with a short exposure time to remove the first residual signal and the second residual signal.

[0147] Fig.11 An example of a configuration of an electronic device is shown.

[0148] Reference Fig.11 , the electronic device 1100 may include a processor 1101 (eg, one or more processors), a memory 1103 (eg, one or more memories), and a sensor 1105 (eg, one or more sensors).

[0149] The processor 1101 according to one or more embodiments may execute the above-mentioned Figures 1 to 10 The processor 1101 may acquire an image frame including a shutter-closed frame corresponding to shutter closing by performing shutter closing during continuous shooting, acquire a measurement signal corresponding to a target image frame, remove a first residual signal corresponding to the target shutter-closed frame from the measurement signal corresponding to the shutter-closed period frame including an image frame between the target image frame and the target shutter-closed frame based on the target shutter-closed frame corresponding to the target image frame, and acquire a target image frame from which a second residual signal is removed based on at least one of the shutter-closed period frames from which the first residual signal is removed.

[0150] The memory 1103 according to one or more embodiments may be a volatile memory or a non-volatile memory, and the memory 1103 may store data to be used for performing the operation recovery (e.g., parameters of the trained first artificial neural network model to the third artificial neural network model). For example, the memory 1103 may include a non-transitory computer-readable storage medium storing instructions, which, when executed by the processor 1101, configures the processor 1101 to perform the operation described herein with reference to Figures 1 to 10 Any one, any combination, or all of the operations and / or methods described.

[0151] The sensor 1105 according to one or more embodiments may include an image sensor. The sensor 1105 may include an image sensor including an OPD. The sensor 1105 may include a hybrid image sensor including an OPD and a SiPD. The sensor 1105 may include an image sensor and an optical structure for improving sensitivity.

[0152] The electronic device 1100 according to one or more embodiments may further include other components not shown in the drawings. For example, the electronic device 1100 may further include a communication module and an input / output interface, the input / output interface including an input device and an output device as a device for interfacing with the communication module. In addition, for example, the electronic device 1100 may further include other components (such as a transceiver, various sensors, and a database).

[0153] Described herein include Figures 1 to 11The electronic device, image signal acquisition device, image signal recovery device, sensor, image sensor, optical structure, processor, memory, electronic device 100, image signal acquisition device 110, image signal recovery device 150, sensor 700, image sensor 710, optical structure 720, electronic device 1100, processor 1101, memory 1103 and sensor 1105 described in the present application are implemented or represent hardware components by hardware components. As described above, or in addition to the above description, examples of hardware components that can be used to perform the operations described in this application appropriately include: controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a limited manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer can execute instructions or software (such as operating systems (OS) and one or more software applications running on OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or a processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may implement a single hardware component or two or more hardware components.As described above, or in addition to the description above, the example hardware components may have any one or more of different processing configurations, examples of which include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0154] Figures 1 to 11 shown in and for Figures 1 to 11 The methods discussed for performing the operations described in this application are performed by computing hardware (e.g., by one or more processors or computers), which is implemented as instructions (e.g., computer or processor / processing device readable instructions) or software as described above to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be performed by a single processor or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation or two or more operations.

[0155] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform methods as described above may be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods as described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by one or more processors or computers. In another example, the instructions or software include high-level code executed by one or more processors or computers using an interpreter. Instructions or software may be written in any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions used herein, wherein the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions used herein disclose algorithms for performing operations performed by the hardware components and methods as described above.

[0156] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media, and therefore, are not signals themselves. As described above, or in addition to the above description, examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R One or more of LTH, BD-RE, Blu-ray or optical disk storage devices, hard disk drives (HDD), solid-state drives (SSD), card-type memories (such as multimedia cards or micro cards (e.g., secure digital (SD) or extreme digital (XD))), magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks, and / or any other devices, any other device is configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed on a networked computer system so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0157] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered in a descriptive sense only, not for the purpose of limitation. The description of the features or aspects in each example will be considered applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents, suitable results may be achieved.

[0158] Therefore, the scope of the disclosure includes the claims and their equivalents in addition to the disclosure above and all the accompanying drawings (that is, all changes within the scope of the claims and their equivalents should be construed as being included in the disclosure).

Claims

1. A method for image acquisition, the method comprising: acquiring, via the sensor, image frames including a shutter-closed frame corresponding to shutter closing by performing shutter closing during continuous photographing; Acquire a measurement signal corresponding to a shutter-closed period frame, where the shutter-closed period frame includes an image frame between a target image frame and a target shutter-closed frame; Based on a target shutter-closed frame corresponding to a target image frame, removing a first residual signal corresponding to the target shutter-closed frame from a measurement signal corresponding to a shutter-closed period frame to obtain a shutter-closed period frame from which the first residual signal is removed; as well as The target image frame from which the second residual signal is removed is generated based on one or more of the shutter-closed period frames from which the first residual signal is removed.

2. The method according to claim 1, wherein: The step of removing the first residual signal comprises: estimating a first residual signal based on the target shutter-closed frame and parameter information of the sensor; and The estimated first residual signal is removed from the measurement signal corresponding to the shutter-closed period frame.

3. The method according to claim 2, wherein: The estimating of the first residual signal includes estimating a residual signal due to a signal generated before shutter closing corresponding to the target shutter closing frame based on the target shutter closing frame and information about a photodiode forming the sensor.

4. The method according to claim 1, wherein: The step of removing the first residual signal includes removing the first residual signal from the measurement signal corresponding to the shutter-closed period frame by inputting the target shutter-closed frame into the first artificial neural network model.

5. The method according to claim 1, wherein: The step of generating a target image frame from which the second residual signal is removed includes generating the target image frame from which the second residual signal is removed by inputting a shutter closing period frame from which the first residual signal is removed into a second artificial neural network model.

6. The method according to claim 1, further comprising: An image frame corresponding to the target shutter-closed frame is restored by inputting a predetermined number of preceding and succeeding image frames based on the target shutter-closed frame into a third artificial neural network model.

7. The method according to claim 1, wherein: The shutter closed frame constructs an image using the residual charge of the photodiodes forming the sensor without acquiring a signal during the time corresponding to the shutter being closed.

8. The method according to claim 1, wherein: The step of generating the target image frame from which the second residual signal is removed includes generating the target image frame from which a residual signal due to a signal after shutter closing corresponding to the target shutter closing frame is removed.

9. The method according to claim 1, wherein: The sensor includes an organic photodiode.

10. The method according to claim 1, wherein: The sensors include hybrid image sensors that include organic photodiodes and silicon photodiodes.

11. The method according to claim 1, wherein: The sensor includes an image sensor and an optical structure for improving sensitivity.

12. The method according to claim 1, wherein: The steps of acquiring an image frame include: Determine the frequency and timing of shutter closing; and Image frames are acquired by performing shutter closing according to the determined frequency and timing.

13. The method according to claim 1, wherein: The step of acquiring the image frame includes acquiring the image frame by periodically performing shutter closing.

14. The method according to claim 1, wherein: The step of removing the first residual signal comprises: Acquire a residual signal between frames during a shutter closing period; and A shutter-closed-period frame from which the first residual signal is removed is acquired based on the residual signal between the shutter-closed-period frames.

15. The method according to claim 14, wherein: The step of generating a target image frame from which the second residual signal is removed comprises: generating a residual signal between shutter-closed period frames from which the first residual signal is removed; and Based on the residual signal between the shutter-closed period frames from which the first residual signal is removed, a target image frame from which the second residual signal is removed is generated.

16. The method according to claim 1, further comprising: acquiring a short-term image frame corresponding to an exposure time shorter than an exposure time of the image frame; as well as Generate high dynamic range images based on short-term image frames.

17. A non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, configure the one or more processors to perform the method according to any one of claims 1 to 16.

18. An electronic device comprising: An image signal acquisition device, comprising: a sensor configured to: acquire image frames including shutter-closed frames corresponding to shutter closing by performing shutter closing during continuous shooting, and acquire measurement signals corresponding to shutter-closed period frames, the shutter-closed period frames including image frames between a target image frame and a target shutter-closed frame; and The image signal recovery device is configured to: based on a target shutter closing frame corresponding to the target image frame, remove the first residual signal corresponding to the target shutter closing frame from the measurement signal corresponding to the shutter closing period frame, and generate a target image frame from which the second residual signal is removed based on one or more of the shutter closing period frames from which the first residual signal is removed, so as to obtain a shutter closing period frame from which the first residual signal is removed.

19. The electronic device according to claim 18, wherein: In order to remove the first residual signal, the image signal restoration device is further configured as follows: estimating a first residual signal based on the target shutter-closed frame and parameter information of the sensor, and The estimated first residual signal is removed from the measurement signal corresponding to the shutter-closed period frame.

20. The electronic device according to claim 19, wherein: In order to estimate the first residual signal, the image signal restoration device is further configured to: A residual signal due to a signal generated before shutter closing corresponding to the target shutter closing frame is estimated based on the target shutter closing frame and information on photodiodes forming the sensor.

21. The electronic device according to claim 18, wherein: In order to remove the first residual signal, the image signal restoration device is further configured as follows: A first residual signal is removed from a measurement signal corresponding to a shutter-closed period frame by inputting a target shutter-closed frame into a first artificial neural network model.

22. The electronic device according to claim 18, wherein: In order to generate a target image frame, the image signal restoration device is further configured to: A target image frame from which the second residual signal is removed is generated by inputting a shutter closing period frame from which the first residual signal is removed into a second artificial neural network model.

23. The electronic device according to claim 18, wherein: The image signal recovery device is further configured as follows: An image frame corresponding to the target shutter-closed frame is restored by inputting a predetermined number of preceding and succeeding image frames based on the target shutter-closed frame into a third artificial neural network model.

24. The electronic device according to claim 18, wherein: The shutter closed frame constructs an image using the residual charge of the photodiodes forming the sensor without acquiring a signal during the time corresponding to the shutter being closed.

25. The electronic device according to claim 18, wherein: In order to generate a target image frame, the image signal restoration device is further configured to: A target image frame is generated from which a residual signal due to a signal after shutter closing corresponding to a target shutter closing frame is removed.

26. The electronic device according to claim 18, wherein: The sensor includes an organic photodiode.

27. The electronic device according to claim 18, wherein: The sensors include hybrid image sensors that include organic photodiodes and silicon photodiodes.

28. The electronic device according to claim 18, wherein: The sensor includes an image sensor and an optical structure for improving sensitivity.

29. The electronic device according to claim 18, wherein: In order to acquire image frames, the image signal acquisition device is further configured as follows: Determines the frequency and timing of shutter closing, and Image frames are acquired by performing shutter closing according to the determined frequency and timing.

30. The electronic device according to claim 18, wherein: In order to acquire image frames, the image signal acquisition device is further configured as follows: Image frames are acquired by periodically performing shutter closing.

31. The electronic device according to claim 18, wherein: In order to remove the first residual signal, the image signal restoration device is further configured as follows: Acquire the residual signal between frames during the shutter closing period, and A shutter-closed-period frame from which the first residual signal is removed is acquired based on the residual signal between the shutter-closed-period frames.

32. The electronic device according to claim 18, wherein: In order to generate a target image frame, the image signal restoration device is further configured to: generating a residual signal between shutter closing period frames from which the first residual signal is removed, and Based on the residual signal between the shutter-closed period frames from which the first residual signal is removed, a target image frame from which the second residual signal is removed is generated.

33. The electronic device according to claim 18, wherein: The image signal acquisition device is further configured to: acquire a short-term image frame corresponding to an exposure time shorter than an exposure time of the image frame, and The image signal restoration apparatus is further configured to generate a high dynamic range image based on the short-term image frame.

34. A method for image acquisition, the method comprising: The recovered signal is generated by the following steps: removing a first residual signal corresponding to the shutter-closed frame from a measurement signal corresponding to at least one image frame acquired after the shutter-closed frame, and removing a second residual signal from the measurement signal corresponding to the at least one image frame, the second residual signal corresponding to one or more image frames acquired after the shutter-closed frame and before the at least one image frame; as well as A target image frame is generated based on the restored signal.

Images