Focal length determination method and electronic device

Abstract

The invention provides a focal length determination method and an electronic device. The electronic device includes: a processor; a camera including a lens; and a memory configured to store one or more instructions executable by the processor. The processor is configured to: receive a plurality of raw images using the lens in response to the one or more instructions being executed; generating a composite image based on the plurality of original images; and determining a focal length of the lens by inputting one or more of the plurality of original images and the composite image to an auto-focus (AF) model.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2023-0110115 filed on August 22, 2023, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field

[0002] The following description relates to an electronic device having a focal length determination method. Background Art

[0003] The camera can automatically perform focusing operations using Contrast Detection Auto Focus (CDAF) and Phase Detection Auto Focus (PDAF).

[0004] CDAF defines a metric that calculates the contrast in an image and determines that the camera is focused at the position of the lens where the contrast is greatest.

[0005] PDAF (Phase Detection Method) splits the light incoming through the lens into two branches, compares them, and determines that the camera is focused when the phases of the two beams of light are the same.

[0006] The above information is presented as background information only to assist with understanding the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above may be applicable as prior art with respect to the disclosure. Summary of the invention

[0007] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0008] In one general aspect, an electronic device includes: a processor; a camera including a lens; and a memory configured to store one or more instructions executable by the processor, wherein the processor is configured to: receive a plurality of raw images using the lens in response to the one or more instructions being executed; generate a composite image based on the plurality of raw images; and determine a focal length of the lens by inputting one or more of the plurality of raw images and the composite image into an auto focus (AF) model.

[0009] The processor may also be configured to: calculate an error between an original image of a reference frame selected from the plurality of original images and original images of remaining frames; and generate a composite image based on an original image having an error within a set threshold and an original image of the reference frame.

[0010] The processor may be configured to calculate an error based on pixel values ​​of the original image of the reference frame and pixel values ​​of the original images of the remaining frames.

[0011] The processor may be further configured to: correct the plurality of original images by frame alignment; and generate a composite image based on the corrected plurality of original images.

[0012] The AF model may be a convolutional neural network (CNN) including a plurality of layers, wherein the CNN may be configured to receive one or more of the plurality of original images and a composite image generated from the plurality of original images, and output a focal length of the composite image.

[0013] The AF model may be configured to determine the focal length by detecting the contrast of the composite image, or to determine the focal length based on the phase difference of the composite image.

[0014] In one general aspect, an electronic device includes: a processor; a camera including a lens; and a memory configured to store one or more instructions executable by the processor, wherein the processor is configured to: receive a plurality of raw images using the lens in response to the one or more instructions being executed; generate a composite image based on the plurality of raw images; and determine a focal length of the lens by inputting the composite image into an autofocus (AF) model.

[0015] The processor may also be configured to: calculate an error between an original image of a reference frame selected from the plurality of original images and original images of remaining frames; and generate a composite image based on an original image having an error within a set threshold and an original image of the reference frame.

[0016] The processor may be further configured to calculate an error based on pixel values ​​of the original image of the reference frame and pixel values ​​of the original images of the remaining frames.

[0017] The processor may be further configured to: correct the plurality of original images by frame alignment; and generate a composite image based on the corrected plurality of original images.

[0018] The AF model may be a convolutional neural network (CNN) including a plurality of layers, wherein the CNN is configured to receive one or more of the plurality of original images and a composite image generated from the plurality of original images, and output a focal length of the composite image.

[0019] The AF model may be configured to determine the focal length by detecting the contrast of the composite image, or to determine the focal length based on the phase difference of the composite image.

[0020] In one general aspect, a focal length determination method includes: receiving a plurality of raw images using a lens; generating a composite image based on the plurality of raw images; and determining a focal length of the lens by inputting the composite image into an auto focus (AF) model.

[0021] The step of generating a composite image may include: calculating an error between an original image of a reference frame selected from the plurality of original images and original images of remaining frames; and generating a composite image based on an original image having an error within a set threshold and the original image of the reference frame.

[0022] The calculating of the error may include calculating the error based on pixel values ​​of the original image of the reference frame and pixel values ​​of the original images of the remaining frames.

[0023] The step of generating the synthesized image may include: correcting the plurality of original images by frame alignment; and generating the synthesized image based on the corrected plurality of original images.

[0024] The AF model may be a convolutional neural network (CNN), which may include a plurality of layers, and receives one or more of the plurality of original images and a composite image generated from the plurality of original images, and outputs a focal length of the composite image.

[0025] The method of determining the focal length may include determining the focal length by detecting a contrast of the composite image, or determining the focal length based on a phase difference of the composite image using an AF model.

[0026] In one general aspect, a method includes: capturing burst images at preset time intervals using an image capture device including one or more lenses; calculating an error between an original image of a reference frame of the burst image and original images of remaining frames of the burst image; merging the burst images into a composite image by comparing the calculated error with a set threshold; inputting the composite image to an auto focus model (AF); and determining a focal length of at least one of the one or more lenses based on an output of the AF model, wherein the AF model is at least one of a machine learning model and a non-machine learning model.

[0027] The machine learning model may be a convolutional neural network model, and the non-machine learning model is one of a phase detection auto focus (PDAF) method and a contrast detection auto focus method (CDAF).

[0028] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 An example electronic device is shown in accordance with one or more embodiments.

[0030] Figure 2A and Figure 2B An example of operation of an example electronic device determining a focal distance using an auto focus (AF) model according to one or more embodiments is illustrated.

[0031] Figure 3An example method of determining focal length in accordance with one or more embodiments is shown.

[0032] Figure 4 Example operations for training an AF model are shown in accordance with one or more embodiments.

[0033] Figure 5 Example errors calculated by an example electronic device from a plurality of original images are shown in accordance with one or more embodiments.

[0034] Figure 6 An example method of determining focal length in accordance with one or more embodiments is shown.

[0035] Throughout the drawings and detailed description, unless otherwise described or provided, the same figure reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be to scale, and the relative sizes, proportions, and depictions of the elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0036] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein and / or the order within the operation are only examples, and are not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for the order of operations and / or the order within the operation that must occur in a specific order. As another example, except for at least a portion of the order of operations and / or the order within the operation that must occur in sequence (e.g., a specific order), the order of operations and / or the order within the operation can be performed in parallel. In addition, for greater clarity and simplicity, the description of features known after understanding the disclosure of the present application can be omitted.

[0037] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. On the contrary, the examples described herein have been provided to illustrate only some of the many possible ways of implementing the methods, devices and / or systems described herein that will be clear after understanding the disclosure of the present application. The use of the term "may" herein with respect to an example or embodiment (e.g., with respect to what an example or embodiment may include or implement) means that there is at least one example or embodiment in which such a feature is included or implemented, and all examples are not limited thereto. The use of the term "example" or "embodiment" herein has the same meaning, for example, the sentence "in one example" has the same meaning as "in one embodiment", and "one or more examples" has the same meaning as "in one or more embodiments".

[0038] The terms used herein are only used to describe various examples and will not be used to limit disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more. As a non-limiting example, the terms "include", "include" and "have" illustrate the existence of the features, quantities, operations, components, elements and / or their combinations of elaboration, but do not exclude the existence or addition of one or more other features, quantities, operations, components, elements and / or their combinations, or replace the existence of the alternative existence of the features, quantities, operations, components, elements and / or their combinations of optional elaboration. In addition, although an embodiment may set forth such a specified existence of the features, quantities, operations, components, elements and / or their combinations of elaboration, the terms "include", "include" and "have" may exist without the characteristics, quantities, operations, components, elements and / or their combinations of elaboration of one or more other embodiments.

[0039] Throughout the specification, when a component or element is described as being "on", "connected to", "bonded to" or "joined to" another component, element or layer, the component or element may be directly (e.g., in contact with) "on", "connected to", "bonded to" or "joined to" another component, element or layer, or one or more other components, elements, layers may reasonably be present in between. When a component, element or layer is described as being "directly on", "directly connected to", "directly joined to" or "directly joined to" another component, element or layer, there may be no other components, elements or layers in between. Similarly, expressions such as "between" and "immediately between" and "adjacent to" and "immediately adjacent to" may also be interpreted as described above.

[0040] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present disclosure belongs. Unless clearly defined as such herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and should not be interpreted in an idealized or overly formal sense.

[0041] Although terms such as "first", "second" and "third", or A, B, (a), (b), etc. may be used herein to describe various members, components, regions, layers or parts, these members, components, regions, layers or parts should not be limited by these terms. Each of these terms is not used to define, for example, the nature, order or sequence of the corresponding member, component, region, layer or part, but is only used to distinguish the corresponding member, component, region, layer or part from other members, components, regions, layers or parts. Therefore, without departing from the teachings of the examples, the first member, first component, first region, first layer or first part referred to in the examples described herein may also be referred to as the second member, second component, second region, second layer or second part.

[0042] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more. Unless the corresponding description and embodiments require such a list (e.g., "at least one of A, B, and C") to be interpreted as having a connecting meaning, the phrases "at least one of A, B, and C", "at least one of A, B, or C", etc. are intended to have a separate meaning, and these phrases "at least one of A, B, and C", "at least one of A, B, or C", etc. also include examples in which one or more of each of A, B, and / or C (e.g., any combination of one or more of each of A, B, and C) may exist.

[0043] Figure 1 An example electronic device 100 is shown in accordance with one or more embodiments.

[0044] Reference Figure 1 , an example electronic device 100 may include a memory 110 , a processor 120 , and a camera 130 .

[0045] The electronic device 100 may include, for example, various computing devices (such as, but not limited to, mobile phones, smart phones, tablet personal computers (PCs), e-book devices, laptop computers, PCs, and servers), various wearable devices (such as smart watches, smart glasses, head-mounted displays (HMDs), or smart clothing), various home appliances (such as smart speakers, smart televisions (TVs), and smart refrigerators), and other devices (such as smart vehicles, smart autonomous service terminals, Internet of Things (IoT) devices, walking assistance devices (WADs), drones, robots, etc.).

[0046] The memory 110 may store various pieces of data used by components of the electronic device 100 (e.g., the processor 120). For example, the various pieces of data may include a plurality of original images captured by a camera, and a composite image generated using the plurality of original images. The various pieces of data may include, for example, software (e.g., an autofocus (AF) model 115) and input data and / or output data for commands related thereto. The memory 110 may store parameters (e.g., weights and biases) of the AF model 115, which will be described in detail below. The model parameters stored in the memory 110 may be loaded into the processor 120 to implement the AF model 115 in the processor 120.

[0047] The program may be stored in the memory 110 as software, and may include, for example, an operating system (OS), middleware, or an application.

[0048] The processor 120 may be a device configured to control various operations of components included in the electronic device 100 or process given data, and may be, for example, a central processing unit (CPU). In one example, the processor 120 may receive (or obtain) a plurality of original images using a lens in response to one or more instructions being executed. However, the example is not limited thereto.

[0049] According to an example, the processor 120 may include hardware components such as a main processor and a secondary processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor or a communication processor and / or a field programmable gate array (FGPA)) as examples, and the secondary processor may operate independently of the main processor or in conjunction with (or in parallel with) the main processor.

[0050] According to an example, the auxiliary processor may include a hardware structure specified for neural network model processing. The neural network model (e.g., AF model 115) may be generated by machine learning. For example, the neural network model may be trained by the electronic device 100, a separate training device, or a separate server. As a non-limiting example, the neural network model may be configured to perform object classification, object recognition, and image recognition based on deep learning by mapping input data and output data to each other in a nonlinear relationship. Such deep learning indicates a machine learning scheme implemented by the processor.

[0051] The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, for example. The neural network model may include multiple neural network layers. In one example, the neural network model may be referred to as a machine learning model. The neural network may include, for example, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), and a bidirectional recursive deep neural network (BRDNN), a deep Q network, or a combination of two or more thereof, but is not limited thereto. The neural network model may additionally or optionally include a software structure in addition to a hardware structure.

[0052] For example, as non-limiting examples, the neural network can be or include, by way of example only, CNN, RNN, perceptron, feedforward (FF), radial basis network (RBF), deep feedforward (DFF), long short-term memory (LSTM), gated recurrent unit (GRU), autoencoder (AE), variational autoencoder (VAE), denoising autoencoder (DAE), sparse autoencoder (SAE), Markov chain (MC), Hopfield network (HN), Boltzmann machine (BM), restricted Boltzmann machine (RBM), deep belief network (DBN), deep convolutional network (DCN), deconvolution network (DN), deep convolutional inverse graphics network (DCIGN), generative adversarial network (GAN), liquid state machine (LSM), external learning machine (ELM), echo state network (ESN), deep residual network (DRN), differentiable neural computer (DNC), neural Turing machine (NTM), capsule network (CN), Cohen network (KN) and attention network (AN), etc.

[0053] In one example, an image capture device (e.g., camera 130) may capture an external environment 150 and obtain one raw image and / or multiple raw images. In one example, camera 130 may obtain multiple raw images according to a set frame during a set time. The multiple raw images captured by camera 130 may be images captured at different time points, and may be burst images captured at preset time intervals, for example. Camera 130 may include one or more lenses. Camera 130 may determine the position of the lens that captures external environment 150 according to the focal length.

[0054] In one example, the plurality of raw images may be burst raw image data obtained from a complementary metal oxide semiconductor (CMOS) image sensor. The plurality of raw images may include pieces of data that may be obtained by the sensor. For example, the plurality of raw images may include an image whose phase difference may be calculated, and a normal image without phase difference obtained by a dual photodiode (2PD) and a quad photodiode (QPD).

[0055] The electronic device 100 may determine the focal length of the lens using the AF model 115. For example, the electronic device 100 may determine the focal length of the lens by inputting one or more original images and / or synthesized images into the AF model 115. The electronic device 100 may determine the position of the lens according to the focal length. The electronic device 100 may control the lens of the camera 130 according to the determined position of the lens.

[0056] The electronic device 100 (eg, the processor 120) may generate a composite image using a plurality of original images. For example, the electronic device 100 may combine a plurality of original images to generate a composite image. The electronic device 100 may generate a composite image based on an average of pixel values ​​at each position of the plurality of original images.

[0057] The electronic device 100 may generate a composite image using all or some of the multiple original images. In one example, the electronic device 100 may generate a composite image by combining original images of a set frame among the multiple original images. In one example, when 12 frames of original images are obtained, the electronic device 100 may generate a composite image using the original images of the last four frames.

[0058] In one example, the electronic device 100 may determine the original image to be used to generate the composite image by comparing the errors of the plurality of original images with a set threshold value. For example, the electronic device 100 (e.g., the processor 120) may calculate the error between the original image of the reference frame selected from the plurality of original images and the original images of the remaining frames. The electronic device 100 may calculate the error as the average value of the difference between each pixel value (e.g., intensity) of the original image of the reference frame and each pixel value of the original image of the remaining frames. For example, the electronic device 100 may compare the pixel values ​​of the original images of the remaining frames at the same position as the position of each pixel of the original image of the reference frame.

[0059] The electronic device 100 (e.g., the processor 120) may determine the original image to be used to generate the composite image by comparing the error between the original image of the reference frame and the original image of the remaining frames with a set threshold. For example, the electronic device 100 may determine the original image of the frame corresponding to the error within a set value of the minimum error among the errors or within a set range of the minimum error among the errors as the original image to be used to generate the composite image.

[0060] For example, the AF model 115 may use one or more of the plurality of raw images and / or the synthesized image to determine the focal length.

[0061] For example, the AF model 115 may use one or more of the multiple original images and / or the synthesized image to determine the focal length according to contrast detection autofocus (CDAF) or phase detection autofocus (PDAF). In one example, CDAF may use a sensor of a camera to detect the contrast between bright and dark areas in an image. PDAF may use two sensors located on opposite sides of a camera lens to detect the phase difference of light entering the lens. In one example, the phase detection autofocus method and the contrast detection autofocus method may be referred to as non-machine learning models.

[0062] The electronic device 100 may determine the focal length according to PDAF using a plurality of original images. For example, the electronic device 100 may capture a plurality of original images while changing the position of the lens. The AF model 115 may calculate the sharpness of the image using one or more of the plurality of original images captured at a plurality of lens positions, respectively. The AF model 115 may determine the lens position at which the maximum sharpness of the image is obtained as the focal length.

[0063] In one example, the electronic device 100 may capture multiple original images while changing the position of the lens, and generate a composite image by combining the multiple original images captured at each position of the lens. The AF model 115 may determine the position of the lens that obtains the maximum sharpness of the image as the focal length using the composite image corresponding to each position of the lens.

[0064] For example, the electronic device 100 may obtain a plurality of original images including two or more images. The AF model 115 may calculate a phase difference between the two or more images according to PDAF using one or more of the plurality of original images including the two or more images. The AF model 115 may determine a moving direction and a moving amount of the lens using the calculated phase difference. The position of the lens according to the moving direction and the moving amount of the lens may correspond to a focal length. The electronic device 100 may control the lens according to the moving direction and the moving amount of the lens determined at the position of the lens where the plurality of original images are obtained.

[0065] The electronic device 100 may generate a composite image including two or more images by combining a plurality of original images including two or more images. The AF model 115 may calculate a phase difference between two or more images included in the composite image according to PDAF. The AF model 115 may determine a moving direction and a moving amount of the lens using the calculated phase difference. The position of the lens according to the moving direction and the moving amount of the lens may correspond to a focal length. The electronic device 100 may control the lens according to the moving direction and the moving amount of the lens determined at the position of the lens where the plurality of original images are obtained.

[0066] In one example, the AF model 115 may include a model trained to output a focal length by inputting one or more of a plurality of original images and / or a composite image. For example, the AF model 115 may be trained to output a focal length by inputting one or more of a plurality of original images. For example, the AF model 115 may be trained to output a focal length by inputting a composite image generated using a plurality of original images.

[0067] The training data for training the AF model 115 may include one or more of the plurality of original images and / or a synthesized image. The ground truth of the plurality of original images and the synthesized image may be determined by various methods. For example, the ground truth of the plurality of original images and the synthesized image may include a focal length calculated according to CDAF using a depth calculation or using a plurality of original images obtained at a plurality of lens positions, the depth being estimated from a multi-view stereo image using a plurality of cameras.

[0068] In one example, the AF model 115 may include a CNN, but is not limited to the above example, and may include various machine learning-based models (e.g., MobileNet-v2, VGGNet, ResNet, MCUNet, etc.). The AF model 115 may also include non-machine learning-based models (such as CDAF and PDAF).

[0069] In one example, the example electronic device 100 (e.g., the processor 120) may correct the multiple original images by performing frame alignment on the multiple original images. The example electronic device 100 may perform regional registration of the multiple original images using various known methods for image alignment. The electronic device 100 may generate a composite image using the multiple corrected original images.

[0070] When a composite image is generated from a plurality of corrected raw images by performing frame alignment, an error between raw images of each frame may be reduced. When a plurality of raw images are corrected, an error in a focal length output by the AF model 115 may be reduced.

[0071] The AF model 115 may reduce the error of the focal length by outputting the focal length using the synthetic image.

[0072] In one example, when the AF model 115 outputs the focal length according to CDAF or PDAF, the error of the focal length output by the AF model 115 increases due to noise and blur of an image with low illumination. When a plurality of original images are obtained at low illumination, and the AF model 115 outputs the focal length using a composite image obtained by combining the plurality of original images, the error of the focal length can be reduced.

[0073] In one example, when a plurality of original images are combined, noise of a composite image may be reduced as shown in Equation 1 below.

[0074] Equation 1: As shown in Equation 1 above, in one example, assuming additive white Gaussian noise (AWGN), the variance of a synthetic image generated using the average of n frames can be reduced to 1 / n. In Equation 1, It can represent a composite image generated by averaging multiple original images. can represent the original image of the i-th frame, and It can represent the standard deviation of multiple original images.

[0075] As described above, when the AF model 115 calculates the focal length of the input image based on CDAF or PDAF, the synthesized image may have reduced noise compared to a plurality of original images, and thus, an error in the focal length output by the AF model 115 may be reduced.

[0076] Table 1 below is a table showing the accuracy of focal lengths measured using a laser spot and focal lengths calculated by the first method, the second method, and the third method.

[0077] Table 1: As shown in Table 1 above, the first method, the second method, and the third method are methods by which the AF model 115 calculates the focal distance according to CDAF.

[0078] The first method is a method in which the AF model 115 calculates the focal length using one original image, the second method is a method in which the focal length is calculated using a composite image which is generated using a plurality of original images, and the third method is a method in which the focal length is calculated using a composite image which is combined using original images having an error smaller than a set threshold among a plurality of original images.

[0079] As shown in Table 1 above, it can be confirmed that the mean absolute error (MAE) and the root mean square error (RMSE) of the second method and the third method are smaller than the MAE and RMSE of the first method.

[0080] Table 2 below is a table showing the accuracy of focal lengths calculated by the first method, the second method, and the third method under low illumination.

[0081] Table 2: As shown in Table 2, it can be confirmed that the accuracy of the focal length calculated by the second method or the third method at low illumination is higher than that of the first method.

[0082] As described above, the electronic device 100 may reduce an error in a focal length by calculating a focal length using a synthesized image having reduced noise compared to a plurality of original images.

[0083] Even when the AF model is trained based on the machine learning model, the noise of the synthesized image is reduced compared to a plurality of original images, and thus, the error of the focal length output by the AF model 115 can be reduced.

[0084] In one example, in the process of training the AF model 115, since the accuracy of the ground truth calculated using the synthetic image with reduced noise may be high, the accuracy of the AF model 115 may be increased.

[0085] Table 3: In Table 3, a first method is a method of using an original image as an input of the AF model 115 and training the AF model 115 using a ground truth value calculated using the one original image, a second method is a method of using an original image as an input of the AF model 115 and training the AF model 115 using a ground truth value calculated using a synthetic image obtained by combining a plurality of original images, and a third method is a method of using a synthetic image obtained by combining a plurality of original images as an input of the AF model 115 and training the AF model 115 using a ground truth value calculated using a synthetic image obtained by combining a plurality of original images.

[0086] As shown in Table 3, it may be confirmed that the accuracy of the focal length output using the AF model 115 trained by the third method may be higher than the accuracy of the focal length output using the AF model 115 trained by the first method and the second method.

[0087] Figure 2A and 2B An example operation of the electronic device 100 determining the focal length 230 using the AF model 115 is shown.

[0088] Reference Figure 2A , the electronic device 100 may obtain a plurality of original images 211 using a camera. The electronic device 100 may generate a composite image 221 based on the plurality of original images 211. The electronic device 100 may input the composite image 221 to the AF model 115 and output a focal length 230.

[0089] Figure 2A An operation in which the electronic device 100 determines the focal length 230 by inputting the synthetic image 221 to the AF model 115 is illustrated, but is not limited thereto.

[0090] In one example, the electronic device 100 (e.g., the processor 120) may determine the focal length 230 by inputting one or more original images of the plurality of original images 211 and the synthesized image 221 to the AF model 115. When the AF model 115 is a machine learning-based model, the AF model 115 may be trained to output the focal length 230 by using one or more original images of the plurality of input original images 211 and the synthesized image 221.

[0091] Reference Figure 2B , the electronic device 100 can obtain multiple original images 211, 212, 213 and 214 at multiple lens positions.

[0092] In one example, when the position of the lens is at the first position, the electronic device 100 may obtain a plurality of original images 211 , and generate the composite image 221 using the plurality of original images 211 obtained at the first position.

[0093] In one example, when the position of the lens is at the second position, the electronic device 100 may obtain a plurality of original images 212 , and generate the composite image 222 using the plurality of original images 212 obtained at the second position.

[0094] In one example, when the position of the lens is at the third position, the electronic device 100 may obtain a plurality of original images 213 and generate the composite image 223 using the plurality of original images 213 obtained at the third position.

[0095] In one example, when the position of the lens is at the fourth position, the electronic device 100 may obtain a plurality of original images 214 and generate the composite image 224 using the plurality of original images 214 obtained at the fourth position.

[0096] The electronic device 100 may determine the focal length 230 by inputting the composite images 221 , 222 , 223 , and 224 to the AF model 115 .

[0097] In one example, the AF model 115 may calculate the sharpness of the synthesized images 221 , 222 , 223 , and 224 corresponding to each position of the lens according to CDAF, and determine the position of the lens at which the maximum sharpness is obtained.

[0098] For example, the AF model 115 may be trained to output a focal length 230 by inputting composite images 221, 222, 223, and 224 obtained by combining a plurality of original images 211, 212, 213, and 214 obtained at each position of the lens (e.g., the first position, the second position, the third position, and the fourth position).

[0099] Figure 3An example method of determining focal length in accordance with one or more embodiments is shown. Figure 3 The operations in the example may be performed in the order and manner shown. However, the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the example shown. Figure 3 The operations shown in may be performed in parallel or simultaneously. Figure 3 One or more blocks and combinations of blocks may be implemented by a computer based on special-purpose hardware that performs the specified functions or a combination of special-purpose hardware and instructions (eg, computer or processor instructions). Figure 3 In addition to the description, Figures 1 to 2B The description also applies to Figure 3 , and is incorporated herein by reference. Therefore, for the sake of brevity, the above description may not be repeated here. Figure 3 The operations may be executed by a processor.

[0100] Reference Figure 3 In operation 310 , the electronic device 100 may obtain a plurality of original images using a lens.

[0101] In one example, the electronic device 100 may generate a composite image using a plurality of original images in operation 320. In one example, the electronic device 100 may generate a composite image using an average value of pixel values ​​of a plurality of original images at the same position.

[0102] The electronic device 100 may generate a composite image using all or some of the plurality of original images. For example, the electronic device 100 may generate a composite image using a set number of original images (eg, original images within a set frame from the last frame) among the plurality of original images.

[0103] In one example, the electronic device 100 may generate a composite image based on the error between multiple original images and a set threshold. The electronic device 100 may calculate the error of the original images of the remaining frames based on the original image of the reference frame (e.g., the last frame). In one example, the electronic device 100 (e.g., the processor 120) may calculate the error using the average value of the difference between the pixel value at each position of the original image of the reference frame and the pixel value at each position of the original image of the remaining frames. The electronic device 100 may set the error with the minimum value among the calculated errors as a reference error. The electronic device 100 may generate a composite image using an original image with an error within a set range from the reference error.

[0104] In one example, in operation 330 , the electronic device 100 may determine a focal length of a lens by inputting one or more original images of a plurality of original images and a synthesized image to the AF model 115 .

[0105] In one example, the AF model 115 may determine the focal length using a CDAF method or a PDAF method.

[0106] In one example, when the AF model 115 determines the focal length according to the CDAF method, the sharpness of one or more original images and the synthesized image among the plurality of original images may be calculated. The AF model 115 may calculate the sharpness at the position of the corresponding lens based on the sharpness of the one or more original images and the sharpness of the synthesized image. In one example, the AF model 115 may calculate the sharpness based on the lens position as an average (or weighted average) of the sharpness of the one or more original images and the sharpness of the synthesized image.

[0107] The AF model 115 may determine the focal length based on the calculated sharpness at the position of each lens.

[0108] In one example, when the AF model 115 determines the focal length according to the PDAF method, each of the movement direction of the lens and the movement amount of the lens may be detected from one or more original images and the synthesized image among the plurality of original images. The AF model 115 may determine the focal length using the movement direction of the lens and the movement amount of the lens detected from the one or more original images, and the movement direction of the lens and the movement amount of the lens detected from the synthesized image.

[0109] In one example, when the movement direction of the lens detected from one or more original images is the same as the movement direction of the lens detected from the synthetic image, the AF model 115 may determine the focal length using an average (or weighted average) of the movement amount of the lens detected from the one or more original images and the movement amount of the lens detected from the synthetic image.

[0110] In one example, when the movement direction of the lens detected from one or more original images is different from the movement direction of the lens detected from the synthesized image, the AF model 115 may determine the focal length based on the movement direction of the lens detected from the synthesized image and the movement amount of the lens. For example, when the movement direction of the lens detected from one or more original images is different from the movement direction of the lens detected from the synthesized image, the electronic device 100 may recapture a plurality of original images using a camera and determine the focal length using the plurality of recaptured original images.

[0111] Figure 4 Example operations of training AF model 115 are shown in accordance with one or more embodiments. Figure 4 The operations in the example may be performed in the order and manner shown. However, the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the example shown. Additionally, Figure 4 The operations shown in may be performed in parallel or simultaneously. Figure 4One or more blocks and combinations of blocks may be implemented by a computer based on special-purpose hardware that performs the specified functions or a combination of special-purpose hardware and instructions (eg, computer or processor instructions). Figure 4 In addition to the description, Figures 1 to 3 The description also applies to Figure 4 , and are incorporated herein by reference. Therefore, for the sake of brevity, the above description may not be repeated here. Figure 4 The operations may be executed by a processor.

[0112] Figure 4 An example in which the AF model 115 is trained by the electronic device 100 is shown, but is not limited to referring to Figure 4 The examples described, and in one example, the AF model 115 may be trained by a training device, an external electronic device 100, or a server separate from the electronic device 100.

[0113] In operation 405 , the electronic device 100 may obtain a plurality of original images using a lens.

[0114] In operation 410 , the electronic device 100 may generate a composite image using a plurality of original images.

[0115] In operation 415, the electronic device 100 may obtain focal lengths of a plurality of original images and a synthesized image. The focal length obtained in operation 415 may indicate a label or ground truth of training data used to train the AF model 115. For example, the focal length may include a focal length calculated using a depth estimated from a multi-view stereo image, or a focal length calculated according to a CDAF method using a plurality of original images or synthesized images captured at a plurality of lens positions. Since a focal length corresponding to a label or ground truth of training data is obtained in operation 415, training of the AF model 115 may be easily performed.

[0116] In operation 420 , the electronic device 100 may determine a focal length of a lens by inputting one or more raw images of a plurality of raw images to the AF model 115 .

[0117] In operation 425 , the electronic device 100 may train the AF model 115 using the acquired focal length and the focal length determined by the AF model 115 .

[0118] For example, the AF model 115 trained in operations 420 and 425 may mean a model trained to output a focal length by inputting one or more raw images among a plurality of raw images.

[0119] In operation 430 , the electronic device 100 may determine a focal length of a lens by inputting one or more original images of a plurality of original images and a synthesized image to the AF model 115 .

[0120] In operation 435 , the electronic device 100 may train the AF model 115 using the acquired focal length and the focal length determined by the AF model 115 .

[0121] For example, the AF model 115 trained in operations 430 and 435 may mean a model trained to output a focal length by inputting one or more original images and a synthesized image among a plurality of original images.

[0122] In operation 440 , the electronic device 100 may determine the focal length of the lens by inputting the composite image to the AF model 115 .

[0123] In operation 445 , the electronic device 100 may train the AF model 115 using the acquired focal length and the focal length determined by the AF model 115 .

[0124] In one example, the AF model 115 trained in operations 440 and 445 may mean a model trained to output a focal length by inputting a synthetic image.

[0125] Can be executed Figure 4 The AF model 115 may be trained by performing one of the operations 420 and 425, the operations 430 and 435, and the operations 440 and 445 shown in the figure. Additionally, in one example, the AF model 115 may be trained by performing two or more of the operations 420 and 425, the operations 430 and 435, and the operations 440 and 445 together. In this example, predetermined weights may be applied to the operations performed together, and the training of the AF model 115 may be affected at the same or different rates.

[0126] Figure 5 Illustrated are examples of errors calculated by the electronic device 100 from a plurality of original images according to one or more embodiments.

[0127] The electronic device 100 may calculate the error between the original image 510-11 of the reference frame (e.g., the last frame) selected from the plurality of original images 510-1, ..., and 510-11 and the original images 510-1, ..., and 510-10 of the remaining frames. When the plurality of original images are captured continuously, there may be differences between the plurality of original images due to changes in illumination and movement of the object. The image of the reference frame to be used to generate the composite image may be selected from the plurality of original images 510-1, ..., and 510-11 as the image satisfying the predetermined condition. For example, the image satisfying the predetermined condition may include an image with the highest sharpness, but may include an image with various conditions for selecting an image that clearly captures an object without shaking (e.g., an image with minimal noise) without limitation. In some examples, when the images are captured continuously, the last image captured may be the image most clearly captured without shaking, and in this example, the image of the last frame may be selected as the image of the reference frame.

[0128] The electronic device 100 may calculate an error between the original image 510-11 of the reference frame and the original images 510-1, ..., and 510-10 of the remaining frames. In one example, the electronic device 100 may calculate an error except for the original image 510-1 of the first frame.

[0129] Figure 5 The error between the original image 510-11 of the reference frame and the original image 510-10 of the tenth frame is shown to be 0.0308. It can be confirmed that the error of the original image 510-10 of the tenth frame is the smallest among the errors of the original images 510-2, ..., and 510-10. The electronic device 100 can determine the error of the original image 510-10 of the tenth frame as a reference error. The electronic device 100 can generate a synthetic image by using the original images 510-8, 510-9 and 510-10 and the original image 510-11 of the reference frame, and the original images 510-8, 510-9 and 510-10 have errors of the original images within a set threshold. In one example, the error within the set threshold may represent an error within a set value (e.g., 0.001) from the reference error or within a set range (e.g., a set percentage (e.g., 5%)) of the reference error. As described above, since the composite image is generated based on the original image of the reference frame that is captured most clearly and the original image having an error within a threshold value from the original image of the reference frame, the electronic device 100 can effectively obtain a composite image that clearly captures the object without shaking, and can use the composite image to obtain the focal length of the lens from the AF model with high accuracy.

[0130] Figure 6 An example of a method of determining focal length according to one or more embodiments is shown. Figure 6The operations in the example may be performed in the order and manner shown. However, the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the example shown. Additionally, Figure 6 The operations shown in may be performed in parallel or simultaneously. Figure 6 One or more blocks and combinations of blocks may be implemented by a computer based on special-purpose hardware that performs the specified functions or a combination of special-purpose hardware and instructions (eg, computer or processor instructions). Figure 6 In addition to the description, Figures 1 to 5 The description also applies to Figure 6 , and are incorporated herein by reference. Therefore, for the sake of brevity, the above description may not be repeated here. Figure 6 The operations may be executed by a processor.

[0131] Reference Figure 6 In operation 610, the electronic device 100 may obtain a plurality of original images through operation of a lens.

[0132] In operation 620 , the electronic device 100 may generate a composite image using a plurality of original images.

[0133] Figure 3 The description of operation 310 and operation 320 may be applied to operation 610 and operation 620 in substantially the same manner.

[0134] In operation 630 , the electronic device 100 may determine the focal length of the lens by inputting the composite image to the AF model 115 .

[0135] In one example, the AF model 115 may determine the focal length from the composite image according to the CDAF method or the PDAF method.

[0136] In one example, the AF model 115 may represent a model trained to output a focal length by inputting a synthetic image.

[0137] Figures 1 to 6The electronic device 100, memory 110, AF model 115, processor 120, camera 130, other components and devices, and other circuit systems, components or devices described herein are implemented as hardware components and implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements (such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve the desired result). In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. The hardware components implemented by a processor or a computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to control the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or the processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component or two or more hardware components. The hardware components may have any one or more of different processing configurations, examples of which include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0138] Perform the steps described in this application and Figures 1 to 6The operation method shown in is performed by computing hardware (e.g., by one or more processors or computers), which is implemented as executing instructions or software as described above to control the operations performed by the method described in this application. For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or another processor and another controller (e.g., as individual operations of the processor implementing the method). One or more processors, or a processor and a controller can perform a single operation or two or more operations.

[0139] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to control a neural network circuit or circuit system to perform the methods described above may be written as a computer program, code segment, instruction, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special purpose computer to perform such control of the operations performed by the neural network circuit or circuit system and method as described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. Instructions or software may be written in any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in this specification, which disclose algorithms for performing the operations performed by the hardware components and methods described above.

[0140] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement control of hardware components to perform the methods described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media (e.g., Figure 1Examples of non-transitory computer-readable storage media include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage device, hard disk drive (HDD), solid state drive (SSD), card memory (such as, multimedia card or micro card (for example, secure digital (SD) or extreme digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device, any other device is configured to store instructions or software and any associated data, data files and data structures in a non-transitory manner and provide instructions or software and any associated data, data files and data structures to one or more processors and computers so that one or more processors and computers can execute instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that the instructions and software and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0141] Although the present disclosure includes specific examples, it will be clear to those of ordinary skill in the art after understanding the disclosure of the present application that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered in a descriptive sense only, not for the purpose of limitation. The description of the features or aspects in each example will be considered applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents, suitable results may be achieved.

[0142] As described above, although examples have been described with reference to limited figures, those skilled in the art may apply various technical modifications and changes based thereon. For example, if the described techniques are performed in a different order and / or if the components in the described systems, architectures, devices or circuits are combined in a different manner and / or replaced or supplemented by other components or their equivalents, appropriate results may be achieved.

[0143] Therefore, the scope of the disclosure may be defined by the claims and their equivalents in addition to the above disclosure, and all variations within the scope of the claims and their equivalents should be construed as being included in the disclosure.

Claims

1. An electronic device, comprising: processor; Cameras, including lenses; and a memory configured to store one or more instructions executable by the processor, The processor is configured as follows: receiving a plurality of raw images using a lens in response to the one or more instructions being executed; generating a composite image based on the plurality of original images; and The focal length of the lens is determined by inputting one or more of the plurality of original images and the composite image into an auto-focus model.

2. The electronic device according to claim 1, wherein: The processor is also configured to: calculating an error between an original image of a reference frame selected from the plurality of original images and original images of remaining frames; and A synthesized image is generated based on an original image having an error within a set threshold and an original image of a reference frame.

3. The electronic device according to claim 2, wherein: The processor is also configured to: An error is calculated based on pixel values ​​of the original image of the reference frame and pixel values ​​of the original images of the remaining frames.

4. The electronic device according to claim 1, wherein: The processor is also configured to: Correcting the plurality of original images by frame alignment; and A composite image is generated based on the corrected plurality of original images.

5. The electronic device according to claim 1, wherein: The autofocus model is a convolutional neural network consisting of multiple layers. Wherein, the convolutional neural network is configured to output a focal length using one or more of the plurality of original images and a composite image generated from the plurality of original images.

6. The electronic device according to claim 1, wherein: The autofocus model is configured as: The focal length is determined by detecting the contrast of the composite image, or based on the phase difference of the composite image.

7. An electronic device comprising: processor; Cameras, including lenses; and a memory configured to store one or more instructions executable by the processor, The processor is configured as follows: receiving a plurality of raw images using a lens in response to the one or more instructions being executed; generating a composite image based on the plurality of original images; and The focal length of the lens is determined by inputting the composite image into an autofocus model.

8. The electronic device according to claim 7, wherein: The processor is also configured to: calculating an error between an original image of a reference frame selected from the plurality of original images and original images of remaining frames; and A synthesized image is generated based on an original image having an error within a set threshold and an original image of a reference frame.

9. The electronic device according to claim 8, wherein: The processor is also configured to: An error is calculated based on pixel values ​​of the original image of the reference frame and pixel values ​​of the original images of the remaining frames.

10. The electronic device according to claim 7, wherein: The processor is also configured to: Correcting the plurality of original images by frame alignment; and A composite image is generated based on the corrected plurality of original images.

11. The electronic device according to claim 7, wherein: The autofocus model is a convolutional neural network consisting of multiple layers. Wherein, the convolutional neural network is configured to output a focal length using one or more of the plurality of original images and a composite image generated from the plurality of original images.

12. The electronic device according to claim 7, wherein: The autofocus model is configured as: The focal length is determined by detecting the contrast of the composite image, or based on the phase difference of the composite image.

13. A method for determining focal length, the method comprising: Using a lens to receive multiple raw images; generating a composite image based on the plurality of original images; as well as The focal length of the lens is determined by inputting the composite image into an autofocus model.

14. The method according to claim 13, wherein: The steps to generate the composite image include: calculating an error between an original image of a reference frame selected from the plurality of original images and original images of remaining frames; and A synthesized image is generated based on an original image having an error within a set threshold and an original image of a reference frame.

15. The method according to claim 14, wherein: The steps to calculate the error include: An error is calculated based on pixel values ​​of the original image of the reference frame and pixel values ​​of the original images of the remaining frames.

16. The method according to claim 13, wherein: The steps to generate the composite image include: Correcting the plurality of original images by frame alignment; and A composite image is generated based on the corrected plurality of original images.

17. The method according to claim 13, wherein: The auto-focus model is a convolutional neural network that includes a plurality of layers and outputs a focal length using one or more of the plurality of original images and a synthetic image generated from the plurality of original images.

18. The method according to claim 13, wherein: Methods for determining focal length include: The focus is determined by detecting the contrast of the composite image or using an autofocus model based on the phase difference of the composite image.

19. A method for determining focal length, comprising: capturing burst images at preset time intervals using an image capture device including one or more lenses; calculating an error between an original image of a reference frame of a burst image and original images of remaining frames of the burst image; The burst images are merged into a composite image by comparing the calculated error with a set threshold; inputting the synthesized image into an autofocus model; and determining a focal length of at least one of the one or more lenses based on an output of an autofocus model, Wherein, the auto-focus model is at least one of a machine learning model and a non-machine learning model.

20. The focal length determination method according to claim 19, wherein: The machine learning model is a convolutional neural network model, and the non-machine learning model is one of a phase detection autofocus method and a contrast detection autofocus method.

Images