Electronic device for processing image, and operation method thereof

Abstract

An electronic device (for example, an electronic device (101) in figure 1 and an electronic device (101) in figure 3) according to one embodiment may comprise a camera, at least one processor, and a memory. When executed individually or collectively by the at least one processor, the memory can instruct the electronic device to: acquire, through an image sensor, a first image frame including first correlation information between first data acquired through a first sub-pixel of the image sensor and second data acquired through a second sub-pixel of the image sensor; on the basis of the first correlation information, perform spatial filtering in which at least one weight is applied to correlation values for a plurality of regions into which the first image frame is divided, so as to acquire second correlation information about a region of interest; perform temporal filtering in which the second correlation information is blended with third correlation information corresponding to a second image frame acquired at a point differing from that of the first image frame, so as to acquire fourth correlation information about the region of interest; and determine depth information on the basis of the fourth correlation information.

Description

Electronic device for processing images and method of operation thereof

[0001] The present disclosure relates to an electronic device for processing images and a method of operating the same.

[0002] In a device that captures a subject (e.g., a camera), the subject must be in precise focus to obtain a clear image (e.g., a still image or a video). Focusing methods for such devices include contrast autofocus and phase autofocus. Contrast autofocus refers to a method of adjusting focus by utilizing the characteristic that when the focus is accurate, the subject's outline becomes sharp and contrast increases, while when the focus is off, the subject's outline becomes blurry and contrast decreases. Phase autofocus refers to a method of measuring the distance to a subject by utilizing the characteristic that the phase changes depending on the focus when the incident light from the lens differs in an image. It does this by using two images with different incident light sources acquired through an imaging element (e.g., an image sensor). The depth value of a position within the image can be determined based on the difference in phase occurring within the two images. While phase autofocus allows for the prediction of the direction of movement of the lens group and can perform focus adjustment operations faster than contrast autofocus, errors may occur.

[0003] When a sensor for phase difference detection is integrated into an imaging element (e.g., CMOS image sensor, CCD image sensor), fast phase difference autofocus (AF) is possible, and phase difference AF is applied to many cameras.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art in relation to the present disclosure.

[0005] An electronic device according to one embodiment may include a camera, at least one processor, and memory. The camera may include an image sensor. At least one processor may include processing circuitry. The memory may store instructions. The image sensor may include a microlens array and unit pixels corresponding to the microlens array. The unit pixels may include a first subpixel and a second subpixel. The instructions may be executed individually or collectively by the at least one processor to enable the electronic device to acquire a first image frame through the image sensor, the first image frame including first correlation information between first data acquired through the first subpixel and second data acquired through the second subpixel. The above instructions may be executed individually or collectively by the at least one processor to enable the electronic device to obtain second correlation information for a region of interest by performing a spatial filter that applies at least one weight to correlation values ​​for a plurality of regions that divide the first image frame based on the first correlation information. The above instructions may be executed individually or collectively by the at least one processor to enable the electronic device to obtain fourth correlation information for a region of interest by performing a temporal filter that blends the second correlation information with third correlation information corresponding to a second image frame obtained at a different time from the first image frame.The above instructions may be executed individually or collectively by the at least one processor to enable the electronic device to determine depth information based on the fourth correlation information.

[0006] A method of operating an electronic device including an image sensor according to one embodiment may include the operation of acquiring a first image frame through the image sensor, the first image frame including first correlation information between first data acquired through a first subpixel of the image sensor and second data acquired through a second subpixel of the image sensor. The method may include the operation of acquiring second correlation information for a region of interest by performing a spatial filter that applies at least one weight to correlation values ​​for a plurality of regions that divide the first image frame based on the first correlation information. The method may include the operation of acquiring fourth correlation information for the region of interest by performing a temporal filter that blends the second correlation information with third correlation information corresponding to a second image frame acquired at a different time from the first image frame. The method may include the operation of determining depth information based on the fourth correlation information.

[0007] In one embodiment, a computer-readable non-transient recording medium may have a computer program recorded thereon that causes an electronic device to perform at least one operation upon execution. The at least one operation may include an operation of acquiring a first image frame through the image sensor that includes first correlation information between first data acquired through a first subpixel of the image sensor and second data acquired through a second subpixel of the image sensor. The at least one operation may include an operation of acquiring second correlation information for a region of interest by performing a spatial filter that applies at least one weight to correlation values ​​for a plurality of regions that divide the first image frame based on the first correlation information. The at least one operation may include an operation of acquiring fourth correlation information for the region of interest by performing a temporal filter that blends the second correlation information with third correlation information corresponding to a second image frame acquired at a different time from the first image frame. The at least one operation may include an operation of determining depth information based on the fourth correlation information.

[0008] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0009] FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

[0010] FIG. 3 is a block diagram illustrating the configuration of an electronic device according to one example.

[0011] FIG. 4 is a diagram illustrating the configuration of an image sensor according to one embodiment.

[0012] FIG. 5 is a diagram illustrating an example of a pixel structure of an image sensor according to one embodiment.

[0013] FIG. 6 is a diagram illustrating an example of a pixel structure of an image sensor according to one embodiment.

[0014] FIG. 7 is a drawing illustrating an example of a pixel structure of an image sensor according to one embodiment.

[0015] FIG. 8 is a diagram illustrating a method for reading photoelectric conversion signals of a plurality of light-receiving elements according to one embodiment.

[0016] FIG. 9 is a drawing illustrating an example of a plurality of image pixels according to one embodiment.

[0017] FIG. 10 is a schematic diagram illustrating a method of an electronic device according to one embodiment performing correlation operations on data for phase difference detection.

[0018] FIG. 11 is a flowchart illustrating a process in which an electronic device according to one embodiment acquires depth information.

[0019] FIG. 12 illustrates the process of an electronic device according to one embodiment outputting correlation information based on a finite impulse filter (FIR) structure.

[0020] FIG. 13 illustrates the process of an electronic device according to one embodiment outputting correlation information based on an infinite impulse filter (IIR) structure.

[0021] FIG. 14 is a block diagram illustrating an example of a spatial filter according to one embodiment.

[0022] FIG. 15 is a diagram illustrating an example of determining weights based on the distance between regions in one embodiment.

[0023] FIG. 16 is a diagram illustrating an example of determining weights based on the difference in pixel values ​​in one embodiment.

[0024] FIG. 17 is a diagram illustrating an example of determining weights based on depth difference in one embodiment.

[0025] FIG. 18 is a block diagram illustrating an example of a time filter according to one embodiment.

[0026] FIG. 19 is a diagram illustrating an example of a method for determining the weights of a time filter according to one embodiment.

[0027] FIG. 20 is a diagram illustrating an example of a method for determining the weights of a time filter according to one embodiment.

[0028] FIG. 21 illustrates an example of correlation information obtained by an electronic device according to one embodiment.

[0029] Hereinafter, embodiments are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the contents of this disclosure. However, the disclosed embodiments may be implemented in various different forms and are not limited to the embodiments described herein.

[0030] In the present disclosure, an electronic device and a method of operation according to various embodiments may be intended to improve the accuracy of phase difference data.

[0031] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

[0032] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0033] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0034] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0035] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0036] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0037] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0038] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0039] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0040] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0041] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0042] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0043] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0044] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0045] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0046] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0047] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0048] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0049] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0050] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0051] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0052] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0053] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0054] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0055] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0056] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0057] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0058] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., download or upload) through or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0059] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0060] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is the target of image capture. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties different from the lens properties of other lens assemblies. The lens assemblies (210) may include, for example, a wide-angle lens or a telephoto lens.

[0061] A flash (220) may emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) may include one or more light-emitting diodes (e.g., RGB (red-green-blue) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. An image sensor (230) may acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through a lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) may include one image sensor selected from image sensors with different properties, such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0062] The image stabilizer (240) may move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operational characteristics of the image sensor (230) (e.g., adjusting read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) containing it. This allows for compensating for at least some of the negative effects caused by the movement on the image being captured. According to one embodiment, the image stabilizer (240) may detect such movement of the camera module (180) or the electronic device (101) using a gyroscope sensor (not shown) or an accelerometer sensor (not shown) placed inside or outside the camera module (180). According to one embodiment, the image stabilizer (240) may be implemented as, for example, an optical image stabilizer. The memory (250) may temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing operation. For example, if image acquisition by the shutter is delayed or multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and the corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Subsequently, when a specified condition is satisfied (e.g., user input or system command), at least a portion of the original image stored in the memory (250) may be acquired and processed by, for example, an image signal processor (260). According to one embodiment, the memory (250) may be configured as at least a portion of the memory (130) or as a separate memory that operates independently thereof.

[0063] The image signal processor (260) can perform one or more image processing on an image acquired through the image sensor (230) or an image stored in memory (250). The one or more image processing may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softing). Additionally or generally, the image signal processor (260) can perform control (e.g., exposure time control, or readout timing control, etc.) over at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may be configured as at least part of the processor (120) or as a separate processor operating independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after further image processing by the processor (120).

[0064] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180) each having different attributes or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera and at least another may be a rear camera.

[0065] FIG. 3 is a block diagram illustrating the configuration of an electronic device (101) according to one example (e.g., the electronic device (101) of FIG. 1).

[0066] In one embodiment, the electronic device (101) may include at least one processor (320) (e.g., processor (120) of FIG. 1, image signal processor (260)), at least one camera (380) (e.g., camera module (180) of FIG. 1 and FIG. 2), and memory (330) (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2). The electronic device (101) may further include at least one display (360) for displaying a screen (e.g., display module (160) of FIG. 1).

[0067] In one embodiment, the memory (330) may store instructions. At least one processor (320) may execute the instructions stored in the memory (330) individually or collectively to perform operations or control components of the electronic device (101). In the present disclosure, the operation of the electronic device (101) may be understood as being performed when the at least one processor (320) executes the instructions.

[0068] In one embodiment, the camera (380) may include an image sensor (e.g., the image sensor (230) of FIG. 2). The electronic device (101) may acquire an image frame containing data acquired through the image sensor of the camera (380). At least one processor (320) may be composed of hardware, at least a portion of which is included in the camera (380) or provided separately from the camera (380). For example, the at least one processor (320) may include an application processor included in the electronic device (101) or a processing unit of the image sensor of the camera (380).

[0069] In one embodiment, the electronic device (101) may acquire a first image frame containing first correlation information through the image sensor of the camera (380). In the present disclosure, the correlation information may refer to information obtained by performing a correlation operation on data acquired from the image sensor. The correlation information may include correlation values ​​for a plurality of regions that divide the image frame. For example, the correlation information may include correlation values ​​obtained from pixels corresponding to a region within the pixel array of the image sensor. For example, the correlation values ​​may include values ​​obtained through SAD (sum of absolute difference), SSD (sum of squared difference), or MAD (mean of absolute difference). The electronic device may determine a region of interest (RoI) among the plurality of regions. For example, the electronic device (101) may determine the region corresponding to the center of the image among the plurality of regions as the region of interest. For example, the electronic device (101) may identify a main subject within the image and determine the region corresponding to the location of the main subject as the region of interest. For example, the electronic device (101) can determine an area corresponding to the location of a touch input received through a display (360) including a touchscreen as an area of ​​interest.

[0070] In one embodiment, the electronic device (101) may obtain second correlation information by applying a spatial filter to the first correlation information. Spatial filtering may mean correcting the correlation value for one region (e.g., region of interest) within an image frame based on the correlation value for another region. The electronic device (101) may apply a temporal filter to the second correlation information. Temporal filtering may mean correcting the correlation information of one image frame based on the correlation information of another image frame. The electronic device (101) may obtain fourth correlation information by blending the second correlation information with third correlation information corresponding to at least one second image frame obtained at a different time from the first image frame (e.g., before the time of acquisition of the first image frame).

[0071] In one embodiment, the electronic device (101) can obtain depth information based on the fourth correlation information. For example, the electronic device (101) can determine a phase difference (or disparity) value corresponding to the point where the difference value included in the fourth correlation information is minimum. The electronic device (101) can obtain depth information corresponding to the determined phase difference value. The electronic device (101) can control the focus adjustment function of the camera (380) based on the depth information. The electronic device (101) can display the image obtained through the camera (380) through the display (360).

[0072] FIG. 4 is a diagram illustrating the configuration of an image sensor (e.g., the image sensor (230) of FIG. 2) according to one embodiment.

[0073] In one embodiment, the image sensor may include a micro lens array (MLA) (411), a color filter (413), a light receiving unit (415), and a processing unit (417).

[0074] In one embodiment, the micro-lens array (411) may be configured such that a beam of light (421) that passes through a lens unit (e.g., lens assembly (210) of FIG. 2) and forms an image on an image sensor is collected by a light receiving element of the light receiving unit (415). The beams of light (423) that pass through the micro-lens array (411) may pass through a color filter (413), at least partially blocking wavelengths other than the band corresponding to a specific color. The beams of light (425) that pass through the color filter (413) may be detected by a light receiving element (e.g., a photodiode) of the light receiving unit (415). The light receiving unit (415) may include a light receiving element that generates a charge and converts it into an electrical signal upon receiving light, and a circuit that selectively reads out the charge of the light receiving element. Between the light receiving unit (415) and the operation unit (417), a circuit for digitizing the signal read from the light receiving unit (415) or reducing noise may be arranged.

[0075] In one embodiment, the microlens array (411) may be arranged to correspond to at least one light-receiving element. For example, when viewed from the direction in which the light beam (421) is incident, the area in which a single microlens included in the microlens array (411) is arranged may overlap at least partially with the area in which a plurality of light-receiving elements are arranged. The microlenses included in the microlens array (411) may be arranged in a different color channel from adjacent microlenses, but a plurality of microlenses corresponding to the same color channel may be arranged adjacent to each other. The arrangement between the microlens array (411), the color filter (413), and the light-receiving unit (415) may be configured differently depending on the type of image sensor.

[0076] In one embodiment, the operation unit (417) may perform an operation to process electrical data (427) output from the light receiving unit (415). The operation unit (417) may output the acquired data based on the result of the operation. The output of the operation unit (417) may be the output of an image sensor (e.g., the image sensor (230) of FIG. 2). In one embodiment, the operation unit (417) may perform an operation to calibrate the read-out data as an operation to process the electrical data (427). For example, the operation performed by the operation unit (417) may include at least one of an operation to reduce the deviation between pixels due to optical asymmetry or the relative position of the sensor, an operation to reduce noise generated in the analog signal, an operation to remove defects, an operation to perform remosaic, or an operation according to a specific application field (e.g., proximity sensor function, timing control function, HDR (high dynamic range) tone mapping function). The sensor output (429) output from the operation unit (417) can be input to at least one processor (e.g., application processor) through an interface.

[0077] In FIG. 4, the color pattern of the color filter (413) is illustrated based on a Bayer pattern, but the color pattern of the color filter (413) according to one embodiment is not limited to that illustrated in FIG. 4. Regions within the color filter (413) corresponding to a plurality of adjacent microlenses may be configured to include the same color channel. For example, an image sensor including the color filter (413) may include a structure in which the pattern illustrated in FIG. 5, FIG. 6, or FIG. 7 is repeated.

[0078] FIG. 5 is a drawing illustrating an example of a pixel structure of an image sensor (e.g., the image sensor (230) of FIG. 2) according to one embodiment.

[0079] In one embodiment, a unit pixel (510) included in an image sensor may include a color filter, at least one microlens, and at least one light receiving element. The microlens may refract light and concentrate it onto at least one light receiving element. The color filter may be configured to transmit light of a specified wavelength (color) band. When light passing through the microlens and the color filter reaches the light receiving element, the light receiving element may output an electrical signal corresponding to the incident light by the photoelectric effect. The light receiving element may also be referred to as a photoelectric conversion element. For example, the light receiving element may include at least one of a photodiode, a pinned-photodiode, a phototransistor, or a photogate. In this disclosure, a photodiode constituting a unit capable of reading an electrical signal may be referred to as a subpixel.

[0080] In one embodiment, a unit pixel (510) of an image sensor may include a first subpixel (531) and a second subpixel (532) positioned at a location corresponding to a first microlens (521). Referring to FIG. 5, the first microlens (521) may be positioned in an area corresponding to a green filter. A microlens positioned adjacent to the first microlens (521) may be positioned in an area corresponding to a color filter different from the color filter corresponding to the first microlens (521). Referring to FIG. 5, the second microlens (522) may be positioned in an area corresponding to a red filter. The third microlens (523) may be positioned in an area corresponding to a blue filter. The fourth microlens (524) may be positioned in an area corresponding to a green filter, which is a different color from the color filters corresponding to the adjacent second microlens (522) and third microlens (523). The image sensor may include a third subpixel (533) and a fourth subpixel (534) positioned at a location corresponding to the second microlens (522). The image sensor may include a fifth subpixel (535) and a sixth subpixel (536) positioned at a location corresponding to the third microlens (523). The image sensor may include a seventh subpixel (537) and an eighth subpixel (538) positioned at a location corresponding to the fourth microlens (524).

[0081] In one embodiment, a structure in which two subpixels (531, 532) are arranged within a unit pixel (510) may be referred to as a dual pixel structure (hereinafter referred to as a '2PD structure') (500). In one embodiment, the image sensor may include a repeating pattern of the pixel structure shown in FIG. 5.

[0082] In one embodiment, subpixels included within a unit pixel (510) can independently output a signal corresponding to incident light. An image sensor can capture an image from a signal obtained from at least some of the subpixels. Referring to FIG. 5, each unit pixel can be divided into two parts in one direction (e.g., the Y direction in FIG. 5) by the arrangement of two subpixels. Photoelectric conversion signals can be independently output from individual subpixels.

[0083] FIG. 6 is a drawing illustrating an example of a pixel structure of an image sensor (e.g., the image sensor (230) of FIG. 2) according to one embodiment.

[0084] In one embodiment, the unit pixel (610) of the image sensor may include a first subpixel (631), a second subpixel (632), a third subpixel (633), and a fourth subpixel (634) positioned at a location corresponding to a first microlens (621). Referring to FIG. 6, the first microlens (621) may be positioned in an area corresponding to a green filter. A microlens positioned adjacent to the first microlens (621) may be positioned in an area corresponding to a color filter different from the color filter corresponding to the first microlens (621). Referring to FIG. 6, the second microlens (622) may be positioned in an area corresponding to a red filter. The third microlens (623) may be positioned in an area corresponding to a blue filter. The fourth microlens (624) may be positioned in an area corresponding to a green filter, which is a different color from the color filters corresponding to the adjacent second microlens (622) and third microlens (623). The image sensor may include a fifth subpixel (635), a sixth subpixel (636), a seventh subpixel (637), and an eighth subpixel (638) positioned at a location corresponding to the second microlens (622). The image sensor may include a ninth subpixel (639), a tenth subpixel (640), an eleventh subpixel (641), and a twelfth subpixel (642) positioned at a location corresponding to the third microlens (623). The image sensor may include a thirteenth subpixel (643), a fourteenth subpixel (644), a fifteenth subpixel (645), and a sixteenth subpixel (646) positioned at a location corresponding to the fourth microlens (624).

[0085] In one embodiment, a structure in which four subpixels (631, 632, 633, 634) are arranged within a unit pixel (610) may be referred to as a '4PD structure' (600). In one embodiment, the image sensor may include a repeating pattern of the pixel structure shown in FIG. 6.

[0086] FIG. 7 is a drawing illustrating an example of a pixel structure of an image sensor (e.g., the image sensor (230) of FIG. 2) according to one embodiment.

[0087] In one embodiment, a unit pixel (711) of an image sensor may include a 2 x 2 array of micro-lenses (721, 722, 723, 724) positioned at a location corresponding to a single color filter (e.g., a green filter). The unit pixel (711) may include four sub-pixels positioned at a location corresponding to each of the micro-lenses (721, 722, 723, 724). Referring to FIG. 7, the image sensor may include a first sub-pixel (731), a second sub-pixel (732), a third sub-pixel (733), and a fourth sub-pixel (734) positioned at a location corresponding to the first micro-lens (721). The image sensor may include a fifth sub-pixel (735), a sixth sub-pixel (736), a seventh sub-pixel (737), and an eighth sub-pixel (738) positioned at a location corresponding to the second micro-lens (722). The image sensor may include a ninth subpixel (739), a tenth subpixel (740), an eleventh subpixel (741), and a twelveth subpixel (742) positioned at a location corresponding to the third microlens (723). The image sensor may include a thirteenth subpixel (743), a fourteenth subpixel (744), a fifteenth subpixel (745), and a sixteenth subpixel (746) positioned at a location corresponding to the fourth microlens (724).

[0088] In one embodiment, a structure in which four micro-lenses (721, 722, 723, 724) are arranged within a unit pixel (711) and four sub-pixels are arranged for each micro-lens may be referred to as a tetra-square pixel structure (hereinafter referred to as a 'hexadeca pixel structure') (700). In one embodiment, the image sensor may include a repeating pattern of the pixel structure shown in FIG. 7.

[0089] The pixel structures (500, 600, 700) illustrated in FIGS. 5 to 7 are for explaining the structure of an image sensor according to one embodiment, and the pixel structure of the image sensor is not limited to the pixel structures (500, 600, 700) illustrated in FIGS. 5 to 7.

[0090] FIG. 8 is a diagram illustrating a method for reading photoelectric conversion signals of a plurality of light-receiving elements according to one embodiment.

[0091] In one embodiment, photoelectric conversion signals output from a plurality of subpixels arranged within a unit pixel may represent the phase of light incident on a light receiving element at different angles of incidence. The signals output from the subpixels may be divided into a first component and a second component depending on the position arranged within the unit pixel. An electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may detect a phase difference for the incident light through a comparison of the first component and the second component.

[0092] In one embodiment, the signal output from each subpixel may represent a value containing color information (e.g., red (R), green (G), blue (B)) corresponding to the color filter of the unit pixel. The signal output from each subpixel may be stored in a memory (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2, or memory (330) of FIG. 3) in the form of raw data. In one embodiment, the signal output from each subpixel may be stored after undergoing a data processing process. For example, the data obtained from each subpixel may be stored to include a value obtained by interpolating the color values ​​of adjacent neighboring unit pixels with the output values ​​of red (R), green (G), and blue (B) contained in the unit pixel. Referring to FIG. 8, the first component may be indicated as 'L' and the second component may be indicated as 'R'.

[0093] FIG. 9 is a drawing illustrating an example of a plurality of image pixels according to one embodiment.

[0094] Referring to FIG. 9, a number of image pixels arranged in a Bayer pattern are shown. FIG. 9 shows a plurality of image pixels consisting of a first row to a fourth row (L1, L2, L3, L4) and a first column to a twelfth column (R1, R2, R3, R4, R5, R6, R7, R8, R9, R10, R11, R12). A pixel group identical to a 2 x 2 array pixel group formed by combining the first row (L1), the second row (L2), the first column (R1), and the second column (R2) can be repeatedly arranged in a first direction (e.g., Y direction) and a second direction (e.g., Z direction).

[0095] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can obtain phase difference information for each pixel from subpixels. A phase difference value including the phase difference information can be determined based on a luminance value detected through the subpixels. Values ​​for the first and second components of incident light obtained from the subpixels can be expressed as data (LY, RY) for luminance (Y).

[0096] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can detect correlation information by performing a correlation operation on phase difference data. The correlation operation can be performed on image pixels within a predetermined area. For example, the correlation operation can be performed on a plurality of image pixels within a predetermined range that are arranged in the same pixel row or column. Referring to FIG. 8, the correlation operation can be performed by reading out data for a plurality of image pixels in a second direction (e.g., horizontal direction (Y direction)). The electronic device can perform a correlation operation on the result of reading out a plurality of image pixels arranged in a first row (L1), a second row (L2), a third row (L3), and a fourth row (L4) in the second direction (e.g., horizontal direction (Y direction)). The electronic device can sequentially perform read operations for the first row (L1), the second row (L2), the third row (L3), and the fourth row (L4). The phase difference information of each pixel read from the first row (L1), the second row (L2), the third row (L3), and the fourth row (L4) can be output as phase difference data containing phase difference information using a line memory included in at least one of an image sensor (e.g., the image sensor (230) of FIG. 2) or an image signal processor (260).

[0097] The area indicated by the dotted line in FIG. 9 may indicate an area that serves as a unit for detecting values ​​(LY component, RY component) for acquiring phase difference information. However, the shape of the area is not limited thereto, and the area may be set for various numbers of pixels.

[0098] In one embodiment, an image frame output from an image sensor may include pixel data containing color information and phase difference data containing phase difference information. For example, the phase difference data may include first component data (e.g., 'LY' data) collected from a first component within a single pixel group and second component data (e.g., 'RY' data) collected from a second component within a single pixel group. The image sensor may collect the 'LY' data and the 'RY' data and output them as a single line (CL). An electronic device may perform a correlation operation for phase difference detection using the LY data and the RY data. FIG. 9 illustrates an example of outputting phase difference data based on two rows (e.g., L1, L2) for convenience of explanation, but is not limited thereto. For example, the phase difference data may be output based on four rows (L1, L2, L3, L4) or more rows.

[0099] FIG. 10 is a schematic diagram illustrating a method of performing correlation operations on data for phase difference detection by an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3).

[0100] In one embodiment, the total sum of the absolute values ​​of the difference between the first component data and the second component data may be referred to as the correlation value. Referring to FIG. 10, FIG. 10 illustrates an example of image data (1000) included in an image frame containing first component data (1010) and second component data (1020). An electronic device may obtain correlation information (1030) for a region of interest (1001) within the image data (1000). The electronic device may obtain correlation values ​​(1031, 1032, 1033, 1034, 1035, 1036) while shifting at least one of the first component data (1010) or the second component data (1020) in one direction or both directions.

[0101] Referring to FIG. 10, the electronic device can obtain a first correlation value (1031) by performing a correlation operation on the first data (1011) and the second data (1021). The first data (1011) may include data located within the region of interest (1001) among the first component data (1010). The second data (1021) may include data located within the region of interest (1001) among the second component data (1020).

[0102] In one embodiment, the electronic device may obtain a second correlation value (1032) by shifting the position of the first component data (1010) by one unit (e.g., 1 pixel) and performing a correlation operation on the third data (1012) and the second data (1021) located within the region of interest (1001) based on the shifted position. The electronic device may obtain a third correlation value (1033) by shifting the position of the first component data (1010) by two units (e.g., 2 pixels) and performing a correlation operation on the fourth data (1013) and the second data (1021) located within the region of interest (1001) based on the shifted position. The electronic device can obtain a fourth correlation value (1034) by shifting the position of the first component data (1010) by three units (e.g., 3 pixels) and performing a correlation operation on the fifth data (1014) and the second data (1021) located within the region of interest (1001) based on the shifted position. The electronic device can obtain a fifth correlation value (1035) by shifting the position of the first component data (1010) by four units (e.g., 4 pixels) and performing a correlation operation on the sixth data (1015) and the second data (1021) located within the region of interest (1001) based on the shifted position. The electronic device can obtain a sixth correlation value (1036) by shifting the position of the first component data (1010) by five units (e.g., 5 pixels) and performing a correlation operation on the seventh data (1016) and the second data (1021) located within the region of interest (1001) based on the shifted position.

[0103] In one embodiment, the electronic device can obtain phase difference information for a subject based on correlation information. For example, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can determine a focal position or depth value based on a position (e.g., +2) having the minimum value (e.g., the third correlation value (1033)) among the correlation values ​​(1031, 1032, 1033, 1034, 1035, 1036).

[0104] FIG. 10 may relate to a method for reading a signal from an image sensor having a 2PD pixel structure (500) of FIG. 5. In the case of an image sensor having a 4PD pixel structure (600) of FIG. 6 or a hexadeca pixel structure (700) of FIG. 7, in addition to phase difference detection in a first direction (e.g., horizontal direction (Y direction)), phase difference detection in a second direction (e.g., vertical direction (Z direction)) may also be performed.

[0105] FIG. 11 is a flowchart (1100) illustrating a process in which an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) according to one embodiment acquires depth information.

[0106] According to one embodiment, in operation 1110, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may acquire a first image frame containing first correlation information. The electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may acquire first data, at least a portion of which is acquired through a first subpixel of an image sensor (e.g., the image sensor (230) of FIG. 2). The first data may include a first component value among phase difference information. The electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may acquire second data, at least a portion of which is acquired through a second subpixel of an image sensor. The second data may include a second component value among phase difference information. An electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can obtain first correlation information through a correlation operation on first data and second data. The first correlation information may include correlation values ​​obtained for each of the regions into which the image frame is divided into multiple parts. For example, the first correlation information may include correlation values ​​calculated based on pixel values ​​for pixels included in the divided regions among the pixel arrays of the image sensor. For example, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can obtain first correlation information including correlation values ​​calculated for each of the regions divided into a 5 x 5 array.

[0107] According to one embodiment, in operation 1120, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may perform spatial filtering on the acquired correlation information. The electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may perform spatial filtering on the first correlation information to acquire second correlation information. The electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may correct the correlation value for one region within the first correlation information based on the correlation value for another region. For example, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may determine a weight for another region and apply the weight to the correlation value for the other region to determine the correlation value for the region of interest. An electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may determine weights based on the degree to which the correlation values ​​of other regions are associated with the correlation values ​​of the region of interest. For example, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may determine weights based on at least one of the distance between regions, the difference in pixel values, the difference in depth, the confidence value, the characteristics of the lens, or the presence or absence of saturated pixels. The pixel values ​​may include, for example, at least one of a value representing a pixel according to a color space (e.g., RGB value or YUV value), a color value, or a brightness value. The region of interest may be any one of a plurality of regions that divide the pixels included in the image sensor. For example, referring to FIG. 15, a plurality of regions (regions q1 to q 25 Spatial filtering of any one of the regions (region q13) can be performed, for example, based on the following mathematical formula 1.

[0108]

[0109] In mathematical formula 1, It can represent the correlation value of the region of interest to which spatial filtering has been applied. is region q from region of interest p. n It can represent a first weight determined based on the distance between regions up to. p may be coordinates representing the location of the region of interest. q n The coordinates may indicate the location of a region within the image. The first weight may have a larger value as the distance to the region of interest decreases. For example, based on a Gaussian distribution, the first weight may be determined such that the closer the region is to the region of interest, the higher the weight, and the further it is from the region of interest, the lower the weight. is the pixel value of the region of interest p (e.g., RGB value or YUV value) and area q n pixel values ​​(e.g., RGB values ​​or YUV values) A second weight determined based on the difference can be represented. The second weight may have a larger value as the difference in pixel values ​​becomes smaller. The pixel value of the area may, for example, be the average or median value of the pixel values ​​of the pixels included within the area among the pixel data. However, the method of determining the pixel value is not limited thereto. is the depth value D of the region of interest p. p and area q n depth value A third weight can be represented that is determined based on the difference. The third weight can have a larger value as the difference in depth values ​​decreases. is region q nA fourth weight can be represented, which is determined based on the confidence of the correlation value. The fourth weight may have a larger value as the confidence increases. In noisy scenes or low-contrast environments, the correlation value may not exhibit a clear V-shape. The confidence information may include information on whether the correlation information clearly exhibits a V-shape. is the region q included in the first correlation information. n The correlation value obtained through a correlation operation on the phase difference data obtained from can be represented. M can represent the number of divided regions (e.g., 25). Apart from Equation 1, in one embodiment, since accuracy decreases as it moves further away from the position corresponding to the central axis of the lens due to the characteristics of the lens, the electronic device may be configured to apply a higher weight to regions closer to the center of the lens.

[0110] According to one embodiment, in operation 1130, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) may perform temporal filtering on correlation information. The electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) may perform temporal filtering by blending the second correlation information with third correlation information for at least one second image frame. The third correlation information may be obtained by performing spatial filtering on the correlation information included in the second image frame. The electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) may determine weights for blending the correlation information. An electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can determine weights based on the similarity between the region of interest of the first image frame and each region of the second image frame, or the similarity between the first image frame and the second image frame.

[0111] For example, when a fourth correlation information is obtained by blending the correlation information of a first image frame with the correlation information of two second image frames, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can calculate the fourth correlation information based on the following mathematical formula 2.

[0112]

[0113] In mathematical formula 2, can represent the fourth correlation information for the Nth image frame. α can represent the weight for the correlation information of the N-2nd image frame. can represent correlation information for the region of interest of the N-2nd image frame. β can represent the weight for the correlation information of the N-1st image frame. can represent correlation information for the region of interest of the N-1th image frame. γ can represent second correlation information for the Nth image frame. For example, may be obtained by performing spatial filtering on the correlation information for the N-1th image frame based on Equation 1. For example, may be obtained by performing spatial filtering based on Equation 1 on the correlation information for the N-1th image frame. For example, α may be determined to have a value smaller than β. For example, β may be determined to have a value smaller than γ. The sum of α, β, and γ may have a specified value (e.g., 1).

[0114] In one embodiment, as a correlation value used for time filtering, a SAD indicating a correlation value for a specific frame TF It can be defined as shown in mathematical formula 3 below.

[0115]

[0116] In the above mathematical formula 3, is a correlation value used for spatial filtering, of Equation 1 It can correspond to.

[0117] According to one embodiment, in operation 1140, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) can determine depth information based on correlation information obtained by performing spatial filtering and temporal filtering. For example, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) can determine a depth value for a subject based on a phase difference value that causes the SAD value to have a minimum value based on the fourth correlation information. For example, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) may drive an actuator that moves a lens to adjust the focus of a camera based on the phase difference value.

[0118] In the present disclosure, the order of operations of the electronic devices (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) shown in the flowcharts is shown for convenience of explanation and the order of operations shown in the flowcharts may be changed. For example, an operation shown later may be performed simultaneously with an operation shown earlier, or may be performed before an operation shown earlier.

[0119] FIG. 12 illustrates a process in which an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) outputs correlation information based on a finite impulse filter (FIR) structure.

[0120] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may input correlation information of the Nth image frame (1200) obtained through a camera (e.g., the camera module (180) of FIG. 1, the camera (380) of FIG. 3)) into a spatial filter (1210). The electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may input the output of the spatial filter (1210) regarding the correlation information of the Nth image frame (1200) into a temporal filter (1230).

[0121] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may input correlation information of the N-1st image frame (1201) previously output to the Nth image frame (1200) into a spatial filter (1211). The electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may perform an operation (1221) to determine whether the motion information of the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) associated with the time when the N-1st image frame (1201) was acquired is smaller than a threshold. Motion information may be information obtained through an acceleration sensor (e.g., sensor module (176) of FIG. 1) included in an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) and stored in a memory (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2, memory (330) of FIG. 3) in association with the N-1th image frame (1201). If the motion information is above a threshold, the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) may discard the correlation information for the N-1th image frame (or the N-1th image frame and the image frame prior thereto) without inputting it into the time filter (1230). When the motion information is smaller than a threshold, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may input the output of the spatial filter (1211) to the temporal filter (1230). In one embodiment, the electronic device may be configured to omit the operation (1221) and input the output of the spatial filter (1211) to the temporal filter (1230) regardless of the magnitude of the motion. Although FIG. 12 is illustrated as the operation (1221) being performed after the operation (1211), the electronic device may perform the operation (1221) before the operation (1211) and determine whether to use the correlation information of the N-1th image frame (1201).

[0122] In one embodiment, if there is a result value obtained by applying a spatial filter (1211) to the correlation information of the N-1th image frame (1201) previously, the operation (1211) may be omitted.

[0123] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may input correlation information of the N-2nd image frame (1202) previously output to the N-1st image frame (1201) into a spatial filter (1212). The electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may perform an operation (1222) to determine whether motion information of the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) associated with the time at which the N-2nd image frame (1202) was acquired is smaller than a threshold. If the motion information is greater than or equal to the threshold, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may discard the correlation information for the N-2nd image frame without inputting it into a time filter (1230). When the motion information is smaller than a threshold, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may input the output of the spatial filter (1212) to the temporal filter (1230). In one embodiment, the electronic device may be configured to omit the operation (1222) and input the output of the spatial filter (1212) to the temporal filter (1230) regardless of the magnitude of the motion. Although FIG. 12 is illustrated as the operation (1222) being performed after the operation (1212), the electronic device may perform the operation (1222) before the operation (1212) and determine whether to use the correlation information of the N-2nd image frame (1202).

[0124] In one embodiment, if there is a result value obtained by applying a spatial filter (1212) to the correlation information of the N-2nd image frame (1202) previously, the operation (1212) may be omitted.

[0125] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may perform a time filter (1230) that blends input correlation information. The electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may obtain corrected correlation information as the output (1240) of the time filter (1230).

[0126] FIG. 13 illustrates a process in which an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) outputs correlation information based on an infinite impulse filter (IIR) structure.

[0127] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may be configured to output correlation information based on an IIR structure. The electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may input correlation information contained in the Nth image frame (1300) to a spatial filter (1310). The electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may input correlation information (1350) blended with correlation information up to the N-1th image frame to a temporal filter (1330). For example, the blended correlation information (1350) may include fifth correlation information obtained by blending the correlation information of a third image frame (e.g., the N-2nd image frame (1202) of FIG. 12) obtained prior to the acquisition of a second image frame (e.g., the N-1st image frame (1201) of FIG. 12) with the correlation information of the second image frame. For example, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may obtain fifth correlation information by blending the result of applying a spatial filter to the correlation information of the second image frame and the correlation information of the third image frame, respectively, or by applying a spatial filter to the result of blending the correlation information of the second image frame and the correlation information of the third image frame. However, it is not limited thereto. In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can determine whether motion information associated with the N-1th image frame is smaller than a threshold. If the motion information is greater than or equal to the threshold, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may omit the operation of inputting blended correlation information (1350) to a time filter (1330).

[0128] In one embodiment, the electronic device can obtain the output (1340) of a time filter (1330) as corrected correlation information for the Nth image frame (1300) based on correlation information (1350) in which correlation information included in the Nth image frame (1300) and correlation information up to the N-1th image frame are blended.

[0129] FIG. 14 is a block diagram illustrating an example of a spatial filter (1410) according to one embodiment (e.g., the spatial filter (1210) of FIG. 12, the spatial filter (1211), the spatial filter (1212) or the spatial filter (1310) of FIG. 13).

[0130] In one embodiment, the spatial filter (1410) can obtain a resized image (1423) by adjusting the size of an image (1421) included in an image frame (e.g., the Nth image (1200), the N-1st image (1201), the N-2nd image (12020) of FIG. 12, or the Nth image (1300) of FIG. 13). Since the correlation value and the pixels of the image (1421) do not correspond one-to-one and a single correlation value may correspond to a region containing multiple pixels, the size of the correlation information (1441) and the size of the image (1241) may be different from each other. The spatial filter (1410) can adjust the size of the image (1421) to a size corresponding to the correlation information (1441) in order to use the pixel values ​​included in the image (1421). For example, the spatial filter (1410) can obtain a resized image (1423) in which a single pixel is used as a representative value representing the pixel values ​​of pixels within a region of the image (1421). The number of correlation values ​​included in the correlation information (1441) may correspond to the number of regions of interest. The spatial filter (1410) can resize the image to the size of the region of interest in order to use the correlation values ​​for similar regions of the image. For example, when obtaining correlation information for a region of interest having a size of 19 x 15 from an image having a size of 4000 x 3000, the spatial filter (1410) can resize the image having a size of 4000 x 3000 to a size of 19 x 15. The spatial filter (1410) can determine the similarity to the image of the region of interest based on the resized image.

[0131] In one embodiment, the spatial filter (1410) may perform an operation 1450 of determining a weight for performing spatial filtering based on depth information (1443) determined based on the resized image (1423), confidence (1430), and correlation information (1441). The spatial filter (1410) may perform an operation 1460 of correcting the correlation information (1441) based on the determined weight. The spatial filter may output the corrected correlation information based on the image (1421), confidence (1430), and the determined weight.

[0132] FIG. 15 is a diagram illustrating an example of determining weights based on the distance between regions in one embodiment.

[0133] FIG. 15 illustrates an example of obtaining corrected correlation information for a region of interest (1510) based on correlation information for regions divided into a 5 x 5 array. Region q 13 In the case of this region of interest (1510), the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) is regions q1 to q 25 The weights for each region q 13 It can be determined based on the distance from. Since correlation information for a region closer to the region of interest may be correlation information for a subject located at a depth similar to that of the subject in the region of interest, a higher weight may be applied to correlation information for a close region.

[0134] For example, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may determine a weight for region q1 (1511) based on the distance (1531) between the center point (1520) of the region of interest (1510) and the center point (1521) of region q1 (1511). The electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may determine a weight for region q 14The weight for (1512) is given from the center point (1520) of the region of interest (1510) to the region q 14 Area q based on the distance (1533) between the center point (1522) of (1512) 14 We can determine the weight for (1512). q 14 The weight for (1512) can be set higher than the weight for q1 (1511).

[0135] FIG. 16 is a diagram illustrating an example of determining weights based on the difference in pixel values ​​in one embodiment.

[0136] In one embodiment, it may be difficult to determine the similarity between each region based solely on correlation information. Therefore, the electronic device may determine a weight for the correlation information of each region based on the color information (e.g., pixel value) of the region corresponding to the correlation information within the color image. For example, since a region in which a subject with a color similar to the region of interest is captured may contain a subject identical or similar to the subject of interest, a higher weight may be assigned to regions with colors similar to the region of interest.

[0137] In one embodiment, when a subject of the first color is photographed in the region of interest (1610), a subject of the first color is photographed in the first region (1611), and a subject of the second color is photographed in the second region (1612), an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may give a higher weight to the correlation information of the first region (1611) than to the second region (1612).

[0138] FIG. 17 is a diagram illustrating an example of determining weights based on depth difference in one embodiment.

[0139] In one embodiment, when near and far subjects are mixed within the range in which the image is captured, accurate depth information may not be obtained when weights are assigned without information about depth. An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may assign a high weight to an area having a depth value similar to the depth value determined for the region of interest (1710).

[0140] Referring to FIG. 17, when capturing images of a field of view (1700) in which a near subject (1701) and a far subject (1702) are mixed, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may assign a higher weight to an area that is estimated to have a depth similar to that of the subject captured in the region of interest (1710). For example, a higher weight may be assigned to a first area (1711) in which the same subject as the region of interest (1710) is captured and a similar distance value is output, compared to a second area (1712) in which a far subject is captured.

[0141] FIG. 18 is a block diagram illustrating an example of a time filter according to one embodiment (e.g., the time filter of FIG. 12 (1230), the time filter of FIG. 13 (1330)).

[0142] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may perform an operation 1560 of determining time filter weights (e.g., α, β, and γ of Equation 2) for a current image frame (1801) and at least one previous image frame (1802). At least one previous image frame (1802) may have the same lens position value as the current image frame (1801).

[0143] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may determine a time filter weight based on the result of a comparison between image frames (1801, 1802). For example, the electronic device may determine depth information (1821, 1822), reliability (1831, 1832), and correlation information (e.g., of Equation 1) for the current image frame (1801) and at least one previous image frame (1802). A time filter weight can be determined based on at least one of )(1841, 1842) or pixel values ​​(e.g., RGB values) (1851, 1852). The electronic device can determine the similarity between image frames (1801, 1802) based on the result of comparing depth information (1821, 1822), correlation information (1841, 1842), or pixel values ​​(e.g., RGB values) (1851, 1852). For example, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may assign a higher time filter weight as the similarity between the current image frame (1801) and at least one image frame (1802) is higher. Additionally, the electronic device may assign a higher time filter weight as the reliability (1832) of the previous image frame (1802) is higher.

[0144] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can blend (1807) the correlation information (1841) of the current image frame (1801) with the correlation information (1842) of at least one previous image frame (1802) based on a determined weight.

[0145] FIG. 19 is a diagram illustrating an example of a method for determining the weights of a time filter (e.g., the time filter (1230) of FIG. 12, the time filter (1330) of FIG. 13) according to one embodiment.

[0146] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can determine the similarity between the region of interest (1910) of the current image frame (1901) and the regions of the previous image frame (1902) (e.g., regions q1 to q25 of the previous image frame (1902)). The electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3)) can blend the correlation values ​​for each region with the correlation value of the region of interest (1901) based on the similarity for each region.

[0147] FIG. 20 is a diagram illustrating an example of a method for determining the weights of a time filter (e.g., the time filter (1230) of FIG. 12, the time filter (1330) of FIG. 13) according to one embodiment.

[0148] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can determine the similarity between a current image frame (2001) and a previous image frame (2002). For example, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can compare each region of the current image frame (2001) with a corresponding region of the previous image frame (2002). The electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3)) can blend the correlation value of the region of interest (2011) with the correlation value of the region (2012) within the previous image frame (2002) corresponding to the region of interest (2011), based on weights according to similarity.

[0149] FIG. 21 illustrates an example of correlation information obtained by an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3).

[0150] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) can obtain corrected correlation information (2102) that can obtain a more accurate phase difference value by applying a spatial filter and a temporal filter to the correlation information (2101) before correction.

[0151] Referring to FIG. 21, the correlation value of a low-contrast subject may have a low slope within the interval (2111) where the correlation value changes. In this case, the electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may have difficulty accurately determining the phase difference value (2121) where the difference is smallest.

[0152] Referring to FIG. 21, the correlation information (2102) corrected using a spatial filter and a temporal filter changes relatively steeply in value within the interval (2112) compared to the interval (2111), so the phase difference value (2122) can be obtained more clearly.

[0153] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) may include a camera (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2, the camera (380) of FIG. 3), at least one processor (e.g., the processor (120) of FIG. 1, the image signal processor (260) of FIG. 2, the at least one processor (320) of FIG. 3), and a memory (e.g., the memory (130) of FIG. 1, the memory (250) of FIG. 2, the memory (330) of FIG. 3). The camera (e.g., the camera module (180) of FIG. 1, the camera module (180) of FIG. 2, the camera (380) of FIG. 3)) may include an image sensor (e.g., the image sensor (230) of FIG. 2). At least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) may include processing circuitry. Memory (e.g., memory (130) of FIG. 1, memory (250) of FIG. 2, memory (330) of FIG. 3) may store instructions. The image sensor (e.g., image sensor (230) of FIG. 2) may include a microlens array (e.g., microlens array (411) of FIG. 4) and unit pixels corresponding to the microlens array (e.g., unit pixel (510) of FIG. 5, unit pixel (610) of FIG. 6, unit pixel (710) of FIG. 7). The above unit pixels (e.g., unit pixel (510) of FIG. 5, unit pixel (610) of FIG. 6, unit pixel (710) of FIG. 7)) may include a first subpixel (e.g., subpixel (531) of FIG. 5, subpixel (631) of FIG. 6, subpixel (633) of FIG. 7, subpixel (731) of FIG. 7, subpixel (733)) and a second subpixel (e.g., subpixel (532) of FIG. 5, subpixel (632) of FIG. 6, subpixel (634) of FIG. 7, subpixel (732) of FIG. 7).The above instructions are executed individually or collectively by the at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) so as to provide a first image frame (e.g., image of FIG. 12) comprising first correlation information between first data obtained by the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) through the first subpixel (e.g., subpixel (531) of FIG. 5, subpixel (631) of FIG. 6, subpixel (633) of FIG. 7, subpixel (731) of FIG. 7) and second data obtained through the second subpixel (e.g., subpixel (532) of FIG. 5, subpixel (632) of FIG. 6, subpixel (634) of FIG. 7, subpixel (732) of FIG. 7) A frame (1200), an image frame (1300) of FIG. 13 can be obtained through the image sensor (e.g., image sensor (230) of FIG. 2).The above instructions are executed individually or collectively by the at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) so that the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) applies at least one weight to correlation values ​​for a plurality of regions (e.g., regions q1 to q25 of FIG. 15 to 16, FIG. 19, or FIG. 20) that divide the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13)) based on the first correlation information, and a spatial filter (e.g., spatial filter (1210) of FIG. 12, spatial filter (1310) of FIG. 13, spatial filter (1410) of FIG. 14) of FIG. 14 By performing the above, it is possible to obtain second correlation information for the region of interest. The above instructions may be executed individually or collectively by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) so that the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3)) may obtain fourth correlation information for the region of interest by performing a temporal filter (e.g., temporal filter (1230) of FIG. 12, temporal filter (1330) of FIG. 13)) that blends the second correlation information with third correlation information corresponding to the second image frame obtained at a different time from the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13).The above instructions may be executed individually or collectively by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) so that the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) determines depth information based on the fourth correlation information.

[0154] In one embodiment, the instructions may be executed individually or collectively by the at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) so that the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) determines the at least one weight based on at least one of the distance between regions, difference in pixel values, difference in depth, or confidence value. The above instructions may be executed individually or collectively by the at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) to enable the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to perform spatial filtering (e.g., spatial filter (1210) of FIG. 12, spatial filter (1310) of FIG. 13, spatial filter (1410) of FIG. 14)) based on the at least one determined weight. The distance between regions may include a distance value from the region of interest to one of the plurality of regions (e.g., regions q1 to q25 of FIG. 15 to 16, FIG. 19, or FIG. 20). The difference in pixel values ​​may include the difference between the pixel value for the region of interest and the pixel value for one of the plurality of regions (e.g., regions q1 to q25 of FIG. 15 to 16, FIG. 19, or FIG. 20). The depth difference may indicate the difference between the depth value for the region of interest and the depth value for one of the plurality of regions (e.g., regions q1 to q25 of FIG. 15 to 16, FIG. 19, or FIG. 20). The reliability value may indicate the reliability of the correlation value for one of the plurality of regions (e.g., regions q1 to q25 of FIG. 15 to 16, FIG. 19, or FIG. 20).

[0155] In one embodiment, the instructions may be executed individually or collectively by the at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) so that the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) acquires the second image frame before acquiring the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13). The above instructions may be executed individually or collectively by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) so that the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) obtains the third correlation information by performing spatial filtering (e.g., spatial filter (1210) of FIG. 12, spatial filter (1310) of FIG. 13, spatial filter (1410) of FIG. 14)) on the second image frame.

[0156] In one embodiment, the instructions may be executed individually or collectively by the at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) so that the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) performs the time filter (e.g., time filter (1230) of FIG. 12, time filter (1330) of FIG. 13)) by applying a first weight to the second correlation information and applying a second weight smaller than the first weight to the third correlation information.

[0157] In one embodiment, the instructions may be executed individually or collectively by the at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) so that the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) can obtain fifth correlation information by blending the correlation information obtained from the second image frame with the correlation information of the third image frame obtained prior to the second image frame. The above instructions may be executed individually or collectively by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) so that the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) obtains the third correlation information by performing a spatial filter operation on the fifth correlation information.

[0158] In one embodiment, the instructions may be executed individually or collectively by the at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) to enable the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to acquire motion information associated with the second image frame. The instructions may be executed individually or collectively by the at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) to enable the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to determine whether to blend the third correlation information with the second correlation information based on the motion information.

[0159] In one embodiment, the instructions may be executed individually or collectively by the at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) so that the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) determines the similarity between the region of interest of the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13) and the regions included in the second image frame, respectively. The above instructions may be executed individually or collectively by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) so that the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) performs the time filter (e.g., time filter (1230) of FIG. 12, time filter (1330) of FIG. 13)) by determining the fourth correlation information using the correlation value for the region included in the second image frame based on the similarity.

[0160] In one embodiment, the instructions may be executed individually or collectively by the at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) so that the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) compares regions included in the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13)) with regions included in the second image frame, respectively, to determine the similarity between the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13)) and the second image frame. The above instructions may be executed individually or collectively by at least one processor (e.g., processor (120) of FIG. 1, image signal processor (260) of FIG. 2, at least one processor (320) of FIG. 3) to enable the electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) to perform the time filter operation by determining the fourth correlation value using the correlation value for the region corresponding to the region of interest among the regions included in the second image frame based on the similarity.

[0161] A method of operating an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (101) of FIG. 3) including an image sensor (e.g., image sensor (230) of FIG. 2) according to one embodiment, comprising: a first image frame (e.g., image frame (1200) of FIG. 12) including first correlation information between first data obtained through a first subpixel (e.g., subpixel (531) of FIG. 5, subpixel (631) of FIG. 6, subpixel (633) of FIG. 7, subpixel (731) of FIG. 7, subpixel (733)) of the image sensor (e.g., image sensor (230) of FIG. 2) and second data obtained through a second subpixel (e.g., subpixel (532) of FIG. 5, subpixel (632) of FIG. 6, subpixel (634) of FIG. 7, subpixel (732) of FIG. 7), subpixel (734) of FIG. 7), The method may include the operation of acquiring an image frame (1300) of 13 through the image sensor (e.g., image sensor (230) of FIG. 2). The method may include the operation of acquiring second correlation information for a region of interest by performing a spatial filter (e.g., spatial filter (1210) of FIG. 12, spatial filter (1310) of FIG. 13, spatial filter (1410) of FIG. 14)) which applies at least one weight to correlation values ​​for a plurality of regions (e.g., regions q1 to q25 of FIG. 15 to 16, FIG. 19, or FIG. 20) that divide the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13) based on the first correlation information.The above method may include an operation of obtaining fourth correlation information for the region of interest by performing a temporal filter (e.g., temporal filter (1230) of FIG. 12, temporal filter (1330) of FIG. 13)) that blends the second correlation information with third correlation information corresponding to the second image frame obtained at a different time from the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13). The above method may include an operation of determining depth information based on the fourth correlation information.

[0162] In one embodiment, the operation of obtaining the second correlation information may include the operation of determining the at least one weight based on at least one of the distance between regions, the difference in pixel values, the difference in depth, or the confidence value. In one embodiment, the operation of obtaining the second correlation information may include the operation of performing spatial filtering (e.g., the spatial filter (1210) of FIG. 12, the spatial filter (1310) of FIG. 13, the spatial filter (1410) of FIG. 14) based on the determined at least one weight. The distance between regions may include a distance value from a region of interest to one of the plurality of regions (e.g., regions q1 to q25 of FIG. 15 to 16, FIG. 19, or FIG. 20). The difference in pixel values ​​may include a difference value between the pixel value for the region of interest and the pixel value for one of the plurality of regions (e.g., regions q1 to q25 of FIG. 15 to 16, FIG. 19, or FIG. 20). The depth difference may indicate the difference between the depth value for the region of interest and the depth value for one of the plurality of regions (e.g., regions q1 to q25 of FIG. 15 to 16, FIG. 19, or FIG. 20). The reliability value may indicate the reliability of the correlation value for one of the plurality of regions (e.g., regions q1 to q25 of FIG. 15 to 16, FIG. 19, or FIG. 20).

[0163] In one embodiment, the method may further include the operation of acquiring the second image frame before acquiring the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13). The method may further include the operation of acquiring the third correlation information by performing spatial filtering on the second image frame (e.g., spatial filter (1210) of FIG. 12, spatial filter (1310) of FIG. 13, spatial filter (1410) of FIG. 14).

[0164] In one embodiment, the operation of obtaining the fourth correlation information may include applying a first weight to the second correlation information and applying a second weight smaller than the first weight to the third correlation information.

[0165] In one embodiment, the method may further include an operation of obtaining fifth correlation information by blending correlation information obtained from the second image frame with correlation information of a third image frame obtained prior to the second image frame. The operation of obtaining third correlation information may include an operation of obtaining third correlation information by performing a spatial filter operation on the fifth correlation information.

[0166] In one embodiment, the method may further include an operation of acquiring motion information associated with the second image frame. The operation of acquiring the fourth correlation information may include an operation of determining whether to blend the third correlation information with the second correlation information based on the motion information.

[0167] In one embodiment, the operation of acquiring the fourth correlation information may include determining the similarity between the region of interest of the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13) and the regions included in the second image frame, respectively. The operation of acquiring the fourth correlation information may further include the operation of performing the time filter (e.g., time filter (1230) of FIG. 12, time filter (1330) of FIG. 13) by determining the fourth correlation information using the correlation value for the regions included in the second image frame based on the similarity.

[0168] In one embodiment, the operation of obtaining the fourth correlation information may include an operation of determining the similarity between the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13) and the second image frame by comparing the regions included in the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13) and the second image frame, respectively. The operation of obtaining the fourth correlation information may further include an operation of performing the time filter operation by determining the fourth correlation value using the correlation value for the region corresponding to the region of interest among the regions included in the second image frame based on the similarity.

[0169] In one embodiment, a computer-readable non-transient recording medium may have a computer program recorded thereon that causes an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (101) of FIG. 3) to perform at least one operation when executed. The above at least one operation may include the operation of acquiring a first image frame (e.g., image frame of FIG. 12 (1200), image frame of FIG. 13 (1300)) through the image sensor (e.g., image sensor of FIG. 2 (230)) which includes first data acquired through a first subpixel (e.g., subpixel of FIG. 5 (531), subpixel of FIG. 6 (631), subpixel of FIG. 7 (633), subpixel of FIG. 7 (731), subpixel of FIG. 7 (733))) of the image sensor (e.g., image sensor of FIG. 2 (230)). The first image frame includes first correlation information between the first data acquired through a first subpixel (e.g., subpixel of FIG. 5 (532), subpixel of FIG. 6 (632), subpixel of FIG. 6 (634), subpixel of FIG. 7 (732), subpixel of FIG. 7 (734)). The above at least one operation may include an operation of obtaining second correlation information for a region of interest by performing a spatial filter (e.g., spatial filter (1210) of FIG. 12, spatial filter (1310) of FIG. 13, spatial filter (1410) of FIG. 14)) which applies at least one weight to correlation values ​​for a plurality of regions (e.g., regions q1 to q25 of FIG. 15 to 16, FIG. 19, or FIG. 20) that divide the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13) based on the first correlation information.The at least one operation may include an operation of obtaining fourth correlation information for the region of interest by performing a temporal filter (e.g., temporal filter (1230) of FIG. 12, temporal filter (1330) of FIG. 13)) that blends the second correlation information with third correlation information corresponding to the second image frame obtained at a different time from the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13). The at least one operation may include an operation of determining depth information based on the fourth correlation information.

[0170] In one embodiment, the operation of obtaining the second correlation information may include the operation of determining the at least one weight based on at least one of the distance between regions, the difference in pixel values, the difference in depth, or the confidence value. The operation of obtaining the second correlation information may include the operation of performing spatial filtering (e.g., the spatial filter (1210) of FIG. 12, the spatial filter (1310) of FIG. 13, the spatial filter (1410) of FIG. 14) based on the determined at least one weight. The distance between regions may include a distance value from a region of interest to one of the plurality of regions (e.g., regions q1 to q25 of FIG. 15 to 16, FIG. 19, or FIG. 20). The difference in pixel values ​​may include a difference value between the pixel value for the region of interest and the pixel value for one of the plurality of regions (e.g., regions q1 to q25 of FIG. 15 to 16, FIG. 19, or FIG. 20). The depth difference may indicate the difference between the depth value for the region of interest and the depth value for one of the plurality of regions (e.g., regions q1 to q25 of FIG. 15 to 16, FIG. 19, or FIG. 20). The reliability value may indicate the reliability of the correlation value for one of the plurality of regions (e.g., regions q1 to q25 of FIG. 15 to 16, FIG. 19, or FIG. 20).

[0171] In one embodiment, the operation of acquiring the fourth correlation information may include determining the similarity between the region of interest of the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13) and the regions included in the second image frame, respectively. The operation of acquiring the fourth correlation information may include performing the time filter (e.g., time filter (1230) of FIG. 12, time filter (1330) of FIG. 13) by determining the fourth correlation information using the correlation value for the region included in the second image frame based on the similarity.

[0172] In one embodiment, the operation of obtaining the fourth correlation information may include an operation of determining the similarity between the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13) and the second image frame by comparing the regions included in the first image frame (e.g., image frame (1200) of FIG. 12, image frame (1300) of FIG. 13) and the second image frame, respectively. The operation of obtaining the fourth correlation information may include an operation of performing the time filter operation by determining the fourth correlation value using the correlation value for the region corresponding to the region of interest among the regions included in the second image frame based on the similarity.

[0173] In the present disclosure, an electronic device and a method of operation thereof according to various embodiments can improve the accuracy of phase difference data by applying a filter to the difference value of the data.

[0174] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description of the present disclosure.

[0175] Methods according to the claims or embodiments described in the specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0176] When implemented in software, a computer-readable storage medium may be provided for storing one or more programs (software modules). One or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. One or more programs include instructions that cause the electronic device to execute methods according to the claims or embodiments described in the specification of this disclosure.

[0177] In the present disclosure, the function or operation performed by an electronic device may be performed by one or more processors executing one or more instructions stored in memory. The function or operation of the electronic device mentioned in the present disclosure may be performed by a single processor executing one or more instructions, or by a combination of multiple processors executing one or more instructions. A processor mentioned in the present disclosure is understood to include a circuit for performing operations or controlling other components of the electronic device. For example, the one or more processors may include a central processing unit (CPU), a micro-processor unit (MPU), an application processor (AP), a communication processor (CP), a neural processing unit (NPU), a system on chip (SoC), an integrated circuit (IC), or an application-specific integrated circuit (ASIC) configured to execute one or more instructions. The one or more processors may be configured to perform the operation of the electronic device described above.

[0178] In the present disclosure, a program (software module, software) may be stored in a random access memory, a non-volatile memory including flash memory, a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic disc storage device, a compact disc-ROM (CD-ROM), digital versatile discs (DVDs), or other forms of optical storage devices, or a magnetic cassette. Alternatively, it may be stored in a memory composed of some or all of these. The memory may be composed of a single storage medium or a combination of multiple storage media. The one or more instructions may be stored in a single storage medium or distributed across multiple storage media.

[0179] Additionally, the above program may be stored on an attachable storage device that can be accessed via a communication network such as the Internet, Intranet, LAN (local area network), WLAN (wide LAN), or SAN (storage area network), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure through an external port. Additionally, a separate storage device on a communication network may be connected to a device performing an embodiment of the present disclosure.

[0180] In the specific embodiments of the present disclosure described above, the components included in the disclosure are expressed in a singular or plural form according to the specific embodiments presented. However, the singular or plural expression is selected to suit the situation presented for convenience of explanation, and the present disclosure is not limited to singular or plural components; even if a component is expressed in the plural form, it may be composed of a singular form, and even if a component is expressed in the singular form, it may be composed of a plural form.

[0181] Additionally, in the present disclosure, terms such as “part,” “module,” etc. may be a hardware component, such as a processor or circuit, and / or a software component executed by a hardware component, such as a processor.

[0182] "Parts" and "modules" may be implemented by a program that is stored on an addressable storage medium and can be executed by a processor. For example, "parts" and "modules" may be implemented by components such as software components, object-oriented software components, class components, and task components, as well as by processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0183] The specific embodiments described in this disclosure are merely examples and do not limit the scope of this disclosure in any way. For the sake of brevity, descriptions of prior electronic configurations, control systems, software, and other functional aspects of said systems may be omitted.

[0184] Additionally, in the present disclosure, “comprising at least one of a, b, or c” may mean “comprising only a, comprising only b, comprising only c, or comprising a combination of two or more (comprising a and b, comprising b and c, comprising a and c, or comprising all of a, b, and c).”

[0185] Meanwhile, although specific embodiments have been described in the detailed description of the present disclosure, it is understood that various modifications are possible within the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof.

[0186] In the present disclosure, the term “if” will be understood, depending on the context, to mean “when, upon,” “in response to a decision,” or “in response to a detection.” Similarly, “when decided to,” or “when [mentioned condition or event] is detected” will be understood, optionally, to mean “when decided,” or “in response to a decision,” “when [mentioned condition or event] is detected,” or “in response to a detection.”

[0187] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit (or processing circuit) may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0188] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0189] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or several hardware combined, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by an app store that distributes applications or a site or server that supplies or distributes various other software.

[0190] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

Claims

1. In an electronic device, A camera including an image sensor; At least one processor including processing circuitry; and It includes memory for storing instructions, The image sensor above includes a microlens array and unit pixels corresponding to the microlens array, and The above unit pixels include a first subpixel and a second subpixel, and The above instructions are executed individually or collectively by the at least one processor, and the electronic device: A first image frame including first correlation information between first data obtained through the first subpixel and second data obtained through the second subpixel is obtained through the image sensor, and Based on the first correlation information above, second correlation information for a region of interest is obtained by performing a spatial filter that applies at least one weight to the correlation values ​​of a plurality of regions that divide the first image frame, and Fourth correlation information for the region of interest is obtained by performing a temporal filter that blends the second correlation information with third correlation information corresponding to a second image frame acquired at a different time from the first image frame. An electronic device that determines depth information based on the above-mentioned fourth correlation information.

2. In Claim 1, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: Determining at least one weight based on at least one of the distance between regions, the difference in pixel values, the difference in depth, or the confidence value, and The spatial filtering is performed based on at least one weight determined above, and The distance between the above regions includes a distance value from the region of interest to one of the plurality of regions, and The difference in the pixel values ​​above includes the difference between the pixel value for the region of interest and the pixel value for one of the plurality of regions, and The above depth difference indicates the difference between the depth value for the region of interest and the depth value for one of the plurality of regions, An electronic device in which the above reliability value indicates the reliability of the correlation value of one of the plurality of regions.

3. In Claim 1, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: Acquire the second image frame before acquiring the first image frame, and An electronic device that obtains the third correlation information by performing spatial filtering on the second image frame.

4. In Claim 3, The above instructions are executed individually or collectively by the at least one processor, so that the electronic device performs the time filtering by applying a first weight to the second correlation information and applying a second weight smaller than the first weight to the third correlation information.

5. In Claim 1, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: A fifth correlation information is obtained by blending the correlation information obtained from the second image frame with the correlation information of the third image frame obtained prior to the second image frame, and An electronic device that obtains the third correlation information by performing a spatial filter operation on the fifth correlation information.

6. In Claim 1, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: Acquiring motion information associated with the above second image frame, and An electronic device that determines whether to blend the third correlation information with the second correlation information based on the above movement information.

7. In Claim 1, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: The similarity between the region of interest of the first image frame and the regions included in the second image frame is determined, respectively, and An electronic device that performs the time filtering by determining the fourth correlation information using the correlation value for the region included in the second image frame based on the similarity.

8. In Claim 1, The above instructions are executed individually or collectively by the at least one processor, and the electronic device: The similarity between the first image frame and the second image frame is determined by comparing the regions included in the first image frame and the regions included in the second image frame, respectively. An electronic device that performs the time filter operation by determining the fourth correlation value using the correlation value for the region corresponding to the region of interest among the regions included in the second image frame based on the similarity.

9. A method of operating an electronic device including an image sensor, The operation of acquiring a first image frame through the image sensor, the first image frame including first correlation information between first data acquired through a first subpixel of the image sensor and second data acquired through a second subpixel of the image sensor; An operation to obtain second correlation information for a region of interest by performing spatial filtering that applies at least one weight to correlation values ​​for a plurality of regions divided from the first image frame, based on the first correlation information; The operation of obtaining fourth correlation information for the region of interest by performing a temporal filter that blends the second correlation information with third correlation information corresponding to a second image frame obtained at a different time from the first image frame; and A method comprising an operation to determine depth information based on the above-mentioned fourth correlation information.

10. In Claim 9, The operation of obtaining the above second correlation information is: An operation of determining at least one weight based on at least one of the distance between regions, the difference in pixel values, the difference in depth, or the confidence value, and The operation includes performing the spatial filtering based on at least one weight determined above, and The distance between the above regions includes a distance value from the region of interest to one of the plurality of regions, and The difference in the pixel values ​​above includes the difference between the pixel value for the region of interest and the pixel value for one of the plurality of regions, and The above depth difference indicates the difference between the depth value for the region of interest and the depth value for one of the plurality of regions, A method in which the above reliability value indicates the reliability of the correlation value of one of the plurality of regions.

11. In Claim 9, The operation of acquiring the second image frame prior to acquiring the first image frame; and A method further comprising the operation of obtaining the third correlation information by performing spatial filtering on the second image frame.

12. In Claim 11, The operation of obtaining the above-mentioned fourth correlation information is: A method comprising applying a first weight to the second correlation information and applying a second weight smaller than the first weight to the third correlation information.

13. In Claim 9, The method further includes the operation of obtaining fifth correlation information by blending the correlation information obtained from the second image frame and the correlation information of the third image frame obtained prior to the second image frame. A method in which the operation of obtaining the third correlation information includes the operation of obtaining the third correlation information by performing a spatial filter operation on the fifth correlation information.

14. In Claim 9, It further includes an operation to acquire motion information associated with the second image frame, and The operation of acquiring the fourth correlation information includes an operation of determining whether to blend the third correlation information with the second correlation information based on the movement information.

15. In a computer-readable non-transient recording medium, when an electronic device is executed: The operation of acquiring a first image frame through the image sensor, the first image frame including first correlation information between first data acquired through a first subpixel of the image sensor and second data acquired through a second subpixel of the image sensor; An operation to obtain second correlation information for a region of interest by performing spatial filtering that applies at least one weight to correlation values ​​for a plurality of regions divided from the first image frame, based on the first correlation information; The operation of obtaining fourth correlation information for the region of interest by performing a temporal filter that blends the second correlation information with third correlation information corresponding to a second image frame obtained at a different time from the first image frame; and A recording medium having a computer program that records a method for performing an operation to determine depth information based on the above-mentioned fourth correlation information.

Images