Electronic device for generating high dynamic range image and operation method thereof

Abstract

An operation method of an electronic device of one embodiment that is disclosed may comprise an operation for performing face detection for each image acquired through at least one camera module, thereby obtaining first face data related to each detected face. The operation method of the electronic device may comprise an operation for receiving a capture input. The operation method of the electronic device may comprise an operation for, on the basis of the pieces of first face data, identifying, from among images acquired before receiving the capture input, a first image and second face data related to a face contained in the first image. The operation method of the electronic device may comprise an operation for acquiring a second image and a third image by using the at least one camera module. The second image may be acquired by having the at least one camera module receive light for a first exposure time. The third image may be acquired by having the at least one camera module receive light for a second exposure time shorter than the first exposure time. The operation method of the electronic device may comprise performing face detection for the second image, thereby obtaining third face data related to a face contained in the second image. The operation method of the electronic device may comprise obtaining a first gain value for the second image on the basis of the second face data and the third face data. The operation method of the electronic device may comprise an operation for obtaining a high dynamic range image by using the first gain value-applied second image and the third image.

Description

Electronic device for generating high dynamic range images and method of operating the same

[0001] The present disclosure relates to an electronic device for generating a high dynamic range image and a method of operating the same.

[0002] With the advancement of digital technology, various types of electronic devices, such as mobile terminals, personal digital assistants (PDAs), electronic notebooks, smartphones, tablet PCs (personal computers), and wearable devices, are becoming widely used. Electronic devices can provide various functions. For example, electronic devices can run at least one application in the foreground and / or background to provide at least one function.

[0003] Electronic devices run on a specified operating system (e.g., Android operating system). TM An electronic device can provide various functions using a camera. For example, an electronic device can support multiple functions provided using a camera.

[0004] The above information may be provided as background information to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art in connection with the present disclosure.

[0005] An electronic device of the disclosed embodiment may include at least one camera module, a display, a memory for storing commands, and at least one processor. The electronic device may perform face detection on each of images acquired through the at least one camera module before receiving a capture input from a user, thereby obtaining first face data regarding the detected faces. Based on the first face data, the electronic device may identify a first image and second face data regarding a face included in the first image among the images acquired before receiving the capture input. The electronic device may obtain a second image and a third image using the at least one camera module. The second image may be obtained by having the at least one camera module receive light for a first exposure time. The third image may be obtained by having the at least one camera module receive light for a second exposure time that is shorter than the first exposure time. The electronic device may obtain third face data regarding a face included in the second image by performing face detection on the second image. The electronic device may obtain a first gain value for the second image based on the second face data and the third face data. The electronic device can obtain a high dynamic range (HDR) image using the second image and the third image to which the first gain value is applied.

[0006] A method of operating an electronic device according to one embodiment of the disclosure may include an operation of acquiring first face data regarding detected faces by performing face detection on each of images acquired through at least one camera module. The method of operating the electronic device may include an operation of receiving a capture input. The method of operating the electronic device may include an operation of identifying, based on the first face data, a first image and second face data regarding a face included in the first image from among images acquired before receiving the capture input. The method of operating the electronic device may include an operation of acquiring a second image and a third image using at least one camera module. The second image may be acquired by having at least one camera module receive light for a first exposure time. The third image may be acquired by having at least one camera module receive light for a second exposure time that is shorter than the first exposure time. The method of operating the electronic device may acquire third face data regarding a face included in the second image by performing face detection on the second image. The method of operating the electronic device may acquire a first gain value for the second image based on the second face data and the third face data. A method of operating an electronic device may include obtaining a high dynamic range (HDR) image using a second image and a third image to which a first gain value is applied.

[0007] FIG. 1 is a block diagram illustrating an electronic device within a network environment according to various embodiments.

[0008] FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

[0009] FIG. 3 is a block diagram illustrating a configuration of an electronic device according to one embodiment.

[0010] FIG. 4 is a diagram conceptually illustrating the configuration of an image sensor according to one embodiment.

[0011] FIG. 5 is a drawing for explaining the operation of an electronic device according to one embodiment.

[0012] FIG. 6 is a diagram for explaining an operation of an electronic device performing multi-frame synthesis according to one embodiment.

[0013] FIG. 7 is a diagram illustrating an operation of performing multi-frame synthesis using images acquired before an electronic device receives a capture input, according to one embodiment.

[0014] FIG. 8 is a diagram illustrating an operation of performing face recognition from each of images acquired by an electronic device before receiving a capture input, according to one embodiment.

[0015] FIG. 9 is a diagram illustrating an operation for an electronic device to obtain a gain value used to generate a high dynamic range (HDR) image according to one embodiment.

[0016] FIG. 10 is a diagram for explaining an operation of an electronic device using an artificial intelligence model according to one embodiment.

[0017] FIG. 11 is a diagram illustrating an operation of performing multi-frame synthesis using images acquired before an electronic device receives a capture input, according to one embodiment.

[0018] FIG. 12 is a diagram illustrating an operation of performing multi-frame synthesis using images acquired before an electronic device receives a capture input, according to one embodiment.

[0019] FIG. 13 is a flowchart illustrating an operation of performing multi-frame synthesis using images acquired before an electronic device receives a capture input, according to one embodiment.

[0020] FIG. 14A is a diagram illustrating an operation of an electronic device performing enhanced high dynamic range (HDR) synthesis using a camera application according to one embodiment.

[0021] FIG. 14B is a diagram illustrating an operation of an electronic device performing enhanced high dynamic range (HDR) synthesis using a camera application according to one embodiment.

[0022] FIG. 14c is a diagram illustrating an operation of an electronic device performing enhanced high dynamic range (HDR) synthesis using a camera application according to one embodiment.

[0023] FIG. 14D is a diagram illustrating an operation of an electronic device performing enhanced high dynamic range (HDR) synthesis using a camera application according to one embodiment.

[0024] FIG. 14E is a diagram illustrating an operation of an electronic device performing enhanced high dynamic range (HDR) synthesis using a camera application according to one embodiment.

[0025] FIG. 15 illustrates an example of a block diagram of a wearable device according to one embodiment.

[0026] FIG. 16A illustrates an example of a perspective view of a wearable device according to one embodiment.

[0027] FIG. 16b illustrates an example of one or more hardware elements disposed within a wearable device, according to one embodiment.

[0028] FIGS. 17A and 17B illustrate an example of an appearance of a wearable device according to one embodiment.

[0029] An electronic device can acquire an image signal using an image sensor included in a camera module. For example, the electronic device can acquire the image signal by controlling the camera module through a camera application. The camera application can generate a control signal including a request related to acquiring the image signal. The camera module can acquire the image signal in response to a request (e.g., a control signal) received from the camera application. The electronic device can acquire still image and / or moving image data by processing the image signal output from the image sensor.

[0030] The human visual perception system can have a wider dynamic range (DR) than typical digital cameras or monitors. Therefore, electronic devices may have difficulty capturing or representing images as perceived by humans. Images with a wider dynamic range than what electronic devices typically process are called high dynamic range (HDR) images. These HDR images can have a wider dynamic range than images captured with typical digital cameras.

[0031] Electronic devices can generate high-dynamic-range images through multi-frame compositing, which involves capturing and synthesizing multiple images corresponding to different exposure times. When an image with significant brightness differences across regions is acquired, the electronic device can acquire a high-dynamic-range image by synthesizing multiple images with different exposure times. For example, the electronic device can generate a high-dynamic-range image in backlit situations.

[0032] Electronic devices can generate high-dynamic-range images using short-exposure and long-exposure images acquired with preset exposure differences. For example, the electronic device can generate high-dynamic-range images using the highlights of the short-exposure image and the shadows of the long-exposure image. However, the brightness of the main object included in the high-dynamic-range image generated by the electronic device may not be sufficiently secured. If the brightness of the main object included in the image is not sufficiently secured, the color and expression of the main object may not be sufficient.

[0033] For example, in a backlit shooting situation, an electronic device can adjust the exposure so that the pixel values ​​corresponding to the bright areas of the image sensor are not saturated. For example, in a backlit shooting situation, the electronic device can adjust the exposure by adjusting the shutter speed quickly so that the picture is darker than in a forward-light shooting situation. In this case, if the main object (e.g., a person) is included in a dark area, the main object may be captured darkly. When the exposure is adjusted to brighten the main object, the main object may be expressed brightly, but the pixel values ​​in the bright areas cannot avoid being saturated. There is a problem that the image quality deteriorates in the area where the pixel values ​​are saturated, such as the loss of object details and color distortion.

[0034] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description of this disclosure.

[0035] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present invention. However, the disclosed embodiments may be implemented in various different forms and are not limited to the embodiments described herein.

[0036] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0037] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0038] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0039] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0040] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0041] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0042] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0043] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0044] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0045] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0046] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0047] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0048] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0049] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0050] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0051] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0052] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0053] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0054] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0055] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0056] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0057] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0058] FIG. 2 is a block diagram of an electronic device (200) including a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260).

[0059] The lens assembly (210) can collect light emitted from a subject that is the target of image capture. The lens assembly (210) can include one or more lenses.

[0060] In one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In such a case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. The lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0061] The flash (220) can emit light that is used to enhance light emitted or reflected from a subject. In one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LEDs, white LEDs, infrared LEDs, or ultraviolet LEDs), or a xenon lamp.

[0062] The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) may include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a BW (black and white) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) may be implemented using, for example, a CCD (charged coupled dEvice) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0063] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting read-out timing, etc.) in response to movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image.

[0064] According to one embodiment, the image stabilizer (240) can detect movement of the camera module (180) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) placed inside or outside the camera module (180). For example, the image stabilizer (240) can be implemented as an optical image stabilizer.

[0065] The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130), or as a separate memory that operates independently therefrom.

[0066] According to one embodiment, the memory (250) can at least temporarily store images acquired and output as preview images. The preview images can include images provided by the electronic device (200) that enable the user to confirm the location, lighting, or composition of the subject to be photographed so that the user can acquire the desired image.

[0067] The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, read-out timing control, etc.) on at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). In one embodiment, the image signal processor (260) may be at least a part of the processor (120). It may be configured as a separate processor that is configured or operates independently from the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after undergoing additional image processing by the processor (120).

[0068] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. For example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0069] FIG. 3 is a block diagram illustrating a configuration of an electronic device according to one embodiment.

[0070] Referring to FIG. 3, an electronic device (101) according to one embodiment may include a camera module (380), a memory (330), and a processor (320). The electronic device (101) according to one embodiment may further include a display (360). The display (360) may be replaced with an external display connected to the electronic device (101). The electronic device (101), the processor (320), the memory (330), the camera module (380), and the display (360) may each correspond to the electronic device (101), the processor (120), the memory (130), the camera module (180), and the display module (160) described above with reference to FIGS. 1 and 2 . However, the components of the electronic device (101) illustrated in FIG. 3 are for describing one embodiment, and the electronic device (101) may include more components than the components illustrated in FIG. 3 or may include other components that may replace at least some of the components. For example, the memory (330) is not limited to a storage medium included in the electronic device (101), but may include a cloud storage external to the electronic device (101).

[0071] According to one embodiment, the camera module (380) may include a lens unit (381) including at least one lens for focusing light (e.g., lens assembly (210) of FIG. 2) and an image sensor (383) (e.g., image sensor (230) of FIG. 2) that converts an optical signal passing through the lens unit (381) into a digital signal.

[0072] The image sensor (383) may include a color filter array including a plurality of light-receiving elements, a plurality of micro-lenses, and a plurality of color channels. The plurality of light-receiving elements may include photodiodes arranged to correspond to one micro-lens in an array having M rows and N columns. Here, M and N may each be a natural number greater than or equal to 2.

[0073] The color filter array included in the image sensor (383) may be configured with a non-Bayer pattern (e.g., a tetra pattern, a nona pattern, a hexadeca pattern). The colors of the element groups of the color filter array configured with the non-Bayer pattern may be matched to correspond to the Bayer pattern.

[0074] According to one embodiment, the image sensor (383) can output an image signal (e.g., raw image data) composed of a non-Bayer pattern. For example, the image sensor (383) including a color filter array composed of a non-Bayer pattern can output an image signal composed of a non-Bayer pattern by determining a pixel value using an output value of a light-receiving element corresponding to an element. The image signal output from the image sensor (383) can be data in which the color pattern is maintained by not changing the color order of the color pattern of the image sensor (383).

[0075] According to one embodiment, the image sensor (383) can output an image signal (e.g., raw image data) configured in a Bayer pattern. For example, an image sensor (383) including a color filter array configured in a non-Bayer pattern can output an image signal configured in a Bayer pattern by binning output values ​​of light-receiving elements corresponding to groups of elements constituting the color filter array.

[0076] According to one embodiment, the image sensor (383) may operate in a high-resolution mode or a crop mode. The high-resolution mode may include a mode in which each of the output values ​​of the light-receiving elements included in the image sensor (383) is used as the pixel value of the pixel corresponding to each of the light-receiving elements. In the present disclosure, 'pixel' may mean the smallest unit constituting a digital image. The resolution of an image may be expressed by the number of pixels included in the image. For example, if an image is composed of axb pixels arranged in a rows and b columns, the resolution of the image may be indicated as axb.

[0077] For example, an electronic device (101) can obtain high-resolution image data by using an image sensor (383) composed of 50 Mp photodetectors and using each of the output values ​​of the 50 Mp photodetectors as the pixel value of the pixel corresponding to the photodetectors. The high-resolution mode can be understood as a full pixel mode.

[0078] The crop mode may include a mode in which the output values ​​of a predetermined number of light-receiving elements (e.g., a predetermined number of light-receiving elements located at the center of the image sensor) among the light-receiving elements are used as pixel values. The electronic device (101) can obtain an image with a narrowed field of view (FOV) through the crop mode, and can provide the user with an experience similar to that of a zoom-in function. For example, when the electronic device (101) operates in a crop mode in which the output values ​​of 12.5 Mp light-receiving elements located at the center of the image sensor (383) composed of 50 Mp light-receiving elements are used as pixel values, the electronic device (101) can provide the user with an experience similar to that of a 2x zoom-in function.

[0079] According to one embodiment, the image sensor (383) can operate in a low-light mode or a multi-frame synthesis mode. The low-light mode may include a mode in which the pixel values ​​of the first pixels corresponding to the first light-receiving elements are used based on the values ​​output from the first light-receiving elements corresponding to the elements of the color filter array matched with the same color of the light-receiving elements included in the image sensor (383). The multi-frame synthesis mode may include a mode in which the image sensor (383) acquires a plurality of frames in which there is a difference in exposure value.

[0080] According to one embodiment, the memory (330) can store instructions that can be executed by the processor (320). The processor (320) can perform operations or control components of the electronic device (101) by executing the instructions stored in the memory (330).

[0081] In the present disclosure, the operation of the electronic device (101) may be understood as being performed by at least one processor (320) executing instructions. According to one embodiment, the processor (320) may include at least one of an application processor (AP), a central processing unit (CPU), an image signal processor (ISP) (e.g., the image signal processor (260) of FIG. 2), a graphical processing unit (GPU), or a neural processing unit (NPU). For example, the at least one processor (320) may include an application processor. The at least one processor (320) may acquire image data based on an image frame including information read out from an image sensor (383).

[0082] According to one embodiment, at least one processor (320) may control the camera module (380) by executing a camera application. For example, the processor (320) may initiate the operation of the camera module (380) through the camera application. For example, the processor (320) may provide the camera module (380) with a request for acquisition of at least one frame (e.g., a frame for preview, a frame for capture) through the camera application. For example, the processor (320) may acquire one or more frames through the camera application. For example, the processor (320) may identify a user input received through the camera application (e.g., a user input for acquisition of a capture image). For example, the processor (320) may acquire image data (e.g., preview image data, draft image data, capture image data, video data) through the camera application. For example, the processor (320) may display the acquired image data using the display (360).

[0083] According to one embodiment, at least one processor (320) can transmit a control signal to the image sensor (383) to perform a read operation. For example, at least one processor (320) can transmit the control signal based on a designated communication method (e.g., I2C (inter-integrated circuit), I3C (improved inter-integrated circuit)). At least one processor (320) can obtain image data output from the image sensor (383). The image data can include an image frame. For example, at least one processor (320) can receive the image data through an interface (e.g., MIPI (mobile industry processor interface)) connected to the image sensor (383).

[0084] According to one embodiment, at least one processor (320) may control the image sensor (383) to acquire an image corresponding to a user input. The processor (320) may control the image sensor (383) to operate based on a selected shooting mode. For example, the image sensor (383) may output an image signal (e.g., raw image data) by reading out each output of the light-receiving elements constituting the image sensor (383) based on a first shooting mode (e.g., high-resolution shooting mode) so that the output corresponds to each pixel corresponding to each of the light-receiving elements. For example, the image sensor (383) may output an image signal (e.g., raw image data) by reading out each output of a predetermined number of first light-receiving elements among a plurality of light-receiving elements so that the output corresponds to each pixel corresponding to each of the first light-receiving elements based on a second shooting mode (e.g., crop shooting mode). For example, the image sensor (383) can output an image signal (e.g., raw image data) by reading out the output of a first group including a plurality of light-receiving elements so that it corresponds to a first pixel corresponding to the first group based on a third shooting mode (e.g., low-light shooting mode). For example, the image sensor (383) can output a short-exposure image signal and a long-exposure image signal by controlling the exposure time of the image sensor (383) so that a preset difference in exposure exists based on a fourth shooting mode (e.g., HDR shooting mode).

[0085] According to one embodiment, at least one processor (320) can obtain image data by performing an operation on an image signal (e.g., raw image data) output from an image sensor (383). For example, the processor (320) can perform at least one operation (e.g., demosaicing, white balance, contrast, saturation value adjustment operation, gamma correction, brightness correction, color correction, sharpening, noise removal, tone mapping, edge enhancement) on the image signal.

[0086] For example, the processor (320) may generate a second image signal (e.g., raw image data composed of a non-Bayer pattern) by performing remosaicking on a first image signal (e.g., raw image data composed of a non-Bayer pattern) output from the image sensor (383). For example, the processor (320) may input an image signal output from the image sensor (383) to an image signal processor (e.g., the image signal processor (260) of FIG. 2) to perform at least one operation (e.g., demosaicing, an operation for adjusting white balance, contrast, saturation value, gamma correction, brightness correction, color correction, sharpening, noise removal, tone mapping, edge enhancement). The processor (320) may store the generated image data in the memory (330).

[0087] For example, the processor (320) can identify an object included in an image using an image signal. For example, the processor (320) can identify an object included in an image by performing an object recognition operation on the image signal. For example, the processor (320) can identify a main object and a sub-object among the objects included in the image. For example, the processor (320) can identify a face included in an image by performing face recognition.

[0088] According to one embodiment, at least one processor (320) can initiate operation of the camera module (380). For example, the processor (320) can initiate operation of the camera module (380) by providing a control signal to a hardware abstraction layer (HAL) that includes an ID (identification) of the camera and a command to initiate operation of the camera.

[0089] According to one embodiment, at least one processor (320) can acquire one or more frames. For example, the processor (320) can acquire one or more frames by controlling the camera module (380) through a request to acquire a frame. For example, the processor (320) can acquire a frame for preview composed of a Bayer pattern. For example, the processor (320) can acquire a frame for capture composed of a non-Bayer pattern.

[0090] According to one embodiment, at least one processor (320) can convert a first image signal composed of a first color pattern (e.g., a non-Bayer pattern) into a second image signal composed of a second color pattern (e.g., a Bayer pattern). For example, the processor (320) can convert the first image signal into the second image signal by performing remosaicing on the first image signal. For example, the processor (320) can convert the first image signal into the second image signal by performing binning on the first image signal. For example, the processor (320) can control a computation unit (e.g., a computation unit (417) of FIG. 4) of the image sensor (383) to convert the first image signal into the second image signal.

[0091] In one embodiment, at least one processor (320) can identify a received user input. For example, the processor (320) can identify a user input for changing a shooting mode (e.g., high-resolution mode, crop mode, low-light mode, multi-frame synthesis mode, still image shooting mode, video shooting mode). For example, the processor (320) can identify a user input related to capturing still images and / or videos. For example, the processor (320) can identify a user input for capturing a multi-frame synthesised image.

[0092] According to one embodiment, at least one processor (320) can obtain preview image data. For example, the processor (320) can generate preview image data by applying an image signal composed of a Bayer pattern output from an image sensor (383) including a color filter of a Bayer pattern to an image signal processor (260). For example, the processor (320) can control an operation unit (e.g., operation unit (417) of FIG. 4) of the image sensor (383) to output an image signal composed of a Bayer pattern by binning an output value of a light-receiving element of the image sensor (383) including a color filter of a non-Bayer pattern (e.g., tetra, nona, hexadeca).

[0093] According to one embodiment, at least one processor (320) can obtain a capture image. For example, the processor (320) can generate a capture image by performing at least one operation (e.g., demosaicing, white balance, contrast, operation for adjusting saturation value, gamma correction, color correction, sharpening, noise removal, tone mapping, edge enhancement) on a first image signal composed of a non-Bayer pattern output from an image sensor (383). For example, the processor (320) can generate a capture image by performing an operation on an image signal of a frame composed of a non-Bayer pattern based on a high-resolution mode. For example, the processor (320) can generate a capture image by performing an operation on an image signal of a frame composed of a non-Bayer pattern based on a crop mode.

[0094] According to one embodiment, at least one processor (320) may display acquired image data using a display (360). For example, the processor (320) may display at least one of preview image data and captured image data in at least a portion of a camera application.

[0095] According to one embodiment, the display (360) can display one or more pieces of information under the control of at least one processor (320). For example, the display (360) can display a user interface (UI) of an electronic device. For example, the display (360) can display an execution screen of an application running on the electronic device. For example, the display (360) can display preview image data and captured image data.

[0096] FIG. 4 is a diagram conceptually illustrating the configuration of an image sensor according to one embodiment. The image sensor of FIG. 4 may correspond to the image sensor (230, 383) described above with reference to FIGS. 2 and 3.

[0097] Referring to FIG. 4, an image sensor according to one embodiment may include a micro lens array (MLA) (411), a color filter array (CFA) (413), a light receiving unit (415), and a computation unit (417).

[0098] In one embodiment, the micro lens array (411) may be arranged so that a light bundle (421) passing through the lens assembly (e.g., lens assembly (210) of FIG. 2) is focused on a light receiving element of the light receiving unit (415). The light bundle (423) passing through the micro lens array (411) may have at least some wavelengths outside of a band corresponding to a specific color blocked as it passes through the color filter array (413).

[0099] According to one embodiment, the color filter array (413) may be positioned at a position corresponding to a sensor pixel of the image sensor (230). For example, the sensor pixel may correspond to a light-receiving element as a component unit of the image sensor (230). Light bundles (425) passing through the color filter array (413) may be detected by a light-receiving element (e.g., a photodiode) of the light-receiving unit (415).

[0100] In one embodiment, the light-receiving elements of the light-receiving unit (415) may include photodiodes arranged in an array having M rows and N columns to correspond to one micro lens. Here, M and N may each be a natural number greater than or equal to 2. The light-receiving unit (415) may include a light-receiving element (e.g., including a light-receiving circuit) that generates a charge when receiving light and converts it into an electrical signal, and a circuit that selectively reads out the charge of the light-receiving element. A circuit for digitizing a signal read out from the light-receiving unit (415) or reducing noise may be further arranged between the light-receiving unit (415) and the calculation unit (417).

[0101] In one embodiment, the microlens array (411) may be arranged to correspond to at least one light-receiving element. For example, when viewed from the direction in which the light bundle (421) is incident, an area in which a single microlens included in the microlens array (411) is arranged may at least partially overlap an area in which a plurality of light-receiving elements are arranged. The microlenses included in the microlens array (411) may be arranged in a different color channel from adjacent microlenses, but a plurality of microlenses corresponding to the same color channel may be arranged adjacent to each other. The arrangement between the microlens array (411), the color filter array (413), and the light-receiving unit (415) may be configured differently depending on the type of image sensor.

[0102] In one embodiment, the color pattern of the color filter array (413) is illustrated based on a Bayer pattern, but the color pattern of the color filter array (413) is not limited to that illustrated in FIG. 4. Areas within the color filter array (413) corresponding to a plurality of adjacent micro lenses may also be configured to include the same color channel.

[0103] In one embodiment, the operation unit (417) can perform an operation to process electrical data (or signal) (427) output from the light receiving unit (415). The operation unit (417) can output data obtained based on the operation result. The output of the operation unit (417) can be a sensor output (429) of the image sensor (230).

[0104] FIG. 5 is a diagram for explaining the operation of an electronic device according to one embodiment. The electronic device (500) and components of the electronic device (500) of FIG. 5 may correspond to the electronic device (101) and components of the electronic device (101) described above with reference to FIGS. 1 to 4. For example, the camera module of FIG. 5 may correspond to the camera modules (180, 380) described above with reference to FIGS. 2 and 3. For example, the camera module of FIG. 5 may acquire an image using the image sensor (230, 383) described above with reference to FIGS. 2, 3, and 4.

[0105] Referring to FIG. 5, the electronic device (500) may include one or more camera modules. For example, the electronic device (500) may include a plurality of camera modules that support different field of view ranges due to differences in focal lengths. The field of view in the present disclosure may be a range that the camera module can view. For example, the field of view range may include a field of view angle and a field of view (FOV) supported by the camera module. For example, the electronic device may include a first camera module that acquires an image in a first field of view range (e.g., ultra-wide angle), a second camera module that acquires an image in a second field of view range (e.g., wide angle), a third camera module that acquires an image in a third field of view range (e.g., telephoto), and a fourth camera module that acquires an image in a fourth field of view range (e.g., super telephoto). In addition, the electronic device may include a fifth camera module that acquires information (e.g., a depth map) regarding the distance between a subject and the electronic device. However, the present invention is not limited thereto.

[0106] According to one embodiment, the plurality of camera modules of the electronic device (500) may be mapped to each other. For example, at least a portion of the sensor pixels of each of the plurality of camera modules may be mapped to each other based on a difference in the focal length of each lens module of the plurality of camera modules and a degree of rotation and / or translation of images acquired from each of the plurality of camera modules. The sensor pixel may include a component unit of an image sensor corresponding to a light-receiving element. By mapping the plurality of camera modules to each other, the coordinates of the image pixels of images acquired through the plurality of camera modules may be matched to each other.

[0107] According to one embodiment, at least one camera module of the electronic device (500) may include a color filter array. The color filter array of the at least one camera module may be configured in a Bayer pattern or a non-Bayer pattern (e.g., a tetra pattern, a nona pattern, a hexadeca pattern).

[0108] According to one embodiment, the electronic device (500) can obtain an image signal using one or more camera modules. For example, the electronic device (500) can control the camera module to obtain an image signal based on a request received from a camera application. For example, the electronic device (500) can obtain an image signal by exposing an image sensor in response to a request received from a camera application. For example, the electronic device (500) can obtain an image signal by reading data from sensor pixels included in at least some of the pixel lines of the image sensor. The pixel lines may include lines of the image sensor in which sensor pixels are arranged. For example, the electronic device (500) can obtain image data composed of image pixels by obtaining a pixel value of each image pixel corresponding to each of the sensor pixels based on data read from the sensor pixels of the image sensor.

[0109] According to one embodiment, the electronic device (500) can obtain image data by processing an image signal output from a camera module. For example, the electronic device (500) can obtain image data by applying the image signal to an image signal processor (e.g., the image signal processor (260) of FIG. 2). The image signal processor can be implemented as a hardware chip or as a hardware block included in a processor (e.g., an application processor (AP), a central processing unit (CPU), a graphical processing unit (GPU), or a neural processing unit (NPU)).

[0110] According to one embodiment, the electronic device can obtain and display a preview image. For example, the electronic device (500) can obtain a preview image in response to a preview request received from a camera application. For example, the electronic device (500) can obtain the preview image by controlling a camera module based on a command for initiating a camera operation generated through the camera application. The electronic device can display the preview image on at least a portion of a display (e.g., the display module (160) of FIG. 1) through the camera application.

[0111] According to one embodiment, the electronic device (500) can acquire image data in response to a user input received through a camera application. For example, the electronic device (500) can acquire still images and / or moving images in response to a user input requesting to acquire still images and / or moving images. The electronic device (500) can acquire image signals of frames for generating still images and / or moving images through an image sensor. The electronic device (500) can generate still images and / or moving images by processing the image signals.

[0112] According to one embodiment, the electronic device (500) can acquire an image based on various shooting modes. For example, the electronic device (500) can acquire an image based on a first shooting mode (e.g., high-resolution shooting mode). For example, the electronic device (500) can acquire an image based on a second shooting mode (e.g., crop shooting mode). For example, the electronic device (500) can acquire an image based on a third shooting mode (e.g., low-light shooting mode). For example, the electronic device (500) can acquire a high-dynamic-range image based on a fourth shooting mode (e.g., multi-frame synthesis shooting mode).

[0113] According to one embodiment, the electronic device (500) can acquire an image based on a shooting mode selected by a user. For example, the electronic device (500) can receive a user input for selecting a shooting mode. For example, the electronic device (500) can control the image sensor to operate in accordance with the selected shooting mode in response to the user input. For example, the electronic device (500) can control the exposure time of the image sensor so that a preset exposure difference exists based on a fourth shooting mode (e.g., a multi-frame synthesis shooting mode).

[0114] According to one embodiment, the electronic device (500) can identify the shooting situation of the electronic device (500). For example, the electronic device (500) can identify the shooting situation of the electronic device (500) based on the sensing value acquired using the sensor module. For example, the electronic device (500) can identify that it is a low-light shooting situation from the illuminance value of the surrounding environment acquired through the illuminance sensor. For example, the electronic device (500) can identify the shooting situation of the electronic device (500) from the pixel values ​​of the image acquired using the camera module. For example, the electronic device (500) can identify that it is a low-light shooting situation from the pixel values ​​of the image acquired using the camera module. For example, the electronic device (500) can identify that it is a multi-frame composite shooting situation based on the bright and dark areas of the image acquired using the camera module. For example, the electronic device (500) can identify a multi-frame composite shooting situation when the ratio of bright areas and dark areas among images acquired using the camera module is greater than a predetermined value.

[0115] According to one embodiment, the electronic device (500) can acquire an image in a shooting mode corresponding to the identified situation. For example, the electronic device can acquire an image in a first mode based on identifying a high-resolution shooting situation. For example, the electronic device can acquire an image in a second mode based on identifying a crop shooting situation. For example, the electronic device can acquire an image in a third mode based on identifying a low-light shooting situation. For example, the electronic device can acquire an image in a fourth mode based on identifying a multi-frame composite shooting situation.

[0116] According to one embodiment, the electronic device (500) can obtain a backlight image using a camera module. For example, the electronic device (500) can capture a person (590) sitting in front of a window using the camera module. The person (590) may correspond to a main object. The electronic device (500) can capture an image without saturating the pixel values ​​of the bright areas, thereby capturing a dark area more darkly. For example, the electronic device (500) can capture an image by lowering the exposure (e.g., capturing by reducing the exposure time) so that the outside scenery seen through the window is clearly visible, thereby capturing an image in which the person (590) is captured darkly.

[0117] According to one embodiment, the electronic device (500) can acquire a high dynamic range image. For example, the electronic device (500) can acquire a high dynamic range image based on the shooting mode being set to a multi-frame synthesis shooting mode. For example, the electronic device (500) can generate a high dynamic range image through multi-frame synthesis that captures and synthesizes multiple images corresponding to different exposure times. When an image having a large difference in brightness for each region is acquired, the electronic device (500) can acquire a high dynamic range image by synthesizing multiple images having differences in exposure times. The electronic device can generate a high dynamic range image using a short exposure image and a long exposure image acquired so that a preset difference in exposure exists. The electronic device can display the generated high dynamic range image.

[0118] FIG. 6 is a diagram illustrating an operation of an electronic device performing multi-frame synthesis according to one embodiment. FIG. 6 is a diagram illustrating an operation of an electronic device (500) described above with reference to FIG. 5 performing multi-frame synthesis to generate a high dynamic range (HDR) image according to one embodiment.

[0119] According to one embodiment, an electronic device can acquire multiple images. For example, the electronic device can acquire multiple image frames based on different exposure values ​​using a camera module. In the disclosed embodiment, the exposure value may include a value corresponding to the amount of exposure that exposes the electronic device to light when acquiring an image. For example, the electronic device may determine a reference exposure value (0Ev) that serves as a reference for shooting settings based on a shooting environment. The electronic device may determine a negative exposure value (-Ev) that decreases the exposure value based on the reference exposure value, or a positive exposure value (+Ev) that increases the exposure value. The multiple image frames acquired by the electronic device through the camera may include at least two or more of a first image frame acquired based on a reference exposure value (e.g., 0Ev), a second image frame acquired based on an exposure value lower than the reference exposure value (-Ev), or a third image frame acquired based on an exposure value higher than the reference exposure value (+Ev).

[0120] Referring to FIG. 6, the electronic device can acquire a first image frame (610) based on a reference exposure value (e.g., 0 Ev) through the camera module. For example, the electronic device can acquire the first image frame (610) based on the first metering result that prevents pixel values ​​of a first area (612) (e.g., pixels corresponding to a building outside the window) from being saturated. Accordingly, a second area (611) corresponding to a person sitting by the window, which is the main object, can be captured darkly.

[0121] Referring to FIG. 6, the electronic device may acquire a second image frame (620) based on an exposure value (e.g., +3 Ev) higher than a reference exposure value. For example, the electronic device may acquire a second image frame (620) corresponding to an exposure value (e.g., +3 Ev) higher than a reference exposure value based on the first metering result. Accordingly, the second area (621) corresponding to a person sitting by the window, which is the main object, may be brighter than the second area (611) of the first image frame (610). Meanwhile, pixel values ​​of the first area (622) (e.g., pixels corresponding to a building outside the window) may be saturated.

[0122] According to one embodiment, an electronic device can acquire a high dynamic range (HDR) image by synthesizing multiple image frames. For example, the electronic device can acquire the high dynamic range image based on a recovery map acquired from an image signal (e.g., raw data) output from an image sensor.

[0123] Referring to FIG. 6, the electronic device can obtain a high dynamic range (HDR) image (630) by synthesizing a first image frame (610) and a second image frame (620). For example, the electronic device can obtain a high dynamic range (HDR) image by synthesizing the first image frame (610) and the second image frame (620) using a recovery map generated based on a difference between an image signal for generating the first image frame (610) and an image quality control value (e.g., a brightness adjustment value, a white balance adjustment value).

[0124] For example, the electronic device can generate a high dynamic range image (630) using pixel values ​​of a first region (612) of a first image frame (610) and pixel values ​​of a second region (621) of a second image frame (620). That is, the pixel values ​​of the first region (632) of the high dynamic range image (630) may be determined based on the pixel values ​​of the first region (612) of the first image frame (610), and the pixel values ​​of the second region (631) may be determined based on the pixel values ​​of the second region (621) of the second image frame (620).

[0125] According to the disclosed embodiment, since the electronic device generates a high dynamic range image (630) using the first image frame (610) and the second image frame (620) acquired based only on the difference in preset exposure, the brightness of the main object included in the high dynamic range image (630) generated by the electronic device may not be sufficiently secured. If the brightness of the main object included in the image is not sufficiently secured, the color and expression of the main object may not be sufficient.

[0126] FIG. 7 is a diagram illustrating an operation of performing high dynamic range (HDR) synthesis using images acquired by an electronic device before receiving a capture input, according to one embodiment. FIG. 7 is a diagram illustrating an operation of the electronic device (500) described above with reference to FIG. 5 performing multi-frame synthesis to generate a high dynamic range (HDR) image, according to one embodiment.

[0127] According to one embodiment, the electronic device may set the shooting mode to a multi-frame synthesis shooting mode. For example, the electronic device may set the shooting mode to the multi-frame synthesis shooting mode in response to a user input related to the shooting mode. For example, the electronic device may set the shooting mode to the multi-frame synthesis shooting mode based on pixel values ​​of an image acquired through a camera module. For example, the electronic device may set the shooting mode to the multi-frame synthesis shooting mode based on identifying that the ratio of bright areas and / or the ratio of dark areas in an image acquired using the camera module is greater than a predetermined value.

[0128] According to one embodiment, the electronic device can acquire image frames based on a multi-frame composite shooting mode. For example, the electronic device can acquire a short-exposure image frame and a long-exposure image frame based on the multi-frame composite shooting mode.

[0129] For example, an electronic device can acquire a short exposure image frame based on a reference exposure value (e.g., 0 Ev) through a camera module. The electronic device can acquire a long exposure image frame based on an exposure value higher than the reference exposure value (e.g., +3 Ev). A long exposure image frame can be an image frame acquired based on an image sensor (e.g., the image sensor (230) of FIG. 2) being exposed for a longer time than when the short exposure image frame was acquired.

[0130] For example, an electronic device may acquire a long exposure image frame based on a reference exposure value (e.g., 0 Ev) through a camera module. The electronic device may acquire a short exposure image frame based on an exposure value lower than the reference exposure value (e.g., -3 Ev). A long exposure image frame may be an image frame acquired based on an image sensor (e.g., the image sensor (230) of FIG. 2) being exposed for a longer time than when a short exposure image frame is acquired.

[0131] According to one embodiment, the electronic device may obtain preview image frames (720). For example, the electronic device may obtain the preview image frames (720) based on a command provided from a camera application. For example, in response to receiving a request from the camera application to obtain a preview image frame, the electronic device may obtain the preview image frames at a predetermined interval using at least one camera module. For example, the electronic device may obtain the preview image frames (720) based on at least one of a short exposure image frame and a long exposure image frame. For example, the electronic device may obtain long exposure image frames as preview image frames (720) from among image frames obtained through the camera module before receiving a capture input from the user. For example, the electronic device may obtain short exposure image frames as preview image frames (720) from among image frames obtained through the camera module before receiving a capture input from the user. For example, the electronic device can obtain composite image frames composed of a short exposure image frame and a long exposure image frame as preview image frames (720).

[0132] According to one embodiment, the electronic device may obtain preview image frames (720) using a plurality of camera modules. For example, the electronic device may obtain preview image frames (720) from each of camera modules that support different field of view. For example, the electronic device may obtain preview image frames (720) through a first camera module that supports a first field of view (e.g., ultra-wide angle). For example, the electronic device may obtain preview image frames (720) through a second camera module that supports a second field of view (e.g., wide angle). For example, the electronic device may obtain preview image frames (720) through a third camera module that supports a third field of view (e.g., telephoto). For example, the electronic device may obtain preview image frames (720) through a fourth camera module that supports a fourth field of view (e.g., super-telephoto).

[0133] According to one embodiment, at least one frame among the preview image frames (720) may be brighter than the second image frame (710). The electronic device may adjust the exposure value to acquire an image with a reference exposure value at which pixel values ​​are not saturated. The electronic device may acquire image frames even while the exposure value is converging to the reference exposure value. The image frames acquired while the exposure value is converging to the reference exposure value may be brighter than the image frames acquired after the exposure value has converged. Accordingly, the preview image frames acquired based on the non-converged exposure value may be brighter than the second image frame (710) acquired after the exposure value has converged. That is, a long exposure image frame acquired before a capture input is received may be a brighter image frame than a long exposure image frame acquired after the capture input is received. Based on the preview image frames acquired based on the non-converged exposure value, a high dynamic range (HDR) image in which the brightness of the main object is appropriately secured may be acquired.

[0134] In one embodiment, the electronic device can identify a face from the preview image frames (720). For example, the electronic device can identify a face from the preview image frames (720) by performing face recognition on the preview image frames (720). For example, the electronic device can identify a face from each of the preview image frames (720) using an artificial intelligence model trained to perform object recognition. For example, the electronic device can identify a location of a face from the preview image frames (720). For example, the electronic device can identify a size of a face from the preview image frames (720). For example, the electronic device can identify a brightness of a face from the preview image frames (720).

[0135] According to one embodiment, the electronic device can identify similar image frames among the preview image frames (720). For example, the electronic device can identify a preview image frame that is different from other preview image frames among the preview image frames (720). For example, the electronic device can identify a preview image frame that is different from other preview image frames by identifying the similarity of the preview image frames. For example, the electronic device can identify a preview image frame that includes a passerby who obscures the main object. The electronic device may not identify a face in a preview image frame that is different from other preview image frames.

[0136] According to one embodiment, the electronic device may store facial data regarding a face identified from the preview image frames (720). For example, the electronic device may store facial data regarding a face in metadata of each of the preview image frames (720). For example, the electronic device may store data including facial data of the preview image frames (720) that matches the preview image frames (720) in a memory. For example, the electronic device may store facial size data regarding a size of a face identified from the preview image frames (720). For example, the electronic device may store facial brightness data regarding a brightness of a face identified from the preview image frames (720).

[0137] According to one embodiment, the electronic device may identify a first preview image frame (720) among the preview image frames (720). For example, the electronic device may identify, based on data about faces in the preview image frames (720), an image frame that includes a face most significantly among the preview image frames (720) as the first preview image frame (720). For example, based on data about faces in the preview image frames (720), the electronic device may identify, based on data about faces in the preview image frames (720), an image frame that includes a face most brightly among the preview image frames (720) as the first preview image frame (720).

[0138] In one embodiment, the electronic device may receive a capture input from a user. For example, the electronic device may receive the capture input through a user interface (UI) provided by a camera application.

[0139] According to one embodiment, the electronic device can acquire a first image frame (740) and a second image frame (710) in response to a capture input. For example, the electronic device can acquire a short exposure image frame as the first image frame (740) based on a reference exposure value (e.g., 0 Ev). For example, the electronic device can acquire a long exposure image frame as the second image frame (710) based on an exposure value higher than the reference exposure value (e.g., +3 Ev). For example, the electronic device can acquire the first image frame (740) and the second image frame (710) by controlling a camera module in response to a capture input.

[0140] For example, the electronic device may acquire a short-exposure image frame acquired at substantially the same time as the time at which the capture input is received as the first image frame (740). For example, the electronic device may acquire a long-exposure image frame acquired at substantially the same time as the time at which the capture input is received as the second image frame (710).

[0141] For example, the electronic device may acquire the first image frame (740) such that the pixel values ​​of the first region (742) (e.g., pixels corresponding to a building outside the window) are not saturated. The second region (741) of the first image frame (740) (e.g., pixels corresponding to a person sitting by the window) may be dark. For example, the second region (711) of the second image frame (710) (e.g., pixels corresponding to a person sitting by the window) may be brighter than the second region (741) of the first image frame (740). Meanwhile, the pixel values ​​of the first region (712) of the second image frame (710) (e.g., pixels corresponding to a building outside the window) may be saturated.

[0142] According to one embodiment, the electronic device may obtain a gain value to be applied to the second image frame (710). For example, the electronic device may obtain the gain value based on the first preview image frame (720). For example, the electronic device may obtain the gain value based on data about the face (e.g., face size data, face brightness data) of the first preview image frame (720). For example, the electronic device may obtain the gain value from the brightness value of the face included in the first preview image frame (720) based on a lookup table (LUT) in which the brightness value of the face and the gain value are mapped. For example, the electronic device may obtain the gain value by performing a weighted sum on the brightness value of the face included in the first preview image frame (720) and the brightness value of the face included in the second image frame (710). For example, the electronic device can obtain a gain value from the size value of the face included in the first preview image frame (720) based on a lookup table (LUT) in which the size value of the face and the gain value are mapped. For example, the electronic device can obtain a gain value by performing a weighted sum on the size value of the face included in the first preview image frame (720) and the size value of the face included in the second image frame (710).

[0143] According to one embodiment, the electronic device may apply a gain value to the second image frame (710). For example, the electronic device may apply the gain value obtained based on the first preview image frame (720) to at least a portion of the second image frame (710). For example, by applying the gain value to the second image frame (710), the electronic device may obtain a third image frame (730) in which at least a portion of the second image frame (710) is brighter. For example, the third image frame (730) may be brighter overall than the second image frame (710). For example, a portion of the third image frame (730) (e.g., an area corresponding to a main object) may be brighter than a portion of the second image frame (710) corresponding to a portion of the third image frame (730) (e.g., an area corresponding to the main object). Accordingly, the second region (731) of the third image frame (730) may be brighter than the second region (711) of the second image frame (710). The first region (732) of the third image frame (730) may be brighter than the first region (712) of the second image frame (710). For example, the third image frame (730) may be acquired in YUV format.

[0144] According to one embodiment, the electronic device can obtain a fourth image frame (750) based on the third image frame (730) and the first image frame (740). The fourth image frame (750) may be a high dynamic range (HDR) image generated through multi-frame synthesis. For example, the electronic device may generate the fourth image frame (750) using pixel values ​​of the first region (742) of the first image frame (740) and pixel values ​​of the second region (731) of the third image frame (730). That is, the pixel values ​​of the first region (752) of the fourth image frame (750) may be determined based on the pixel values ​​of the first region (742) of the first image frame (740), and the pixel values ​​of the second region (751) may be determined based on the pixel values ​​of the second region (731) of the third image frame (730). For example, the electronic device can obtain a fourth image frame (750) by synthesizing the first image frame (740) and the third image frame (730) using a recovery map.

[0145] According to one embodiment, the electronic device may perform image processing on the fourth image frame (750). For example, the electronic device may perform at least one of operations for adjusting white balance, contrast, and saturation values, gamma correction, brightness correction, color correction, sharpening, noise removal, tone mapping, and edge enhancement on the fourth image frame (750). For example, the electronic device may adjust the brightness value of the second region (751) based on identifying whether the brightness value of the second region (751) of the fourth image frame (750) corresponds to a predetermined brightness value. For example, in response to identifying that the average brightness value of pixels constituting the second region (751) of the fourth image frame (750) is less than a preset brightness value, the electronic device may adjust the brightness value of the second region (751) so that the brightness value of the second region (751) corresponds to the preset brightness value. For example, the electronic device may adjust the brightness value of the face so that the brightness value of the face corresponds to the preset brightness value based on identifying whether the brightness value of the face included in the fourth image frame (750) corresponds to the preset brightness value.

[0146] According to the disclosed embodiment, an image in which at least a portion of the main object, i.e., a person, is brighter than in the embodiment described above with reference to FIG. 6 can be obtained.

[0147] FIG. 8 is a diagram illustrating an operation of performing facial recognition on each of the images acquired by an electronic device before receiving a capture input, according to one embodiment. FIG. 8 is a diagram illustrating an operation of the electronic device (500) described above with reference to FIG. 5 performing facial recognition on image frames.

[0148] In one embodiment, the electronic device may receive a capture input (801) from a user. For example, the electronic device may receive the capture input (801) from the user at a first time (t0). For example, the electronic device may receive the capture input (801) through a user interface (UI) provided by a camera application.

[0149] In one embodiment, the electronic device may acquire a first image frame (810) in response to a capture input. For example, the electronic device may acquire the first image frame (810) acquired at substantially the same time as the time at which the capture input is received.

[0150] According to one embodiment, the electronic device can acquire image frames (820, 830) before a capture input is received. For example, the electronic device can acquire image frames (820, 830) based on a command provided from a camera application. For example, the electronic device can acquire image frames (820, 830) at predetermined intervals. For example, the electronic device can acquire image frames (820, 830) at a second time (t1) and a third time (t2) before receiving a capture input from a user. The image frames (820, 830) can be preview image frames.

[0151] In one embodiment, the electronic device may acquire image frames using at least one camera module. For example, the electronic device may acquire image frames (820, 830) from each of the camera modules supporting different field of view ranges.

[0152] In one embodiment, the electronic device can identify a face (811, 821, 831) from each of the image frames (810, 820, 830). For example, the electronic device can identify multiple faces contained in each of the image frames (810, 820, 830).

[0153] For example, the electronic device can identify a face (811, 821, 831) from each of the image frames (810, 820, 830) based on the matching of feature points included in the faces (811, 821, 831). For example, the electronic device can identify a face (811, 821, 831) from each of the image frames (810, 820, 830) based on whether a skin color of the face (811, 821, 831) is similar to a predetermined skin color. For example, the electronic device can identify a face (811, 821, 831) based on identifying features of landmarks (e.g., eyes, teeth) of the image frames (810, 820, 830).

[0154] For example, the electronic device can identify a face (811, 821, 831) from each of the image frames (810, 820, 830) based on the result of identifying the face output from the artificial intelligence model (e.g., the artificial intelligence model (1010) of FIG. 10) by inputting the image frames (810, 820, 830) to an artificial intelligence model trained to identify faces (811, 821, 831). For example, the electronic device can identify a face (811, 821, 831) from each of the image frames (810, 820, 830) based on the result of identifying the face output from the artificial intelligence model (e.g., the artificial intelligence model (1010) of FIG. 10) trained to detect and identify faces (811, 821, 831) using statistical cluster information.

[0155] According to one embodiment, the electronic device can obtain face data of the image frames (810, 820, 830). For example, the electronic device can obtain face data including face size data regarding the size of the faces of the image frames (810, 820, 830) and face brightness data regarding the brightness of the faces of the image frames (810, 820, 830). For example, the electronic device can obtain face size data and / or face data from faces that are recognized as similar among the faces (811, 821, 831) included in the image frames (810, 820, 830). For example, the electronic device can obtain face size data and face brightness data of each of a plurality of faces included in each of the image frames (810, 820, 830).

[0156] According to one embodiment, the electronic device can divide each of the image frames (810, 820, 830) into a plurality of patches. For example, the electronic device can divide each of the image frames (810, 820, 830) into MxN (M and N are natural numbers) patches.

[0157] According to one embodiment, an electronic device can obtain face size data. For example, the electronic device can obtain the face size data by identifying the number of pixels in an area corresponding to a face among pixels constituting an image. For example, the electronic device can obtain the face size data based on a plurality of patches that divide an image. For example, the electronic device can obtain the face size data based on the number of patches corresponding to a face among MxN (M and N are natural numbers) patches. When the electronic device determines the size on a pixel basis, the electronic device can accurately identify the size of a face within an image frame, but this may increase the amount of computation. The electronic device can quickly obtain the face size data by identifying the face size based on dividing the size of the face within the image frame by the number of patches that divide the image.

[0158] According to one embodiment, an electronic device can obtain face brightness data. For example, the electronic device can obtain face size data by identifying pixel values ​​of pixels in an area corresponding to a face among pixels constituting an image. For example, the electronic device can obtain face brightness data based on a plurality of patches that divide an image. For example, the electronic device can obtain face brightness data based on brightness values ​​of patches that are divided into predetermined steps (e.g., 128 steps). For example, the electronic device can obtain face brightness data based on brightness values ​​of patches corresponding to a face among MxN (M and N are natural numbers) patches. For example, the electronic device can obtain face brightness data based on an average of brightness values ​​of patches corresponding to a face. The electronic device can quickly obtain face brightness data by identifying pixel values ​​corresponding to a face based on dividing a face within an image frame into patches that divide the image.

[0159] In one embodiment, the electronic device may store facial data. For example, the electronic device may store facial data for each of the image frames (810, 820, 830). For example, the electronic device may store facial size data and facial brightness data for each of the image frames (810, 820, 830). For example, the electronic device may store facial data in each of the metadata of the image frames (810, 820, 830).

[0160] According to one embodiment, an electronic device can identify a face of a primary object among a plurality of faces included in an image frame. For example, the electronic device can identify the face of the primary object based on the sizes of the plurality of faces. For example, the electronic device can identify the largest face among the plurality of faces as the face of the primary object. For example, the electronic device can identify the face of the primary object based on the orientation of the plurality of faces. For example, the electronic device can identify the face closest to the front among the plurality of faces as the face of the primary object. For example, the electronic device can identify the face of the primary object based on the positions of the plurality of faces. For example, the electronic device can identify the face closest to a predetermined position (e.g., the center, a third point of the screen) among the plurality of faces as the face of the primary object. For example, the electronic device can identify the face most similar to a face stored in the electronic device as the face of the primary object among the plurality of faces. For example, the electronic device can identify the face of the main object based on the identification result output from the artificial intelligence model by applying the image frame to an artificial intelligence model that identifies the main object (e.g., the artificial intelligence model (1010) of FIG. 10).

[0161] FIG. 9 is a diagram illustrating an operation for an electronic device to obtain a gain value used to generate a high dynamic range (HDR) image according to one embodiment. FIG. 9 is a diagram illustrating an operation for an electronic device (500) described above with reference to FIG. 5 to obtain a gain value described above with reference to FIG. 7.

[0162] According to one embodiment, the electronic device can identify a main object from preview image frames (e.g., the preview image frames 720 of FIG. 7). For example, the electronic device can identify the main object based on the arrangement, composition, etc. of objects within the preview image frames (e.g., the preview image frames 720 of FIG. 7). For example, the electronic device can identify that the main object is a person's face. For example, the electronic device can not perform the operation of the electronic device of FIG. 9 based on identifying that the main object of the preview image frames (e.g., the preview image frames 720 of FIG. 7) is not a person's face.

[0163] According to one embodiment, the electronic device may identify a first preview image frame among the preview image frames (e.g., the preview image frames 720 of FIG. 7). For example, the electronic device may identify an image frame containing a face most significantly among the preview image frames (e.g., the preview image frames 720 of FIG. 7) as the first preview image frame based on facial data of the preview image frames (e.g., the preview image frames 720 of FIG. 7). For example, the electronic device may identify an image frame containing a face most brightly among the preview image frames (e.g., the preview image frames 720 of FIG. 7) as the first preview image frame based on facial data of the preview image frames (e.g., the preview image frames 720 of FIG. 7). For example, the electronic device may identify the third image frame (830) described above with reference to FIG. 8 as the first preview image frame.

[0164] According to one embodiment, the electronic device may obtain a first face brightness value (911). For example, the electronic device may obtain the first face brightness value (911) of a first preview image frame that includes the brightest face among the preview image frames. For example, the electronic device may obtain the first face brightness value (911) of a first preview image frame that includes the highest value among average values ​​of brightness values ​​of a face region among the preview image frames. For example, the electronic device may obtain the first face brightness value (911) based on face brightness data of the first preview image frame. For example, the electronic device may obtain the first face brightness value (911) based on brightness values ​​of patches corresponding to a face among a plurality of patches dividing the first preview image frame. For example, the electronic device may obtain the first face brightness value (911) based on face brightness data obtained from metadata of the first preview image frame. For example, the electronic device may identify the first face brightness value (911) of the first preview image frame as 100.

[0165] According to one embodiment, the electronic device may obtain a second face brightness value (921). For example, the electronic device may obtain the second face brightness value (921) from a long exposure image frame (e.g., the second image frame (710) of FIG. 7) obtained by receiving a capture input from a user. For example, the electronic device may obtain the second face brightness value (921) based on brightness values ​​of patches corresponding to a face among a plurality of patches dividing the long exposure image frame. For example, the electronic device may obtain the second face brightness value (921) based on face brightness data obtained from metadata of the long exposure image frame (e.g., the second image frame (710) of FIG. 7). For example, the electronic device may identify the second face brightness value (921) as 70.

[0166] In one embodiment, the electronic device can identify a first weight (912) and a second weight (922). For example, the electronic device can identify a first weight (912) applied to a first preview image frame and a second weight (922) applied to a long exposure image frame (e.g., a second image frame (710) of FIG. 7). For example, the first weight (912) and the second weight (922) can be weights relating to an application ratio of a first brightness of the first preview image frame and a second brightness of a long exposure image frame (e.g., a 710 of FIG. 7). For example, the first weight (912) and the second weight (922) can be different values ​​or the same value. For example, the second weight (922) can be a value greater than, less than, or the same as the first weight (912). For example, the sum of the first weight (912) and the second weight (922) may be 1. For example, the electronic device may identify the first weight (912) and the second weight (922) as 0.5. For example, the electronic device may determine the first weight (912) and the second weight (922) based on the illuminance of the surrounding environment when acquiring the first preview image frame and the illuminance of the surrounding environment when acquiring the long exposure image frame. For example, the electronic device may determine the first weight (912) and the second weight (922) based on the exposure value of the first preview image frame identified from the first preview image frame and the exposure value of the long exposure image frame identified from the long exposure image frame.

[0167] According to one embodiment, the electronic device can obtain a third face brightness value (941). For example, the electronic device can obtain the third face brightness value (941) by performing a weighted sum (930) of a first face brightness value (911) and a second face brightness value (921). For example, the electronic device can obtain the third face brightness value (941) by adding a first face brightness value (911) to which a first weight (912) is applied and a second face brightness value (921) to which a second weight (922) is applied. For example, the electronic device can obtain the third face brightness value (941) based on mathematical expression 1.

[0168]

[0169] In mathematical expression 1, A1 may be a first face brightness value (911), B1 may be a second face brightness value (921), C1 may be a third face brightness value (941), 1-n may be a first weight (912), and n may be a second weight (922).

[0170] For example, if the first face brightness value (911) is 100, the second face brightness value (921) is 70, and the first weight (912) and the second weight (922) are 0.5, the third face brightness value (941) can be obtained as 85.

[0171] According to one embodiment, the electronic device may obtain a first face size value (951). For example, the electronic device may obtain the first face size value (951) of a first preview image frame that includes the largest face among the preview image frames. For example, the electronic device may obtain the first face size value (951) of a first preview image frame that includes the entire shape of the face among the preview image frames. For example, the electronic device may obtain the first face size value (951) based on the face size data of the first preview image frame. For example, the electronic device may obtain the first face size value (951) of an area that at least partially overlaps a face included in each of the preview image frames among the faces included in the first preview image frame. For example, the electronic device may obtain the first face size value (951) based on the number of patches corresponding to faces among a plurality of patches dividing the first preview image frame. For example, the electronic device can obtain the first face size value (951) based on the face size data obtained from the metadata of the first preview image frame. For example, the electronic device can obtain the first face size value (951) as 9.

[0172] According to one embodiment, the electronic device can obtain a second face size value (961). For example, the electronic device can obtain the second face size value (961) from a long exposure image frame (e.g., the second image frame (710) of FIG. 7) obtained by receiving a capture input from a user. For example, the electronic device can obtain the second face size value (961) based on the number of patches corresponding to a face among a plurality of patches dividing the long exposure image frame. For example, the electronic device can obtain the second face size value (961) based on face size data obtained from metadata of the long exposure image frame (e.g., the second image frame (710)). For example, the electronic device can obtain the second face size value (961) as 6.

[0173] In one embodiment, the electronic device can identify a third weight (952) and a fourth weight (962). For example, the electronic device can identify a third weight (952) applied to a first preview image frame and a fourth weight (962) applied to a long exposure image frame (e.g., a second image frame (710)). For example, the third weight (952) and the fourth weight (962) can be weights relating to an application ratio of a first size of the first preview image frame to a second size of the long exposure image frame (e.g., a second image frame (710)). For example, the third weight (952) and the fourth weight (962) can be different values ​​or the same value. For example, the third weight (952) can be a greater value, a lesser value, or the same value as the fourth weight (962). For example, the sum of the third weight (952) and the fourth weight (962) can be 1. For example, the electronic device may identify the third weight (952) and the fourth weight (962) as 0.5, respectively. For example, the electronic device may determine the third weight (952) and the fourth weight (962) based on the illuminance of the surrounding environment when acquiring the first preview image frame and the illuminance of the surrounding environment when acquiring the long exposure image frame. For example, the electronic device may determine the third weight (952) and the fourth weight (962) based on the exposure value of the first preview image frame identified from the first preview image frame and the exposure value of the long exposure image frame identified from the long exposure image frame.

[0174] According to one embodiment, the electronic device can obtain a third face size value (981). For example, the electronic device can obtain the third face size value (981) by performing a weighted sum (970) on the first face size value (951) and the second face size value (961). For example, the electronic device can obtain the third face size value (981) by adding the first face size value (951) to which the third weight (952) is applied and the second face size value (961) to which the fourth weight (962) is applied. For example, the electronic device can obtain the third face size value (981) based on Mathematical Expression 2.

[0175]

[0176] In mathematical expression 2, A2 may be a first face size value (951), B2 may be a second face size value (961), C2 may be a third face size value (981), 1-m may be a third weight (952), and m may be a fourth weight (962).

[0177] For example, the electronic device can obtain the third face size value (981) as 7.5 when the first face size value (951) is 9, the second face size value (961) is 6, and the third weight (952) and the fourth weight (962) are 0.5.

[0178] According to one embodiment, the electronic device may obtain the first gain value (942) based on the third face brightness value (941). For example, the electronic device may identify the first gain value (942) from the third face brightness value (941) based on a lookup table (LUT) in which the face brightness value and the gain value are mapped. For example, the electronic device may identify the first gain value based on the lookup table mapped as shown in Table 1.

[0179] Gain value range of step face brightness value 10 ≤ face brightness value ≤ 501.7250 ≤ face brightness value ≤ 1001.53100 ≤ face brightness value ≤ 1501.34150 ≤ face brightness value ≤ 2001.1

[0180] For example, if the third face brightness value (941) is 85, the electronic device can identify the first gain value (942) as 1.5.

[0181] According to one embodiment, the electronic device can obtain the second gain value (982) based on the third face size value (981). For example, the electronic device can identify the second gain value (982) from the third face size value (981) based on a lookup table (LUT) in which the face size value and the gain value are mapped. For example, the electronic device can identify the first gain value based on the lookup table mapped as shown in Table 2.

[0182] Gain value range of step face size value 10 ≤ face size value ≤ 1000.92100 ≤ face size value ≤ 2001.13200 ≤ face size value 1.3

[0183] For example, if the third face size value (981) is 7.5, the electronic device can identify the second gain value (982) as 0.9.

[0184] According to one embodiment, the electronic device can obtain a third gain value (991). For example, the electronic device can obtain the third gain value (991) by multiplying (990) the first gain value (942) and the second gain value (982). For example, when the first gain value (942) is 1.5 and the second gain value (982) is 0.9, the electronic device can obtain the third gain value (991) as 1.35.

[0185] According to one embodiment, the electronic device may apply a third gain value (991) to at least a portion of a long exposure image frame (e.g., the second image frame (710) of FIG. 7) acquired in response to receiving a capture input.

[0186] For example, the electronic device may apply the third gain value (991) to adjust the overall brightness of a long exposure image frame (e.g., the second image frame (710) of FIG. 7). For example, the electronic device may apply the third gain value (991) to adjust the brightness of a face included in the long exposure image frame (e.g., the second image frame (710) of FIG. 7). For example, the electronic device may apply the third gain value (991) to adjust the brightness of an area corresponding to a person included in the long exposure image frame (e.g., the second image frame (710) of FIG. 7).

[0187] For example, the electronic device can obtain an image frame (e.g., the third image frame (730) of FIG. 7) by applying a third gain value (991) to a long-exposure image frame (e.g., the second image frame (710)). The image frame to which the third gain value (991) is applied may be a brighter image than the long-exposure image frame (e.g., the second image frame (710)).

[0188] FIG. 10 is a diagram illustrating an operation of an electronic device using an artificial intelligence model according to one embodiment. FIG. 10 is a diagram illustrating an artificial intelligence model (1010) used by the electronic device (500) of FIG. 5 to acquire an image frame. The artificial intelligence model (1010) may have updated weights by performing forward propagation using training data (1020). The artificial intelligence model (1010) may have updated weights by performing back propagation using verification data.

[0189] In one embodiment, the artificial intelligence model (1010) may be trained to identify objects. For example, the artificial intelligence model (1010) may be trained through training data (1020) to identify objects from image frames. For example, the artificial intelligence model (1010) may be trained using training data (1020) containing objects such as people, flowers, animals, plants, cars, bicycles, and airplanes. For example, the artificial intelligence model (1010) may be trained to identify objects included in an image frame and output classified result data (1090) in response to the image frame being applied as input data (1030).

[0190] In one embodiment, the artificial intelligence model (1010) may be trained to identify a primary object from an image frame (e.g., image frames 810, 820, 830 of FIG. 8). For example, a primary object may refer to an object that appears important among the objects included in the image frame.

[0191] For example, the artificial intelligence model (1010) may be trained to identify and classify main objects and sub-objects from an image frame and output result data (1090) in response to the image frame being applied as input data (1030). For example, the artificial intelligence model (1010) may identify a main object among objects included in an image (e.g., a person, a flower, an animal, a plant, a car, a bicycle, an airplane). For example, the artificial intelligence model (1010) may be trained to identify an object located at a predetermined location (e.g., the center of the image frame, a third point), a largest object, an object that is sharper than other objects, and an object that contrasts with other objects as a main object among objects included in an image frame.

[0192] In one embodiment, the artificial intelligence model (1010) may be trained to identify a face from an image frame (e.g., 810, 820, 830 of FIG. 8). For example, the artificial intelligence model (1010) may be trained to identify a face from an image frame by learning training data (1020) containing a human face, and output data related to the face as result data (1090). For example, the artificial intelligence model (1010) may detect a facial region by considering the skin color and / or race of a person included in the image frame. For example, the artificial intelligence model (1010) may be trained to identify the face of a person with a light or dark skin color. For example, the artificial intelligence model (1010) may be trained to identify a facial region and / or a facial color based on identifying landmarks from an image frame. For example, the artificial intelligence model (1010) may be trained to identify an area corresponding to a face and / or the skin color of a face from an image frame based on features such as eyes, nose, mouth, ears, teeth, or cheekbones.

[0193] For example, the artificial intelligence model (1010) may be trained to obtain information about the size and brightness of a face. For example, the artificial intelligence model (1010) may be trained to output data about the size and brightness of a face as result data (1090) based on a plurality of patches that divide an image frame. According to one embodiment, the artificial intelligence model (1010) may be trained to perform image processing on at least a portion of an image frame. For example, the artificial intelligence model (1010) may be trained to perform at least one of operations for adjusting white balance, contrast, and saturation values, gamma correction, brightness correction, color correction, sharpening, noise removal, tone mapping, and edge enhancement.

[0194] In one embodiment, the artificial intelligence model (1010) may be trained to adjust the brightness value of the main object. For example, the artificial intelligence model (1010) may be trained to output, as result data (1090), an image frame in which the brightness value of the main object is adjusted to correspond to a preset brightness value. For example, the artificial intelligence model (1010) may be trained to output, by adjusting the brightness of a person included in an image frame generated through multi-frame synthesis, to correspond to a preset brightness value. For example, the artificial intelligence model (1010) may be trained to adjust the overall brightness value of the image frame. For example, the artificial intelligence model (1010) may be trained to adjust the brightness value of at least a portion (e.g., a face) of the main object (e.g., a person) included in the image frame.

[0195] An artificial intelligence model (1010) according to one embodiment may be an artificial neural network model written in a specified language and including a plurality of layers and / or operations (or calculations). The artificial intelligence model (1010) according to one embodiment may be one of a feedforward neural network (FNN), a deep neural network (DNN), a convolutional neural network (CNN), a region with convolution neural network (R-CNN), a region proposal network (RPN), a recurrent neural network (RNN), a stacking-based deep neural network (S-DNN), a state-space dynamic neural network (S-SDNN), a Deconvolution Network, a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, a Fully Convolutional Network, a long short-term memory (LSTM) Network, a Classification Network, or a combination of two or more of the above, but is not limited to the examples described above. An artificial intelligence model (1010) according to one embodiment can learn from specified data, acquire input data, and perform operations based on the input data to generate output data. In addition to a hardware structure, the artificial intelligence model (1010) may additionally or alternatively include a software structure.

[0196] FIG. 11 is a diagram illustrating an operation of performing high dynamic range (HDR) synthesis using images acquired before an electronic device receives a capture input, according to one embodiment. FIG. 11 is a diagram illustrating an operation of performing multi-frame synthesis by the electronic device (500) of FIG. 5 to acquire an HDR image.

[0197] In one embodiment, the electronic device may receive a capture input (1101). For example, the electronic device may receive the capture input (1101) through a user interface (UI) provided by a camera application.

[0198] According to one embodiment, the electronic device can obtain image frames (1130, 1140) before the capture input (1101) is received. For example, the electronic device can obtain a long exposure preview image frame (1130) and a short exposure preview image frame (1140) using the first camera module. For example, the electronic device can obtain the long exposure preview image frame (1130) and the short exposure preview image frame (1140) alternately. For example, the electronic device can obtain the long exposure preview image frame (1130) and the short exposure preview image frame (1140) so that they have a difference of a preset exposure value (e.g., 3 Ev). For example, the electronic device can store the image frames (1130, 1140). For example, the electronic device can store the image frames (1130, 1140) in a buffer.

[0199] According to one embodiment, the electronic device may analyze (1150) a face included in long exposure preview image frames (1130). For example, the electronic device may identify a face included in each of the long exposure preview image frames (1130). For example, the electronic device may obtain face data regarding a face included in the long exposure preview image frames (1130). For example, the electronic device may obtain face size data regarding a size of a face included in the long exposure preview image frames (1130). For example, the electronic device may obtain face brightness data regarding the brightness of a face included in the long exposure preview image frames (1130). The operation of the electronic device analyzing (1150) a face may be analogized to and applied to the operation of the electronic device described above with reference to FIG. 8.

[0200] In one embodiment, the electronic device may acquire image frames to be synthesized in high dynamic range (HDR) in response to receiving a capture input (1101). For example, the electronic device may acquire image frames (1110, 1120) after receiving the capture input (1101).

[0201] For example, the electronic device can acquire image frames (1110, 1120) acquired at substantially the same time as the time at which the capture input is received. For example, the electronic device can acquire a long exposure image frame (1110) based on a reference exposure value (e.g., 0 Ev). For example, the electronic device can acquire a short exposure image frame (1120) based on an exposure value (e.g., -3 Ev) lower than the reference exposure value (e.g., 0 Ev).

[0202] For example, the electronic device may acquire the single-exposure image frame (e.g., 1120 or 1121) acquired within the closest time from the time the capture input (1101) is received as the image frame to be synthesized as high dynamic range (HDR). For example, the electronic device may acquire the single-exposure image frame (1121) from the buffer in response to identifying that a first time from the time the capture input (1101) is acquired when the single-exposure image frame (1120) is acquired is longer than a second time from the time the capture input (1101) is acquired.

[0203] According to one embodiment, at least one image frame among the image frames (1130) acquired before receiving the capture input (1101) may be brighter than a long exposure image frame (1110) acquired after receiving the capture input (1101). For example, at least one image frame among the long exposure image frames (1130) acquired before receiving the capture input (1101) may be an image frame acquired while the exposure value is converging to a reference exposure value.

[0204] According to one embodiment, the electronic device can identify the first preview image frame based on the result of analyzing (1150) the face of the long exposure preview image frames (1130). For example, the electronic device can identify the first preview image frame based on the face data of the long exposure preview image frames (1130). For example, the electronic device can identify the first preview image frame among the long exposure preview image frames (1130) based on the face size data and / or the face brightness data.

[0205] According to one embodiment, the electronic device may determine (1160) a gain value based on a result of analyzing (1150) a face of long exposure preview image frames (1130). For example, the electronic device may determine (1160) a gain value based on a first preview image frame and a long exposure image frame (1110) identified based on face data. For example, the electronic device may determine (1160) a gain value based on a weighted sum of a face brightness value of the first preview image frame and a face brightness value of the long exposure image frame (1110). For example, the electronic device may determine (1160) a gain value based on a weighted sum of a face size value of the first preview image frame and a face size value of the long exposure image frame (1110). The operation of the electronic device determining (1160) a gain value may be analogized to the operation of the electronic device described above with reference to FIG. 9.

[0206] According to one embodiment, the electronic device may obtain a long exposure image frame (1111) to which a gain value is applied by applying (1170) a gain value to the long exposure image frame (1110). For example, the electronic device may apply (1170) a gain value determined (1160) based on a weighted sum of a face brightness value of a first preview image frame and a face brightness value of a long exposure image frame (1110) to the long exposure image frame (1110). For example, the electronic device may apply (1170) a gain value determined (1160) based on a weighted sum of a face size value of a first preview image frame and a face size value of a long exposure image frame (1110) to the long exposure image frame (1110).

[0207] According to one embodiment, the electronic device can perform high dynamic range (HDR) synthesis (1180). For example, the electronic device can generate an HDR image (1190) by synthesizing a long exposure image frame (1111) to which a gain value is applied and a short exposure image frame (1120). For example, the electronic device can generate an HDR image (1190) by synthesizing at least one frame among long exposure preview image frames (1130), a long exposure image frame (1111) to which a gain value is applied, and a short exposure image frame (1120). For example, the electronic device can generate the HDR image (1190) by using a massive multi frame (MMF) of an image signal processor included in the electronic device (e.g., the image signal processor (260) of FIG. 2).

[0208] For example, the electronic device can generate a high dynamic range (HDR) image (1190) by synthesizing a short exposure image frame (e.g., 1120 or 1121) acquired within a time closest to the time at which the capture input (1101) is received with a long exposure image frame (1111) to which a gain value is applied. For example, the electronic device can generate a high dynamic range (HDR) image (1190) by synthesizing a short exposure image frame (1121) acquired from a buffer with a long exposure image frame (1111) in response to identifying that a first time at which the short exposure image frame (1120) is acquired from the time at which the capture input (1101) is received is longer than a second time at which the short exposure image frame (1121) is acquired before the time at which the capture input (1101) is received.

[0209] According to one embodiment, the electronic device can set the number of long exposure preview image frames (1130) used to perform high dynamic range (HDR) synthesis.

[0210] For example, based on the shooting environment, the electronic device can set the number of long exposure preview image frames (1130) used to perform high dynamic range (HDR) synthesis. For example, based on identifying that it is a low-light shooting environment, the electronic device can set a larger number of long exposure preview image frames (1130) compared to a normal shooting environment. For example, in the case of a normal shooting environment, the electronic device can perform high dynamic range (HDR) synthesis using five long exposure preview image frames (1130). For example, based on identifying that it is a low-light shooting environment below a reference exposure value based on predetermined shooting parameters (e.g., aperture value, shutter speed, sensitivity), the electronic device can perform high dynamic range (HDR) synthesis using 13 long exposure preview image frames (1130).

[0211] For example, the electronic device can set the number of long exposure preview image frames (1130) used to perform high dynamic range (HDR) synthesis for each region of the image frame. For example, the electronic device can synthesize a plurality of long exposure preview image frames (1130) for dark portions of the image frame, and a smaller number of long exposure preview image frames (1130) for bright portions. For example, the electronic device can synthesize 13 long exposure preview image frames (1130) for dark portions of the image frame, and a single long exposure preview image frame (1130) for bright portions. According to one embodiment, the electronic device can perform image processing on a high dynamic range (HDR) image (1190). For example, when the brightness value of a second region (e.g., the second region (751) of the fourth image frame (750) of FIG. 7) for a main object of a high dynamic range (HDR) image (1190) is less than a preset brightness value, the electronic device may adjust the brightness value of the second region (e.g., the second region (751) of FIG. 7) so that the brightness value of the second region (e.g., the second region (751) of FIG. 7) corresponds to the preset brightness value.

[0212] According to the disclosed embodiment, the electronic device can generate a high dynamic range (HDR) image that includes a main object that is brighter than the high dynamic range (HDR) image generated by synthesizing a long exposure image frame (1110) and a short exposure image frame (1120).

[0213] FIG. 12 is a diagram illustrating an operation of performing high dynamic range (HDR) synthesis using images acquired before an electronic device receives a capture input, according to one embodiment. FIG. 12 is a diagram illustrating an operation of performing multi-frame synthesis by the electronic device (500) of FIG. 5 to acquire an image with a high dynamic range (HDR).

[0214] In one embodiment, the electronic device may receive a capture input (1201). For example, the electronic device may receive the capture input (1201) through a user interface (UI) provided by a camera application.

[0215] According to one embodiment, the electronic device can acquire image frames (1230a, 1230b, 1240a, 1240b) before the capture input (1201) is received.

[0216] For example, the electronic device can acquire a long exposure preview image frame (1230a) and a short exposure preview image frame (1240a) using a first camera module that supports a first field of view (e.g., wide angle). For example, the electronic device can alternately acquire the long exposure preview image frame (1230a) and the short exposure preview image frame (1240a). For example, the electronic device can acquire the long exposure preview image frame (1230a) and the short exposure preview image frame (1240a) so as to have a difference of a preset exposure value (e.g., 3 Ev).

[0217] For example, the electronic device can obtain a long exposure preview image frame (1230b) and a short exposure preview image frame (1240b) using a second camera module that supports a second field of view (e.g., telephoto). For example, the electronic device can obtain the long exposure preview image frame (1230b) and the short exposure preview image frame (1240b) alternately. For example, the electronic device can obtain the long exposure preview image frame (1230b) and the short exposure preview image frame (1240b) so as to have a difference of a preset exposure value (e.g., 3 Ev).

[0218] For example, the electronic device can acquire only long exposure image frames (1230b) using one camera module (e.g., the second camera module) among the camera modules (e.g., the first camera module, the second camera module), and can acquire only short exposure image frames (1240a) using the other camera (e.g., the first camera module).

[0219] For example, the electronic device can store image frames (1230a, 1230b, 1240a, 1240b). For example, the electronic device can store the image frames (1230a, 1230b, 1240a, 1240b) in a buffer.

[0220] According to one embodiment, the electronic device may analyze (1250) a face included in long exposure preview image frames (1230a, 1230b). For example, the electronic device may identify a face included in each of the long exposure preview image frames (1230a, 1230b). For example, the electronic device may obtain face data regarding a face included in the long exposure preview image frames (1230a, 1230b). For example, the electronic device may obtain face size data regarding the size of a face included in the long exposure preview image frames (1230a, 1230b). For example, the electronic device may obtain face brightness data regarding the brightness of a face included in the long exposure preview image frames (1230a, 1230b). The operation of the electronic device analyzing (1250) a face may be analogized to and applied to the operation of the electronic device described above with reference to FIG. 8.

[0221] In one embodiment, the electronic device may acquire image frames to be synthesized in high dynamic range (HDR) in response to receiving a capture input (1201). For example, the electronic device may acquire image frames (1210, 1220) after receiving the capture input (1201).

[0222] For example, the electronic device can acquire image frames (1210, 1220) acquired at substantially the same time as the time at which the capture input is received. For example, the electronic device can acquire a long exposure image frame (1210) based on a reference exposure value (e.g., 0 Ev). For example, the electronic device can acquire a short exposure image frame (1220) based on an exposure value (e.g., -3 Ev) lower than the reference exposure value (e.g., 0 Ev).

[0223] For example, the electronic device may acquire the single-exposure image frame (e.g., 1220 or 1221) acquired within the closest time from the time the capture input (1201) is received as the image frame to be synthesized into high dynamic range (HDR). For example, the electronic device may acquire the single-exposure image frame (1221) from the buffer in response to identifying that a first time from the time the capture input (1201) is acquired when the single-exposure image frame (1220) is acquired is longer than a second time from the time the capture input (1201) is acquired when the single-exposure image frame (1221) is acquired before the time the capture input (1201) is received.

[0224] According to one embodiment, at least one image frame among the image frames (1230a) acquired before receiving the capture input (1201) may be brighter than a long exposure image frame (1210) acquired after receiving the capture input (1201). For example, at least one image frame among the long exposure image frames (1230a) acquired before receiving the capture input (1201) may be an image frame acquired while the exposure value is converging to a reference exposure value.

[0225] According to one embodiment, the electronic device can identify the first preview image frame based on the result of analyzing (1250) the face of the long exposure preview image frames (1230a, 1230b). For example, the electronic device can identify the first preview image frame based on the face data of the long exposure preview image frames (1230a, 1230b). For example, the electronic device can identify the first preview image frame among the long exposure preview image frames (1230a, 1230b) based on the face size data and / or the face brightness data.

[0226] According to one embodiment, the electronic device may determine (1260) a gain value based on a result of analyzing (1250) a face of the long exposure preview image frames (1230a, 1230b). For example, the electronic device may determine (1260) a gain value based on a first preview image frame and a long exposure image frame (1210) identified based on facial data. For example, the electronic device may determine (1260) a gain value based on a weighted sum of a face brightness value of the first preview image frame and a face brightness value of the long exposure image frame (1210). For example, the electronic device may determine (1260) a gain value based on a weighted sum of a face size value of the first preview image frame and a face size value of the long exposure image frame (1210). The operation of the electronic device determining (1260) a gain value may be analogized to the operation of the electronic device described above with reference to FIG. 9.

[0227] According to one embodiment, the electronic device may obtain a long exposure image frame (1211) to which a gain value is applied by applying (1270) a gain value to the long exposure image frame (1210). For example, the electronic device may apply (1270) a gain value determined based on a weighted sum of a face brightness value of a first preview image frame and a face brightness value of a long exposure image frame (1210) to the long exposure image frame (1210). For example, the electronic device may apply (1270) a gain value determined based on a weighted sum of a face size value of a first preview image frame and a face size value of a long exposure image frame (1210) to the long exposure image frame (1210).

[0228] According to one embodiment, the electronic device can perform high dynamic range (HDR) synthesis (1280). For example, the electronic device can generate a high dynamic range (HDR) image (1290) by synthesizing a long exposure image frame (1211) to which a gain value is applied and a short exposure image frame (1220). For example, the electronic device can generate a high dynamic range (HDR) image (1290) by synthesizing at least one frame among long exposure preview image frames (1230a), a long exposure image frame (1211) to which a gain value is applied, and a short exposure image frame (1220). For example, the electronic device can generate the high dynamic range (HDR) image (1290) by using an image signal processor included in the electronic device (e.g., a massive multi frame (MMF) of the image signal processor (260) of FIG. 2).

[0229] For example, the electronic device can generate a high dynamic range (HDR) image (1290) by synthesizing a short exposure image frame (e.g., 1220 or 1221) acquired within a time closest to the time at which the capture input (1201) is received with a long exposure image frame (1211) to which a gain value is applied. For example, the electronic device can generate a high dynamic range (HDR) image (1290) by synthesizing a short exposure image frame (1221) acquired from a buffer with a long exposure image frame (1211) in response to identifying that a first time at which the short exposure image frame (1220) is acquired from the time at which the capture input (1201) is received is longer than a second time at which the short exposure image frame (1221) is acquired before the time at which the capture input (1201) is received.

[0230] According to one embodiment, the electronic device can set the number of long exposure preview image frames (1230a) used to perform high dynamic range (HDR) synthesis.

[0231] For example, based on the shooting environment, the electronic device can set the number of long exposure preview image frames (1230a) used to perform high dynamic range (HDR) synthesis. For example, based on identifying that it is a low-light shooting environment, the electronic device can set a larger number of long exposure preview image frames (1230a) than in a normal shooting environment. For example, in the case of a normal shooting environment, the electronic device can perform high dynamic range (HDR) synthesis using five long exposure preview image frames (1230a). For example, based on identifying that it is a low-light shooting environment below a reference exposure value based on predetermined shooting parameters (e.g., aperture value, shutter speed, sensitivity), the electronic device can perform high dynamic range (HDR) synthesis using 13 long exposure preview image frames (1230a).

[0232] For example, the electronic device can set the number of long exposure preview image frames (1230a) used to perform high dynamic range (HDR) synthesis for each region of the image frame. For example, the electronic device can synthesize a plurality of long exposure preview image frames (1230a) for a dark portion of the image frame, and a small number of long exposure preview image frames (1230a) for a bright portion. For example, the electronic device can synthesize 13 long exposure preview image frames (1230a) for a dark portion of the image frame, and a single long exposure preview image frame (1230a) for a bright portion.

[0233] According to one embodiment, the electronic device can perform image processing on a high dynamic range (HDR) image (1290). For example, when a brightness value of a second region (e.g., the second region (751) of FIG. 7) for a main object of the high dynamic range (HDR) image (1290) is less than a preset brightness value, the electronic device can adjust the brightness value of the second region (e.g., the second region (751) of FIG. 7) so that the brightness value of the second region (e.g., the second region (751) of FIG. 7) corresponds to the preset brightness value.

[0234] According to the disclosed embodiment, the electronic device can generate a high dynamic range (HDR) image that includes a main object that is brighter than a high dynamic range (HDR) image generated by synthesizing a long exposure image frame (1210) and a short exposure image frame (1220).

[0235] FIG. 13 is a flowchart illustrating an operation of performing high dynamic range (HDR) synthesis using images acquired by an electronic device before receiving a capture input, according to one embodiment. FIG. 13 is a diagram illustrating an operation of the electronic device (500) of FIG. 5 acquiring an HDR image. Each operation of FIG. 13 may be performed by the processor (120) of FIG. 1. At least one operation among the operations of FIG. 13 may be omitted, and an operation not illustrated may be added.

[0236] Referring to operation 1310, the electronic device may acquire multiple images and perform face detection on the acquired images. For example, the electronic device may acquire preview images. For example, the electronic device may acquire long-exposure preview images and short-exposure preview images based on a multi-frame composite shooting mode. For example, the electronic device may perform face detection on each of the preview images. Operation 1310 of the electronic device may be analogously applied to the operation described above with reference to FIG. 8. Duplicate details are omitted.

[0237] Referring to operation 1320, the electronic device may obtain first face data regarding a face detected from each of the plurality of images. For example, the electronic device may obtain first face data including face size data regarding the size of each face in the long-exposure preview images and face brightness data regarding the brightness of the face. Operation 1320 of the electronic device may be analogously applied to the operation described above with reference to FIG. 8. Duplicate details are omitted.

[0238] Referring to operation 1330, the electronic device may receive a capture input from a user. For example, the electronic device may receive the capture input through a user interface (UI) provided by a camera application.

[0239] Referring to operation 1340, the electronic device can identify a first image and second facial data of the first image based on first facial data.

[0240] For example, the electronic device may identify the first image based on first facial data about faces in the preview images. For example, the electronic device may identify the image containing the largest face among the preview images as the first image based on the first facial data. For example, the electronic device may identify the image frame containing the brightest face among the preview images as the first image based on the first facial data.

[0241] For example, the electronic device may identify second facial data regarding a face included in a first image. For example, the electronic device may identify facial size data regarding the size of the face in the first image. For example, the electronic device may identify facial brightness data regarding the brightness of the face in the first image.

[0242] Referring to operation 1350, the electronic device can acquire a second image and a third image using a camera module. For example, the electronic device can acquire a second image, which is a long exposure image, based on a reference exposure value (e.g., 0 Ev) in response to a capture input. For example, the electronic device can acquire a third image, which is a short exposure image, based on an exposure value (e.g., -3 Ev) lower than the reference exposure value (e.g., 0 Ev). For example, the electronic device can acquire the second image and the third image acquired at substantially the same time as the time at which the capture input is received.

[0243] Referring to operation 1360, the electronic device may perform face detection on the second image, thereby obtaining third face data regarding the face included in the second image. For example, the electronic device may obtain third face data including face size data regarding the size of the face included in the second image and face brightness data regarding the brightness of the face. Operation 1360 of the electronic device may be analogized to the operation of the electronic device described above with reference to FIG. 8. Duplicate details are omitted.

[0244] Referring to operation 1370, the electronic device may obtain a first gain value for the second image based on the second face data and the third face data. For example, the electronic device may obtain the first gain value based on a weighted sum of a face brightness value of a first image included in the second face data and a face brightness value of a second image included in the third face data. For example, the electronic device may obtain the first gain value based on a weighted sum of a face size value of a first image included in the second face data and a face size value of a second image included in the third face data. Operation 1370 of the electronic device may be analogized to and applied to the operation of the electronic device described above with reference to FIG. 9. Duplicate details are omitted.

[0245] Referring to operation 1380, the electronic device may obtain a high dynamic range (HDR) image using the second image and the third image to which the first gain value is applied. For example, the electronic device may apply the first gain value obtained through operation 1370 to the second image. For example, the electronic device may perform multi-frame synthesis based on the third image and the second image to which the first gain value is applied. For example, the electronic device may generate a high dynamic range image using pixel values ​​of a dark area of ​​the second image to which the first gain value is applied (e.g., the second area (731) of the third image frame (730) of FIG. 7) and pixel values ​​of a bright area of ​​the third image (e.g., the first area (742) of the first image frame (740) of FIG. 7). For example, the electronic device may perform image processing to adjust at least some brightness values ​​of the generated high dynamic range image.

[0246] According to one embodiment, the electronic device may omit operation 1370 based on determining that the detected face does not correspond to the primary object, even if a face is detected from the second image in operation 1360. If the electronic device determines that the detected face does not correspond to the primary object, the electronic device may obtain a high dynamic range (HDR) image by performing multi-frame synthesis of the second image without the first gain value applied and the third image.

[0247] Referring to operation 1390, the electronic device may display a high dynamic range (HDR) image. For example, the electronic device may display the high dynamic range image generated in operation 1380.

[0248] In one embodiment, the operation of the electronic device performing High Dynamic Range (HDR) synthesis may be enabled or disabled based on a user's settings. For example, the electronic device may enable or disable the High Dynamic Range (HDR) synthesis described above with reference to FIG. 13 based on a user's input selecting enable and disable in the settings screen of a camera application. For example, the electronic device may not generate an HDR image in a backlit situation based on a user's setting to disable High Dynamic Range (HDR) synthesis. In this case, the electronic device may obtain an image in which bright areas are saturated or an image in which the main object is captured darkly.

[0249] FIGS. 14A to 14E are diagrams illustrating operations of an electronic device performing enhanced high dynamic range (HDR) synthesis using a camera application according to one embodiment. FIGS. 14A to 14E are diagrams illustrating operations of the electronic device (500) of FIG. 5 acquiring an image in a high dynamic range (HDR). Each of the operations of FIGS. 14A to 14E may be performed by the processor (120) of FIG. 1.

[0250] Referring to FIG. 14A, an electronic device (1400) according to one embodiment may display at least one object (1411, 1412, 1413, 1414). The objects (1411, 1412, 1413, 1414) may be objects related to an application executed by the electronic device (1400). For example, the objects (1411, 1412, 1413, 1414) may include icons. For example, the objects (1411, 1412, 1413, 1414) may include widgets.

[0251] According to one embodiment, the electronic device (1400) may receive an input from a user (1490) selecting at least one object (1411, 1412, 1413, 1414). For example, the electronic device (1400) may receive a touch input from a user (1490) selecting a first object (1414) corresponding to a camera application.

[0252] In one embodiment, the electronic device (1400) may perform a defined action in response to an input from a user (1490). For example, the electronic device (1400) may execute a camera application in response to a touch input from a user (1490) selecting a first object.

[0253] Referring to FIG. 14B, an electronic device (1400) according to one embodiment can display a preview image. For example, the electronic device (1400) can acquire images using a camera (e.g., the camera module (180) of FIG. 1) by executing a camera application. For example, the electronic device (1400) can display images acquired using a camera (e.g., the camera module (180) of FIG. 1).

[0254] According to one embodiment, the electronic device (1400) can receive an input from a user (1490) for a preview image (1420). For example, the electronic device (1400) can receive a touch input from the user (1490) for setting a region of interest (1430) among the preview image (1420). For example, the electronic device (1400) can set the region of interest (1430) on the preview image (1420) based on the touch input from the user (1490). For example, the electronic device (1400) can set an area where the touch input from the user (1490) is received as the region of interest (1430). For example, the electronic device (1400) can perform object identification based on the touch input from the user (1490) and thereby set an area corresponding to the face of the subject as the region of interest (1430).

[0255] In one embodiment, the electronic device (1400) can track a region of interest (1430). For example, the electronic device (1400) can track the region of interest (1430) by analyzing images acquired using a camera (e.g., the camera module (180) of FIG. 1). For example, the electronic device (1400) can track the region of interest (1430) so that the camera (e.g., 180 of FIG. 1) focuses on the region of interest (1430).

[0256] Referring to FIG. 14C, an electronic device (1400) according to one embodiment may receive an input of a user (1490) for setting a camera. For example, the electronic device (1400) may receive a touch input of a user (1490) for manipulating a UI (e.g., a toggle switch, a button, an icon, a check box). For example, the electronic device (1400) may receive a touch input of a user (1490) regarding setting a function (1441) for acquiring a high dynamic range image. For example, the electronic device (1400) may receive a touch input of a user (1490) regarding setting a function (1442) for acquiring an enhanced high dynamic range image. The function for acquiring an enhanced high dynamic range image may include a function by an operation for acquiring a high dynamic range image described above with reference to FIGS. 7 to 13.

[0257] Referring to FIG. 14D , an electronic device (1400) according to an embodiment may indicate that a function for acquiring a high dynamic range image is operating. For example, the electronic device (1400) may display text (1451) indicating that the function for acquiring a high dynamic range image is operating. For example, the electronic device (1400) may display an indicator (1452) indicating that the function for acquiring a high dynamic range image is operating. For example, the electronic device (1400) may display the text (1451) and / or the indicator (1452) together with a preview image (1420). For example, the electronic device (1400) may display the text (1451) and / or the indicator (1452) so as to be overlaid on at least a portion of the preview image (1420).

[0258] According to one embodiment, the electronic device (1400) may indicate that a function for acquiring an enhanced high dynamic range image is activated.

[0259] For example, the electronic device (1400) may display text indicating that the function of acquiring an enhanced high dynamic range image is in operation. For example, the electronic device (1400) may display first text indicating that the function of acquiring a high dynamic range image is in operation and second text indicating that the function of acquiring an enhanced high dynamic range image is in operation in a different color. For example, the electronic device (1400) may display second text indicating that the brightness has been adjusted for a region of interest (1430) (e.g., a region corresponding to a face).

[0260] For example, the electronic device (1400) may display an indicator indicating that the function of acquiring an enhanced high dynamic range image is operating. For example, the electronic device (1400) may display a first indicator indicating that the function of acquiring a high dynamic range image is operating and a second indicator (e.g., an icon, an alarm) having a different shape and / or color from the first indicator indicating that the function of acquiring a high dynamic range image is operating. For example, the electronic device (1400) may display a second indicator indicating that the brightness has been adjusted for a region of interest (e.g., a region corresponding to a face).

[0261] For example, the electronic device (1400) may display a second text and / or a second indicator together with a preview image (1420). For example, the electronic device (1400) may display the second text and / or the second indicator so as to be overlaid on at least a portion of the preview image (1420).

[0262] In one embodiment, the electronic device (1400) may stop the function of acquiring a high dynamic range image and / or an enhanced high dynamic range image. For example, the electronic device (1400) may stop the function of acquiring a high dynamic range image and / or an enhanced high dynamic range image when the illumination of the shooting environment changes. For example, the electronic device (1400) may stop the function of acquiring an enhanced high dynamic range image based on a determination that the face of the subject is not identifiable from the preview image.

[0263] In one embodiment, the electronic device (1400) may not display text and / or indicators indicating that the function is operating based on the cessation of the function of acquiring high dynamic range images and / or enhanced high dynamic range images. For example, the electronic device (1400) may stop displaying text and / or indicators based on the determination that the illumination of the photographing environment has changed. For example, the electronic device (1400) may stop displaying text and / or indicators based on the determination that the face of the subject is not identifiable from the preview image.

[0264] Referring to FIG. 14E, an electronic device (1400) according to an embodiment may acquire a high dynamic range image based on a function for acquiring a high dynamic range image being set. For example, the electronic device (1400) may acquire a long exposure image frame and a short exposure frame based on a difference in a defined exposure value. For example, the electronic device (1400) may acquire a long exposure image frame and a short exposure frame alternately. For example, the electronic device (1400) may acquire a high dynamic range image by synthesizing the long exposure image frame and the short exposure frame acquired alternately. For example, in response to the function for acquiring a high dynamic range image (e.g., the function for acquiring a high dynamic range image (1441) of FIG. 14C) being set, the electronic device (1400) may perform the operation of the electronic device (500) described above with reference to FIG. 6. For example, in response to the electronic device (1400) being set to have a function (1442) for acquiring an enhanced high dynamic range image, the operation of the electronic device (1400) described above with reference to FIGS. 7 to 13 may be performed.

[0265] In one embodiment, the electronic device (1400) can display a high dynamic range image (1460). For example, the electronic device (1400) can display the high dynamic range image (1460) as a preview image. For example, the electronic device (1400) can display the high dynamic range image (1460) as a review image.

[0266] Fig. 15 illustrates an example of a block diagram of a wearable device according to one embodiment. The camera (1525) of the wearable device (103) of Fig. 15 may be applied with embodiments of the camera modules (180, 380) described above with reference to Figs. 2 to 4. The embodiments of Figs. 5 to 14e may be implemented by the wearable device (103). The wearable device (103) may correspond to an embodiment of the electronic device (101) of Fig. 1.

[0267] Referring to FIG. 15, a wearable device (103) according to one embodiment may include at least one of a processor (1510), a memory (1515), a display (1520), a camera (1525), a sensor (1530), or a communication circuit (1535). The processor (1510), the memory (1515), the display (1520), the camera (1525), the sensor (1530), and the communication circuit (1535) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus (1502). The type and / or number of hardware components included in the wearable device (103) are not limited to those illustrated in FIG. 15. For example, the wearable device (103) may include only some of the hardware components illustrated in FIG. 15. The elements (e.g., layers and / or modules) within the memory described below may be logically separated. The elements within the memory (1515) may be included within a hardware component that is separate from the memory (1515). The operation performed by the processor (1510) using each element within the memory (1515) is one embodiment, and the processor (1510) may perform a different operation from the above operation through at least one of the elements within the memory (1515).

[0268] A processor (1510) of a wearable device (103) according to one embodiment may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors (1510) may be one or more. For example, the processor (1510) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core.

[0269] A memory (1515) of a wearable device (103) according to one embodiment may include a hardware component for storing data and / or instructions input and / or output to a processor (1510). The memory (1515) may include, for example, a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multimedia card (eMMC).

[0270] In one embodiment, a display (1520) of a wearable device (103) can output visualized information to a user of the wearable device (103). For example, the display (1520) can be controlled by a processor (1510) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display (1520) can include a flat panel display (FPD) and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs).

[0271] In one embodiment, the camera (1525) of the wearable device (103) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The plurality of optical sensors included in the camera (1525) may be arranged in the form of a two-dimensional array. The camera (1525) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously to generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (1525) may mean one (a) two-dimensional frame data acquired from the camera (1525). For example, video data captured using the camera (1525) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (1525) according to a frame rate. The camera (1525) may further include a flash light positioned toward the direction in which the camera (1525) receives light and outputs light toward the direction.

[0272] According to one embodiment, the wearable device (103) may include a plurality of cameras, for example, cameras (1525), arranged in different directions. A first camera among the plurality of cameras may be referred to as a motion recognition camera (e.g., motion recognition cameras 1660-2 and 1660-3 of FIG. 16B ), and a second camera may be referred to as a gaze tracking camera (e.g., gaze tracking camera 1660-1 of FIG. 16B ). The wearable device (103) may identify a position, shape, and / or gesture of a hand using an image acquired using the first camera. The wearable device (103) may identify a direction of a gaze of a user wearing the wearable device (103) using an image acquired using the second camera. For example, the direction in which the first camera faces may be opposite to the direction in which the second camera faces.

[0273] According to one embodiment, a sensor (1530) of a wearable device (103) may generate electrical information that may be processed by a processor (1510) and / or a memory (1515) of the wearable device (103) from non-electronic information related to the wearable device (103). The information may be referred to as sensor data. The sensor (1530) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (103), and an inertial measurement unit (IMU) for detecting a physical motion of the wearable device (103).

[0274] In one embodiment, the communication circuit (1535) of the wearable device (103) may include hardware components for supporting transmission and / or reception of electrical signals between the wearable device (103) and an external electronic device. The communication circuit (1535) may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (1535) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), and / or 6G.

[0275] According to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (1510) of the wearable device (103) may be stored in the memory (1515) of the wearable device (103). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the wearable device (103) and / or the processor (1510) may perform at least one of the operations described above with reference to FIG. 5 or FIG. 14 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. Hereinafter, the fact that an application is installed in a wearable device (103) may mean that one or more instructions provided in the form of an application are stored in a memory (1515), and that the one or more applications are stored in a format executable by the processor (1510) (e.g., a file having an extension specified by the operating system of the wearable device (103)). For example, the application may include a program and / or a library related to a service provided to a user.

[0276] Referring to FIG. 15, programs installed in the wearable device (103) may be classified into any one of different layers, including an application layer (1540), a framework layer (1550), and / or a hardware abstraction layer (HAL) (1580), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware of the wearable device (103) (e.g., a display (1520), a camera (1525), and / or a sensor (1530)) may be classified within the hardware abstraction layer (1580). The framework layer (1550) may be referred to as an XR framework layer in that it includes one or more programs for providing an XR (extended reality) service. For example, FIG. 15 illustrates layers within the memory (1515) as being divided, but the layers may be logically divided. However, the present invention is not limited thereto. Depending on the embodiment, the layers may be stored in a designated area within the memory (1515).

[0277] For example, within the framework layer (1550), programs designed to target at least one of the hardware abstraction layer (1580) and / or the application layer (1540) (e.g., a position tracker (1571), a space recognizer (1572), a gesture tracker (1573), an eye-gaze tracker (1574), and / or a face tracker (1575)) may be classified. Programs classified within the framework layer (1550) may provide an executable API (application programming interface) based on other programs.

[0278] For example, within the application layer (1540), programs designed to target users controlling wearable devices (103) may be classified. Examples of programs classified within the application layer (1540) include, but are not limited to, an XR (extended reality) system user interface (UI) and / or an XR application (1542). For example, programs (e.g., software applications) classified within the application layer (1540) may call an API (application programming interface) to cause execution of functions supported by programs classified within the framework layer (1550).

[0279] For example, the wearable device (103) may display one or more visual objects on the display (1520) for performing interaction with a user for using a virtual space based on the execution of the XR system UI (1541). A visual object may refer to an object that can be deployed on a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (103) may provide a service for controlling functions available in a virtual space to the user based on the execution of the XR system UI (1541).

[0280] Referring to FIG. 15, a lightweight renderer (1543) and / or an XR plug-in (1544) are illustrated as being included within the XR system UI (1541), but are not limited thereto. For example, the XR system UI (1541) may cause execution of a function supported by the lightweight renderer (1543) and / or the XR plug-in (1544) included within the application layer (1540).

[0281] For example, the wearable device (103) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (1543). The lightweight renderer (1543) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (1543) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (103) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (1544). The XR plugin (1544) can be referred to as an open XR native client from the perspective of defining (or configuring) the entire rendering pipeline.

[0282] For example, the wearable device (103) may display a screen representing at least a portion of a virtual space on the display (1520) based on the execution of the XR application (1542). The XR plug-in (1544-1) included in the XR application (1542) may be referenced by the XR plug-in (1544) of the XR system UI (1541). Descriptions of the XR plug-in (1544-1) that overlap with the description of the XR plug-in (1544) may be omitted. The wearable device (103) may cause the execution of the screen composition manager (1551) based on the execution of the XR application (1542).

[0283] According to one embodiment, the wearable device (101) may provide a virtual space service based on the execution of the screen composition manager (1551). For example, the screen composition manager (1551) may include a platform (e.g., an Android platform) for supporting the virtual space service. The wearable device (103) may display the posture of a virtual object representing the user's posture rendered using data acquired through a sensor (1530) based on the execution of the screen composition manager (1551) on the display. The screen composition manager (1551) may be referred to as a composition presentation manager (CPM).

[0284] For example, the screen composition manager (1551) may include a runtime service (1552). As an example, the runtime service (1552) may be referred to as an OpenXR runtime module. The wearable device (103) may be used to provide at least one of a pose prediction function, a frame timing function, and / or a spatial input function to a user through the wearable device (103) based on the execution of the runtime service (1552). As an example, the wearable device (103) may be used to perform rendering for a virtual space service to a user based on the execution of the runtime service (1552). For example, an application (e.g., unity or an OpenXR native application) may be implemented based on the execution of the runtime service (1552).

[0285] For example, the screen configuration manager (1551) may include a pass-through library (1553). Based on the execution of the pass-through library (1553), the wearable device (103) may display a screen representing a virtual space on the display (1520), while displaying another screen representing a real space acquired through the camera (1525) by overlaying at least a portion of the screen.

[0286] For example, the screen composition manager (1551) may include a renderer (not shown). The wearable device (101) may render a screen to be displayed on the display by compositing virtual layers (or virtual nodes) rendered based on sensor data (e.g., sensing data acquired through a camera (1525) or a sensor (1530)) and pass-through layers (or pass-through nodes) acquired through a pass-through library (1553) using the screen composition manager (1551). The virtual layers may be referred to as virtual nodes and / or virtual surfaces. The wearable device (101) may render each of the virtual layers or all of the virtual layers through the screen composition manager (1551).

[0287] For example, the screen configuration manager (1551) may include an input manager (1554). Based on the execution of the input manager (1554), the wearable device (103) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (1570). The wearable device (103) may initiate execution of at least one of the functions of the wearable device (103) using the acquired data.

[0288] For example, the perception abstract layer (1560) can be used for data exchange between the screen composition manager (1551) and the perception service layer (1570). From the perspective of being used for data exchange between the screen composition manager (1551) and the perception service layer (1570), the perception abstract layer (1560) can be referred to as an interface. For example, the perception abstract layer (1560) can be referred to as OpenPX and / or PPAL (perception platform abstract layer). The perception abstract layer (1560) can be used for a perception client and a perception service.

[0289] According to one embodiment, the recognition service layer (1570) may include one or more programs for processing data acquired from a sensor (1530) (or a camera (1525)). The one or more programs may include at least one of a position tracker (1571), a space recognizer (1572), a gesture tracker (1573), an eye tracker (1574), and / or a face tracker (1575). The type and / or number of the one or more programs included in the recognition service layer (1570) are not limited to those illustrated in FIG. 15.

[0290] For example, the wearable device (103) can identify the pose of the wearable device (103) using the sensor (1530) based on the execution of the position tracker (1571). The wearable device (103) can identify the 6 degrees of freedom pose (6 DOF pose) of the wearable device (103) using data acquired using the camera (1525) and the IMU based on the execution of the position tracker (1571). The position tracker (1571) can be referred to as a head tracking (HeT) module.

[0291] For example, the wearable device (103) may be used to construct a three-dimensional virtual space surrounding the wearable device (103) (or a user of the wearable device (103)) based on the execution of the space recognizer (1572). The wearable device (103) may reconstruct the three-dimensional surroundings of the wearable device (103) using data acquired using the camera (1525) based on the execution of the space recognizer (1572). The wearable device (103) may identify at least one of a plane, a slope, or stairs based on the three-dimensionally reconstructed surroundings of the wearable device (103) based on the execution of the space recognizer (1572). The space recognizer (1572) may be referred to as a scene understanding (SU) module.

[0292] For example, the wearable device (103) may be used to identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (103) based on the execution of the gesture tracker (1573). As an example, the wearable device (103) may identify a pose and / or gesture of a hand of a user using data acquired from a sensor (1530) based on the execution of the gesture tracker (1573). As an example, the wearable device (103) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using a camera (1525) based on the execution of the gesture tracker (1573). The gesture tracker (1573) may be referred to as a hand tracking (HaT) module and / or a gesture tracking module.

[0293] For example, the wearable device (103) may identify (or track) eye movements of a user of the wearable device (103) based on the execution of the gaze tracker (1574). As an example, the wearable device (103) may identify eye movements of the user using data acquired from at least one sensor based on the execution of the gaze tracker (1574). As an example, the wearable device (103) may identify eye movements of the user based on data acquired using a camera (1525) (e.g., the gaze tracking camera (1660-1) of FIG. 16B) and / or an infrared light emitting diode (IR LED) based on the execution of the gaze tracker (1574). The gaze tracker (1574) may be referred to as an eye tracking (ET) module and / or a gaze tracking module.

[0294] For example, the recognition service layer (1570) of the wearable device (103) may further include a face tracker (1575) for tracking the user's face. For example, the wearable device (103) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (1575). The wearable device (103) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (1575). As an example, the wearable device (103) may identify the movement of the user's face and / or the user's expression based on data (e.g., an image) acquired using a camera based on the execution of the face tracker (1575).

[0295] FIG. 16A illustrates an example of a perspective view of a wearable device according to one embodiment. The wearable device (103) of FIG. 16A may correspond to an embodiment of the wearable device (103) described above with reference to FIG. 15.

[0296] According to one embodiment, the wearable device (103) may have the form of glasses that are wearable on a body part of the user (e.g., the head). The wearable device (103) may include a head-mounted display (HMD). For example, the housing of the wearable device (103) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (103) may include one or more straps that are capable of being twined around the user's head, and / or one or more temples that are detachably attachable to the ears of the head.

[0297] Referring to FIG. 16A, according to one embodiment, a wearable device (103) may include at least one display (1650) and a frame (1600) supporting at least one display (1650).

[0298] According to one embodiment, the wearable device (103) can be worn on a part of a user's body. The wearable device (103) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to the user wearing the wearable device (103). For example, the wearable device (103) can display a virtual reality image provided from at least one optical device (1682, 1684) of FIG. 16B on at least one display (1650) in response to a user's designated gesture acquired through a motion recognition camera (or motion tracking camera) (1660-2, 1660-3) of FIG. 16B.

[0299] According to one embodiment, at least one display (1650) may provide visual information to a user. For example, at least one display (1650) may include a transparent or translucent lens. At least one display (1650) may include a first display (1650-1) and / or a second display (1650-2) spaced apart from the first display (1650-1). For example, the first display (1650-1) and the second display (1650-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0300] Referring to FIG. 16B, at least one display (1650) can provide visual information transmitted from external light to a user through a lens included in the at least one display (1650), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, the at least one display (1650) can include a first surface (1631) and a second surface (1632) opposite to the first surface (1631). A display area can be formed on the second surface (1632) of the at least one display (1650). When a user wears the wearable device (103), external light can be transmitted to the user by being incident on the first surface (1631) and transmitted through the second surface (1632). As another example, at least one display (1650) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (1682, 1684) on a real screen transmitted through external light, in a display area formed on the second surface (1632).

[0301] In one embodiment, at least one display (1650) may include at least one waveguide (1633, 1634) that diffracts light emitted from at least one optical device (1682, 1684) and transmits the diffracted light to a user. The at least one waveguide (1633, 1634) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (1633, 1634). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (1633, 1634) may be propagated to the other end of the at least one waveguide (1633, 1634) by the nano-pattern. At least one waveguide (1633, 1634) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)), or at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (1633, 1634) may be arranged within the wearable device (103) to guide a screen displayed by at least one display (1650) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (1633, 1634).

[0302] The wearable device (103) can analyze an object included in a real image collected through a shooting camera (1660-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (1650). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (103) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (103) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using the multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (103) can view an image displayed on at least one display (1650).

[0303] According to one embodiment, the frame (1600) may be formed as a physical structure that allows the wearable device (103) to be worn on the user's body. According to one embodiment, the frame (1600) may be configured so that, when the user wears the wearable device (103), the first display (1650-1) and the second display (1650-2) can be positioned corresponding to the user's left and right eyes. The frame (1600) may support at least one display (1650). For example, the frame (1600) may support the first display (1650-1) and the second display (1650-2) to be positioned corresponding to the user's left and right eyes.

[0304] Referring to FIG. 16A, the frame (1600) may include a region (1620) that at least partially contacts a portion of the user's body when the user wears the wearable device (103). For example, the region (1620) of the frame (1600) that contacts a portion of the user's body may include a region that contacts a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (103) makes contact with. According to one embodiment, the frame (1600) may include a nose pad (1610) that contacts a portion of the user's body. When the wearable device (103) is worn by the user, the nose pad (1610) may contact a portion of the user's nose. The frame (1600) may include a first temple (1604) and a second temple (1605) that contact another part of the user's body that is distinct from the part of the user's body.

[0305] For example, the frame (1600) may include a first rim (1601) that surrounds at least a portion of the first display (1650-1), a second rim (1602) that surrounds at least a portion of the second display (1650-2), a bridge (1603) that is disposed between the first rim (1601) and the second rim (1602), a first pad (1611) that is disposed along a portion of the edge of the first rim (1601) from one end of the bridge (1603), a second pad (1612) that is disposed along a portion of the edge of the second rim (1602) from the other end of the bridge (1603), a first temple (1604) that extends from the first rim (1601) and is fixed to a portion of the ear of the wearer, and a second temple (1605) that extends from the second rim (1602) and is fixed to a portion of the ear opposite the ear. There are. The first pad (1611) and the second pad (1612) can be in contact with a part of the user's nose, and the first temple (1604) and the second temple (1605) can be in contact with a part of the user's face and a part of the user's ear. The temples (1604, 1605) can be rotatably connected to the rim through the hinge units (1606, 1607) of FIG. 16B. The first temple (1604) can be rotatably connected to the first rim (1601) through the first hinge unit (1606) disposed between the first rim (1601) and the first temple (1604). The second temple (1605) may be rotatably connected to the second rim (1602) via a second hinge unit (1607) disposed between the second rim (1602) and the second temple (1605). In one embodiment, the wearable device (103) may use a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (1600) to identify an external object (e.g., a user's fingertip) touching the frame (1600) and / or a gesture performed by the external object.

[0306] According to one embodiment, the wearable device (103) may include hardware that performs various functions. For example, the hardware may include a battery module (1670), an antenna module (1675), at least one optical device (1682, 1684), speakers (e.g., speakers (1655-1, 1655-2)), a microphone (e.g., microphones (1665-1, 1665-2, 1665-3)), a light-emitting module (not shown), and / or a printed circuit board (PCB) (1690) (e.g., a printed circuit board). The various hardware may be arranged within the frame (1600).

[0307] According to one embodiment, microphones (e.g., microphones 1665-1, 1665-2, 1665-3) of the wearable device (103) may be disposed on at least a portion of the frame (1600) to acquire sound signals. A first microphone (1665-1) disposed on the bridge (1603), a second microphone (1665-2) disposed on the second rim (1602), and a third microphone (1665-3) disposed on the first rim (1601) are illustrated in FIG. 16B, but the number and arrangement of the microphones (1665) are not limited to the embodiment of FIG. 16B. When the number of microphones (1665) included in the wearable device (103) is two or more, the wearable device (103) can identify the direction of a sound signal by using a plurality of microphones arranged on different parts of the frame (1600).

[0308] According to one embodiment, at least one optical device (1682, 1684) may project a virtual object onto at least one display (1650) to provide various image information to a user. For example, at least one optical device (1682, 1684) may be a projector. At least one optical device (1682, 1684) may be disposed adjacent to at least one display (1650) or may be included within at least one display (1650) as a part of at least one display (1650). According to one embodiment, the wearable device (103) may include a first optical device (1682) corresponding to a first display (1650-1) and a second optical device (1684) corresponding to a second display (1650-2). For example, at least one optical device (1682, 1684) may include a first optical device (1682) positioned at an edge of a first display (1650-1) and a second optical device (1684) positioned at an edge of a second display (1650-2). The first optical device (1682) may transmit light to a first waveguide (1633) positioned on the first display (1650-1), and the second optical device (1684) may transmit light to a second waveguide (1634) positioned on the second display (1650-2).

[0309] In one embodiment, the camera (1660) may include a recording camera (1660-4), an eye tracking camera (ET CAM) (1660-1), and / or a motion recognition camera (1660-2, 1660-3). The recording camera (1660-4), the eye tracking camera (1660-1), and the motion recognition cameras (1660-2, 1660-3) may be positioned at different locations on the frame (1600) and may perform different functions. The eye tracking camera (1660-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (103). For example, the wearable device (103) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (1660-1). The wearable device (103) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (1660-1). The wearable device (103) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (103) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (1660-1). The wearable device (103) can render an image (or screen) displayed on at least one display (1650) based on the position of the user's eyes. For example, the visual quality of a first area related to the gaze within the image and the visual quality (e.g., resolution, brightness, saturation, grayscale, or PPI (pixels per inch)) of a second area distinguished from the first area may be different from each other.The wearable device (103) can obtain an image (or screen) having a visual quality of a first area matching the user's gaze and a visual quality of a second area using foveated rendering. For example, if the wearable device (103) supports an iris recognition function, user authentication can be performed based on iris information obtained using a gaze tracking camera (1660-1). An example in which the gaze tracking camera (1660-1) is positioned toward both eyes of the user is illustrated in FIG. 16B, but the embodiment is not limited thereto, and the gaze tracking camera (1660-1) can be positioned solely toward the user's left eye or right eye.

[0310] In one embodiment, the capturing camera (1660-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (1660-4) can be used to obtain a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (1660-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (1650). The at least one display (1650) can display a single image in which information about an actual image or background including an image of the specific object obtained using the capturing camera (1660-4) and a virtual image provided through at least one optical device (1682, 1684) are superimposed. The wearable device (103) can compensate for depth information (e.g., the distance between the wearable device (103) and an external object acquired through a depth sensor) using an image acquired through the capture camera (1660-4). The wearable device (103) can perform object recognition using an image acquired using the capture camera (1660-4). The wearable device (103) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capture camera (1660-4). The wearable device (103) can perform a pass-through function to display an image acquired through the capture camera (1660-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (1650) while displaying a screen. The shooting camera (1660-4) may be referred to as a high resolution (HR) camera or a photo video (PV) camera.The camera (1660-4) may provide auto focus (AF) and optical image stabilization (OIS) functions. The camera (1660-4) may include a global shutter (GS) camera and / or a rolling shutter (RS) camera. In one embodiment, the camera (1660-4) may be positioned on a bridge (1603) positioned between the first rim (1601) and the second rim (1602).

[0311] The gaze tracking camera (1660-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (103) and matching the user's gaze with visual information provided to at least one display (1650). For example, when the wearable device (103) looks straight ahead, the wearable device (103) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (1650). The gaze tracking camera (1660-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (1660-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (1660-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (1660-1) may be positioned within the first rim (1601) and / or the second rim (1602) to face the direction in which the user wearing the wearable device (103) is positioned.

[0312] The gesture recognition cameras (1660-2, 1660-3) can recognize the movement of the user's entire body, such as the user's torso, hands, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (1650). The gesture recognition cameras (1660-2, 1660-3) can recognize the user's gesture (gesture recognition), obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (1650). The processor can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition cameras (1660-2, 1660-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 DOF pose). The processor may perform gesture recognition and / or object tracking functions using the motion recognition cameras (1660-2, 1660-3). In one embodiment, the motion recognition cameras (1660-2, 1660-3) may be disposed on the first limb (1601) and / or the second limb (1602). The motion recognition cameras (1660-2, 1660-3) may include a global shutter (GS) camera (e.g., a global shutter (GS) camera) used for head tracking, hand tracking, and / or spatial recognition based on one of a three-degree-of-freedom pose or a six-degree-of-freedom pose. The GS camera may include two or more stereo cameras to track fine movements. As an example, the GS camera may be included in the gaze tracking camera (1660-1) for tracking the gaze of a user.

[0313] The camera (1660) included in the wearable device (103) is not limited to the above-described gaze tracking camera (1660-1) and motion recognition cameras (1660-2, 1660-3). For example, the wearable device (103) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (103) can identify an external object based on a sensor for identifying the distance between the wearable device (103) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (1660) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (103) may include a camera (1660) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (103).

[0314] Although not shown, in one embodiment, the wearable device (103) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being captured using the camera (1660). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame (1600) and the hinge units (1606, 1607).

[0315] According to one embodiment, the battery module (1670) may supply power to electronic components of the wearable device (103). In one embodiment, the battery module (1670) may be disposed within the first temple (1604) and / or the second temple (1605). For example, the battery module (1670) may be a plurality of battery modules (1670). The plurality of battery modules (1670) may be disposed within each of the first temple (1604) and the second temple (1605). In one embodiment, the battery module (1670) may be disposed at an end of the first temple (1604) and / or the second temple (1605).

[0316] The antenna module (1675) can transmit signals or power to the outside of the wearable device (103), or receive signals or power from the outside. In one embodiment, the antenna module (1675) can be positioned within the first temple (1604) and / or the second temple (1605). For example, the antenna module (1675) can be positioned close to one surface of the first temple (1604) and / or the second temple (1605).

[0317] The speaker (1655) can output an audio signal to the outside of the wearable device (103). The audio output module may be referred to as a speaker. In one embodiment, the speaker (1655) may be positioned within the first temple (1604) and / or the second temple (1605) so as to be positioned adjacent to the ear of a user wearing the wearable device (103). For example, the speaker (1655) may include a second speaker (1655-2) positioned within the first temple (1604) and thus adjacent to the user's left ear, and a first speaker (1655-1) positioned within the second temple (1605) and thus adjacent to the user's right ear.

[0318] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state to visually provide information regarding a specific state of the wearable device (103) to the user. For example, when the wearable device (103) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (1601) and / or the second rim (1602).

[0319] Referring to FIG. 16B, according to one embodiment, a wearable device (103) may include a printed circuit board (PCB) (1690). The PCB (1690) may be included in at least one of the first temple (1604) or the second temple (1605). The PCB (1690) may include an interposer positioned between at least two sub-PCBs. One or more hardware components included in the wearable device (103) may be positioned on the PCB (1690). The wearable device (103) may include a flexible PCB (FPCB) for interconnecting the hardware components.

[0320] According to one embodiment, the wearable device (103) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (103) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (103). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (103) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (103) based on the IMU.

[0321] Figures 17a and 17b illustrate an example of an exterior appearance of a wearable device according to one embodiment. The wearable device (103) of Figures 17a and 17b may correspond to an embodiment of the wearable device (103) described above with reference to Figure 15. An example of an exterior appearance of a first side (1710) of a housing of the wearable device (103) according to one embodiment is illustrated in Figure 17a, and an example of an exterior appearance of a second side (1720) opposite to the first side (1710) may be illustrated in Figure 17b.

[0322] Referring to FIG. 17A, according to one embodiment, a first surface (1710) of a wearable device (103) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (103) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (1604) and / or the second temple (1605) of FIGS. 16A and 16B). A first display (1650-1) for outputting an image to a left eye among the user's two eyes, and a second display (1650-2) for outputting an image to a right eye among the two eyes, may be disposed on the first surface (1710). The wearable device (103) is formed on the first surface (1710) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (1650-1) and the second display (1650-2).

[0323] According to one embodiment, the wearable device (103) may include cameras (1660-1) for photographing and / or tracking both eyes of the user adjacent to each of the first display (1650-1) and the second display (1650-2). The cameras (1660-1) may be referred to as the gaze tracking camera (1660-1) of FIG. 16B. According to one embodiment, the wearable device (103) may include cameras (1660-5, 1660-6) for photographing and / or recognizing the face of the user. The cameras (1660-5, 1660-6) may be referred to as FT cameras. The wearable device (103) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (1660-5, 1660-6). For example, the wearable device (103) may change the texture and / or shape of a portion of an avatar (e.g., a portion of an avatar representing a human face) using information obtained by cameras (1660-5, 1660-6) (e.g., FT cameras) and representing a facial expression of a user wearing the wearable device (103).

[0324] Referring to FIG. 17b, a camera (e.g., cameras (1660-7, 1660-8, 1660-9, 1660-10, 1660-11, 1660-12)) and / or a sensor (e.g., a depth sensor (1730)) for obtaining information related to the external environment of the wearable device (103) may be disposed on a second surface (1720) opposite to the first surface (1710) of FIG. 17a. For example, the cameras (1660-7, 1660-8, 1660-9, 1660-10) may be disposed on the second surface (1720) for recognizing external objects. Cameras (1660-7, 1660-8, 1660-9, 1660-10) may be referenced to the motion recognition cameras (1660-2, 1660-3) of FIG. 16b.

[0325] For example, using cameras (1660-11, 1660-12), the wearable device (103) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (1660-11) can be placed on the second face (1720) of the wearable device (103) to obtain an image to be displayed through the second display (1650-2) corresponding to the right eye among the two eyes. The camera (1660-12) can be placed on the second face (1720) of the wearable device (103) to obtain an image to be displayed through the first display (1650-1) corresponding to the left eye among the two eyes. As an example, the wearable device (101) can obtain a single screen using a plurality of images obtained through the cameras (1660-11, 1660-12). Cameras (1660-11, 1660-12) may be referenced to the shooting camera (1660-4) of FIG. 16b.

[0326] According to one embodiment, the wearable device (103) may include a depth sensor (1730) disposed on the second face (1720) to identify a distance between the wearable device (103) and an external object. Using the depth sensor (1730), the wearable device (103) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (103). Although not shown, a microphone may be disposed on the second face (1720) of the wearable device (103) to obtain a sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0327] In one embodiment, the wearable device (103) can acquire a high dynamic range image. For example, the wearable device (103) can acquire a high dynamic range image using at least one camera (1660-7, 1660-8, 1660-9, 1660-10, 1660-11, 1660-12). For example, the wearable device (103) can acquire the high dynamic range image described above with reference to FIG. 6.

[0328] For example, the wearable device (103) can acquire long exposure image frames and short exposure frames based on the difference in exposure values ​​defined by using at least one camera (1660-7, 1660-8, 1660-9, 1660-10, 1660-11, 1660-12). For example, the wearable device (103) can acquire long exposure image frames and short exposure frames alternately. For example, the wearable device (103) can acquire a high dynamic range image by synthesizing long exposure image frames and short exposure frames acquired alternately. For example, the wearable device (103) can acquire a high dynamic range image by synthesizing long exposure image frames and short exposure frames acquired using the camera (1660-11) that acquires an image to be displayed through the second display (1650-2). For example, the wearable device (103) can acquire a high dynamic range image by synthesizing a long exposure image frame and a short exposure frame acquired using a camera (1660-12) that acquires an image to be displayed through the first display (1650-1).

[0329] According to one embodiment, the wearable device (103) can display a high dynamic range image. For example, the wearable device (103) can display the high dynamic range image through the first display (1650-1) and the second display (1650-2). For example, the first display (1650-1) and the second display (1650-2) can be positioned at positions corresponding to the left and right eyes of the user, respectively. For example, the wearable device (103) can display a first high dynamic range image acquired using a camera (1660-12) through the first display (1650-1). For example, the wearable device (103) can display a second high dynamic range image acquired using a camera (1660-11) through the second display (1650-2). The first high dynamic range image may include a composite image of long exposure image frames and short exposure frames acquired using camera (1660-12). The second high dynamic range image may include a composite image of long exposure image frames and short exposure frames acquired using camera (1660-11).

[0330] In one embodiment, the wearable device (103) can acquire enhanced high dynamic range images. For example, the wearable device (103) can acquire enhanced high dynamic range images using cameras (1660-11, 1660-12). For example, the wearable device (103) can acquire enhanced high dynamic range images as described above with reference to FIGS. 7 to 13.

[0331] For example, the wearable device (103) can acquire long exposure image frames (e.g., 1110, 1130 of FIG. 11) using one camera (e.g., 1660-11) among the cameras (1660-11, 1660-12) and can acquire short exposure image frames (e.g., 1120, 1140 of FIG. 11) using the other camera (e.g., 1660-12). For example, the wearable device (103) can acquire the face size data and face brightness data described above with reference to FIG. 8 by analyzing the face from the long exposure image frames (e.g., 1130 of FIG. 11). For example, the wearable device (103) can obtain a long exposure image frame (e.g., 1111 of FIG. 11) to which a gain value has been applied by applying a gain value (e.g., 1170 of FIG. 11) to a long exposure image frame (e.g., 1110 of FIG. 11).

[0332] For example, the wearable device (103) can generate an enhanced high dynamic range (HDR) image (e.g., 1190 of FIG. 11) by synthesizing a long exposure image frame (e.g., 1111 of FIG. 11) to which a gain value is applied and a short exposure image frame (e.g., 1120 of FIG. 11). For example, the electronic device can generate an enhanced high dynamic range (HDR) image (e.g., 1190 of FIG. 11) by synthesizing at least one frame among long exposure preview image frames (e.g., 1130 of FIG. 11), a long exposure image frame (e.g., 1111 of FIG. 11) to which a gain value is applied and a short exposure image frame (e.g., 1120 of FIG. 11). For example, the electronic device can generate a high dynamic range (HDR) image (e.g., 1190 of FIG. 11) by synthesizing a short exposure image frame (e.g., 1120 or 1121) acquired within the closest time from the time a capture input (e.g., 1101 of FIG. 11) is received with a long exposure image frame (e.g., 1111 of FIG. 11) to which a gain value is applied.

[0333] An electronic device of the disclosed embodiment may include at least one camera module, a display, a memory for storing commands, and at least one processor. The electronic device may perform face detection on each of images acquired through the at least one camera module before receiving a capture input from a user, thereby obtaining first face data regarding the detected faces. Based on the first face data, the electronic device may identify a first image and second face data regarding a face included in the first image among the images acquired before receiving the capture input. The electronic device may obtain a second image and a third image using the at least one camera module. The second image may be obtained by having the at least one camera module receive light for a first exposure time. The third image may be obtained by having the at least one camera module receive light for a second exposure time that is shorter than the first exposure time. The electronic device may obtain third face data regarding a face included in the second image by performing face detection on the second image. The electronic device may obtain a first gain value for the second image based on the second face data and the third face data. The electronic device can obtain a high dynamic range (HDR) image using the second image and the third image to which the first gain value is applied.

[0334] According to one embodiment, the electronic device can identify a first image among images acquired before receiving a capture input based on the first facial data.

[0335] According to one embodiment, the electronic device can identify a first image among images acquired before receiving a capture input based on at least one of face size data and face brightness data included in the first face data.

[0336] According to one embodiment, the electronic device can obtain face size data based on the number of patches corresponding to a face included in the image among a plurality of patches that divide an image obtained through at least one camera module.

[0337] According to one embodiment, the electronic device can obtain face brightness data based on brightness values ​​of patches corresponding to a face included in the image among a plurality of patches that divide an image obtained through at least one camera module.

[0338] According to one embodiment, the electronic device can obtain the first gain value based on applying weights to each of the second facial data and the third facial data.

[0339] According to one embodiment, the electronic device can obtain third face brightness data by weighting first face brightness data of a first image included in the second face data and second face brightness data of a second image included in the third face data. The electronic device can identify a second gain value corresponding to the third face brightness data. The electronic device can obtain a first gain value based on the second gain value.

[0340] According to one embodiment, the electronic device can obtain third face size data by weighting first face size data of a first image included in the second face data and second face size data of a second image included in the third face data. The electronic device can identify a third gain value corresponding to the third face size data. The electronic device can obtain a first gain value based on the second gain value and the third gain value.

[0341] According to one embodiment, the electronic device may determine the shooting mode of the electronic device as a first mode based on an image acquired prior to receiving a capture input. The first mode may be a shooting mode for acquiring a high dynamic range (HDR) image. When the shooting mode of the electronic device is the first mode, the electronic device may acquire a second image and a third image in response to the capture input.

[0342] In one embodiment, the first image may be an image acquired through a first camera module among at least one camera module. The second image and the third image may be images acquired through a second camera module among at least one camera module. The first field of view of the first camera module may be narrower than the second field of view of the second camera module.

[0343] A method of operating an electronic device according to one embodiment of the disclosure may include an operation of acquiring first face data regarding detected faces by performing face detection on each of images acquired through at least one camera module. The method of operating the electronic device may include an operation of receiving a capture input. The method of operating the electronic device may include an operation of identifying, based on the first face data, a first image and second face data regarding a face included in the first image from among images acquired before receiving the capture input. The method of operating the electronic device may include an operation of acquiring a second image and a third image using at least one camera module. The second image may be acquired by having at least one camera module receive light for a first exposure time. The third image may be acquired by having at least one camera module receive light for a second exposure time that is shorter than the first exposure time. The method of operating the electronic device may acquire third face data regarding a face included in the second image by performing face detection on the second image. The method of operating the electronic device may acquire a first gain value for the second image based on the second face data and the third face data. A method of operating an electronic device may include obtaining a high dynamic range (HDR) image using a second image and a third image to which a first gain value is applied.

[0344] In one embodiment, the act of identifying the first image may include an act of identifying the first image among images acquired prior to receiving the capture input, based on the first facial data.

[0345] In one embodiment, the operation of identifying the first image may include an operation of identifying the first image among images acquired before receiving the capture input based on at least one of face size data and face brightness data included in the first face data.

[0346] In one embodiment, the face size data may be obtained based on the number of patches corresponding to a face included in the image among a plurality of patches that divide an image acquired through at least one camera module.

[0347] In one embodiment, the face brightness data may be obtained based on brightness values ​​of patches corresponding to a face included in the image among a plurality of patches that divide an image acquired through at least one camera module.

[0348] In one embodiment, the operation of obtaining the first gain value may include the operation of obtaining the first gain value based on applying a weight to each of the second facial data and the third facial data.

[0349] In one embodiment, the operation of obtaining the first gain value may include an operation of obtaining third face brightness data by weighting first face brightness data of a first image included in the second face data and second face brightness data of a second image included in the third face data. The operation of obtaining the first gain value may include an operation of identifying a second gain value corresponding to the third face brightness data. The operation of obtaining the first gain value may include an operation of obtaining the first gain value based on the second gain value.

[0350] In one embodiment, the operation of obtaining the first gain value may include an operation of obtaining third face size data by weighting first face size data of a first image included in the second face data and second face size data of a second image included in the third face data. The operation of obtaining the first gain value may include an operation of identifying a third gain value corresponding to the third face size data. The operation of obtaining the first gain value may include an operation of obtaining the first gain value based on the second gain value and the third gain value.

[0351] In one embodiment, the operation of acquiring the second image and the third image may include an operation of determining a shooting mode of the electronic device as a first mode based on the images acquired before receiving the capture input. The first mode may be a shooting mode for acquiring a high dynamic range (HDR) image. When the shooting mode of the electronic device is the first mode, the operation of acquiring the second image and the third image may include an operation of acquiring the second image and the third image in response to the capture input.

[0352] In one embodiment, the first image may be an image acquired through a first camera module among at least one camera module. The second image and the third image may be images acquired through a second camera module among at least one camera module. The first field of view of the first camera module may be narrower than the second field of view of the second camera module.

[0353] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

[0354] The methods according to the embodiments described in the claims or specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0355] When implemented in software, a computer-readable storage medium storing one or more programs (software modules) may be provided. The one or more programs stored in the computer-readable storage medium are configured for execution by one or more processors within an electronic device. The one or more programs include instructions that cause the electronic device to execute methods according to embodiments described in the claims or specification of the present disclosure.

[0356] In the present disclosure, the functions or operations performed by the electronic device may be performed by one or more processors executing one or more instructions stored in a memory. The functions or operations of the electronic device mentioned in the present disclosure may be performed by one processor executing one or more instructions, or may be performed by a combination of multiple processors executing one or more instructions. The processor mentioned in the present disclosure may be understood to include circuitry for performing calculations or controlling other components of the electronic device. For example, the one or more processors may include a central processing unit (CPU), a microprocessor unit (MPU), an application processor (AP), a communication processor (CP), a neural processing unit (NPU), a system on a chip (SoC), or an integrated circuit (IC) configured to execute one or more instructions. The one or more processors may be configured to perform the operations of the electronic device described above.

[0357] In the present disclosure, a program (software module, software) may be stored in a non-volatile memory including a random access memory (RAM), a flash memory, a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a magnetic disc storage device, a compact disc ROM (CD-ROM), a digital versatile disc (DVD) or other forms of optical storage, a magnetic cassette. Or, it may be stored in a memory formed by a combination of some or all of these. The memory may be formed by a single storage medium, or may be formed by a combination of a plurality of storage media. The one or more commands may be stored in a single storage medium, or may be distributed and stored in a plurality of storage media.

[0358] Additionally, the program may be stored on an attachable storage device that is accessible via a communication network such as the Internet, an intranet, a local area network (LAN), a wide LAN (WLAN), or a storage area network (SAN), or a combination thereof. Such a storage device may be connected to a device performing an embodiment of the present disclosure via an external port. Additionally, a separate storage device on the communication network may be connected to a device performing an embodiment of the present disclosure.

[0359] In the specific embodiments of the present disclosure described above, components included in the disclosure are expressed singularly or plurally, depending on the specific embodiment presented. However, the singular or plural expressions are selected to suit the presented situation for convenience of explanation, and the present disclosure is not limited to singular or plural components. Components expressed in plural may be composed of singular elements, or components expressed in singular may be composed of plural elements.

[0360] Additionally, in the present disclosure, terms such as “part”, “module”, etc. may refer to a hardware component such as a processor or circuit, and / or a software component executed by a hardware component such as a processor.

[0361] A "component" or "module" may be implemented by a program stored in an addressable storage medium and executed by a processor. For example, a "component" or "module" may be implemented by components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

[0362] The specific implementations described in this disclosure are merely exemplary and do not limit the scope of the present disclosure in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted.

[0363] Additionally, in the present disclosure, “comprising at least one of a, b, or c” may mean “comprising only a, including only b, including only c, or including a combination of two or more (including a and b, including b and c, including a and c, or including all of a, b, and c).

[0364] While the detailed description of this disclosure has described specific embodiments, it should be understood that various modifications are possible without departing from the scope of this disclosure. Therefore, the scope of this disclosure should not be limited to the described embodiments, but should be defined not only by the scope of the claims described below, but also by equivalents thereof.

Claims

1. In electronic devices, At least one camera module; display; memory that stores commands; and Contains at least one processor, The above instructions are executed by the at least one processor, thereby causing the electronic device to: Before receiving a capture input from a user, first face data regarding the detected faces are obtained by performing face detection on each of the images acquired through the at least one camera module, In response to the above capture input: Based on the first facial data, identify a first image and second facial data regarding a face included in the first image among the images acquired before receiving the capture input, Acquiring a second image and a third image using at least one camera module, wherein the second image is acquired by the at least one camera module receiving light for a first exposure time, and the third image is acquired by the at least one camera module receiving light for a second exposure time shorter than the first exposure time, By performing face detection on the second image, third face data regarding the face included in the second image is obtained, Based on the second face data and the third face data, a first gain value for the second image is obtained, By using the second image and the third image to which the first gain value is applied, a high dynamic range image is obtained. Electronic devices.

2. In paragraph 1, The above instructions are executed by the at least one processor, thereby causing the electronic device to: Based on the first face data, identifying the first image among the images acquired before receiving the capture input. Electronic devices.

3. In paragraph 2, The above instructions are executed by the at least one processor, thereby causing the electronic device to: Identifying the first image among the images acquired before receiving the capture input based on at least one of the face size data and the face brightness data included in the first face data. Electronic devices.

4. In paragraph 3, The above instructions are executed by the at least one processor, thereby causing the electronic device to: Acquire the face size data based on the number of patches corresponding to the face included in the image among a plurality of patches that divide the image acquired through the at least one camera module. Electronic devices.

5. In paragraph 3, The above instructions are executed by the at least one processor, thereby causing the electronic device to: Acquire the face brightness data based on the brightness values ​​of patches corresponding to a face included in the image among a plurality of patches that divide the image acquired through the at least one camera module. Electronic devices.

6. In paragraph 1, The above instructions are executed by the at least one processor, thereby causing the electronic device to: To obtain the first gain value based on applying a weight to each of the second facial data and the third facial data, Electronic devices.

7. In paragraph 6, The above instructions are executed by the at least one processor, thereby causing the electronic device to: By weighting the first face brightness data of the first image included in the second face data and the second face brightness data of the second image included in the third face data, third face brightness data is obtained, Identify a second gain value corresponding to the third facial brightness data, To obtain the first gain value based on the second gain value, Electronic devices.

8. In paragraph 7, The above instructions are executed by the at least one processor, thereby causing the electronic device to: By weighting the first face size data of the first image included in the second face data and the second face size data of the second image included in the third face data, third face size data is obtained, Identify a third gain value corresponding to the third face size data, To obtain the first gain value based on the second gain value and the third gain value, Electronic devices.

9. In paragraph 1, The above instructions are executed by the at least one processor, thereby causing the electronic device to: Based on the image acquired before receiving the capture input, the shooting mode of the electronic device is determined as a first mode, the first mode being a shooting mode for acquiring the high dynamic range image, When the shooting mode of the electronic device is the first mode, the second image and the third image are acquired in response to the capture input. Electronic devices.

10. In paragraph 1, The first image is an image obtained through a first camera module among the at least one camera module, The second image and the third image are images acquired through the second camera module among the at least one camera module, The first field of view of the first camera module is a narrower range than the second field of view of the second camera module. Electronic devices.

11. In the method of operating an electronic device, An operation of acquiring first face data regarding detected faces by performing face detection on each of the images acquired through at least one camera module; The action of receiving capture input; An operation of identifying a first image and second facial data regarding a face included in the first image among images acquired before receiving the capture input based on the first facial data; An operation of acquiring a second image and a third image using at least one camera module, wherein the second image is acquired by the at least one camera module receiving light for a first exposure time, and the third image is acquired by the at least one camera module receiving light for a second exposure time shorter than the first exposure time; An operation of acquiring third face data regarding a face included in the second image by performing face detection on the second image; An operation of obtaining a first gain value for the second image based on the second face data and the third face data; and An operation of obtaining a high dynamic range image by using the second image and the third image to which the first gain value is applied, How an electronic device operates.

12. In paragraph 11, The operation of identifying the first image above is: An operation of identifying the first image among images acquired before receiving the capture input based on the first facial data, How an electronic device operates.

13. In paragraph 11, The operation of obtaining the above first gain value is: An operation of obtaining the first gain value based on applying a weight to each of the second facial data and the third facial data, How an electronic device operates.

14. In paragraph 11, The operation of obtaining the second image and the third image is as follows: An operation of determining a shooting mode of the electronic device as a first mode based on an image acquired before receiving the capture input, the first mode being a shooting mode for acquiring the high dynamic range image; and When the shooting mode of the electronic device is the first mode, an operation of acquiring the second image and the third image in response to the capture input is included. How an electronic device operates.

15. In paragraph 11, The first image is an image obtained through a first camera module among the at least one camera module, The second image and the third image are images acquired through the second camera module among the at least one camera module, The first field of view of the first camera module is a narrower range than the second field of view of the second camera module. How an electronic device operates.

Images